An end-to-end fault knowledge ontology construction system based on multi-modal fusion and deep learning

Abstract

The application discloses an end-to-end fault knowledge ontology construction method and system based on multi-modal fusion and deep learning. The method comprises the following steps: obtaining maintenance log text data and sensor operation data of an industrial equipment; preprocessing the maintenance log text, and complementing missing symptoms based on the mapping relationship between fault modes and symptoms; adopting a BERT-CRF entity recognition model to extract four types of entities, i.e., fault modes, fault levels, occurrence frequencies and solutions, and the four types of entities together with component fields and the complemented symptoms constitute six types of core entities; adopting a four-element feature splicing relationship extraction model to identify semantic relationships between entities, such as hasLevel, hasFrequency and hasSolution; adopting a cross-modal graph neural network to fuse text and sensor modal features, and screening credible knowledge triples through heterogeneous graph convolution, modal attention and triple scoring; converting the six types of core entities, relationships and credible triples into ontology instances conforming to the OWL specification according to the ontology structure of six types of concepts and five types of relationships, generating an OWL ontology file and visualizing the file in layers. The above process is uniformly arranged by a main process control module, and is fully automated.

Description

Technical Field

[0001] This invention relates to the interdisciplinary fields of artificial intelligence, natural language processing, knowledge graphs and industrial fault diagnosis, specifically to an end-to-end fault knowledge ontology construction method and system based on multimodal fusion and deep learning, which is applicable to the knowledge extraction and automated generation of standardized ontology from industrial equipment maintenance logs. Background Technology

[0002] In the field of industrial equipment operation and maintenance, various faults inevitably occur during long-term equipment operation. After troubleshooting, maintenance technicians typically record fault-related information in the form of maintenance logs, including the faulty component, fault mode, fault level, frequency of occurrence, cause of fault, and handling solution. These maintenance logs contain a wealth of valuable fault domain knowledge and are important data assets for enterprises to conduct fault prediction, intelligent diagnosis, and knowledge management.

[0003] However, in actual production environments, these maintenance logs are mostly stored in unstructured or semi-structured formats in files such as Excel spreadsheets and Word documents, with inconsistent formats and non-standard descriptions. At the same time, various sensors deployed on industrial equipment (such as temperature sensors, vibration sensors, pressure sensors, etc.) continuously generate a large amount of real-time operating data. This sensor data and the maintenance log text data belong to different modalities, are isolated from each other, and have not been effectively integrated and utilized.

[0004] Knowledge ontology, as a formal method of knowledge representation, can describe domain concepts and their relationships in a structured way, and is the foundation for building intelligent fault reasoning systems. OWL (Web Ontology Language) is currently the most widely used ontology description language, with excellent semantic expressive power and tool ecosystem support. Constructing fault knowledge into an OWL ontology enables standardized storage, shared reuse, and automated reasoning of knowledge.

[0005] The existing technologies similar to this invention mainly include the following categories: (1) Rule-based knowledge extraction method: Fault-related entities are extracted from maintenance log text by manually writing regular expressions or keyword dictionaries. This method requires extensive participation from domain experts in rule formulation, and has poor robustness and low recall when faced with diverse expressions.

[0006] (2) Knowledge graph construction methods based on a single modality: These methods rely solely on text data or sensor data for fault knowledge extraction and modeling, failing to achieve effective fusion of multimodal data. For example, using only LSTM or basic pre-trained models for named entity recognition of text, the extracted fault entities are directly stored in a database or simple graph. The feature spaces of text modality and sensor modality are heterogeneous, and direct splicing or simple weighted fusion yields poor results, leading to insufficient completeness of fault knowledge and omitting important fault feature information contained in sensor data.

[0007] (3) Semi-automated ontology construction method: This method requires domain experts to manually define the ontology concept hierarchy and relation framework, and then use ontology editing tools to enter instances one by one. Although the ontology quality is high, the efficiency is extremely low and it is difficult to adapt to large-scale data scenarios.

[0008] (4) Existing research has also proposed a fault knowledge graph construction scheme for specific equipment (such as motors). It extracts knowledge from text data through a dual-channel mechanism of deep learning models and field mapping rules, and constructs a knowledge graph after entity alignment, attribute fusion and conflict resolution for fault diagnosis reasoning. However, the data source of such schemes is still limited to text data such as structured tables and maintenance work orders, and sensor operation data is not included in the knowledge construction process; its ontology model relies on experts to pre-build it with modeling tools, and the knowledge graph construction process also includes a manual review process, which cannot achieve end-to-end automated generation from original maintenance logs to standardized ontology.

[0009] In summary, the existing technologies generally have the following problems: (1) insufficient knowledge extraction accuracy; (2) lack of multimodal data fusion; (3) weak entity relationship extraction capability; (4) low ontology structuring degree; (5) high dependence on manual labor and low processing efficiency; (6) process break and lack of end-to-end capability; (7) poor knowledge reusability; (8) low standardization degree and difficulty in migration; (9) lack of traceability. Summary of the Invention

[0010] (a) The technical problem to be solved by the present invention In view of the shortcomings of the prior art, the technical problem to be solved by the present invention is: (1) Solve the problem of insufficient knowledge extraction accuracy: By using a sequence labeling method based on a combination of pre-trained language model and conditional random field, the deep semantic understanding ability of the pre-trained language model and the sequence label consistency constraint of the conditional random field are fully utilized to improve the accuracy of extracting core entities such as fault mode, fault level, occurrence frequency and solution from maintenance log text.

[0011] (2) Solve the problem of missing multimodal data fusion: By designing a cross-modal graph neural network fusion module, the text features of maintenance logs and the features of equipment sensor data are mapped to a unified feature space. The graph convolutional network is used to learn the relationship between nodes, and the information of different modalities is dynamically fused through the modal attention mechanism to improve the completeness of fault knowledge.

[0012] (3) Solve the problem of weak entity relationship extraction capability: By using a relationship extraction model based on a pre-trained language model, the model comprehensively utilizes global semantic features and local entity location features to accurately identify core relationships between entities such as hasLevel, hasFrequency, and hasSolution.

[0013] (4) Solve the problems of low ontology structuring and process fragmentation: By automatically constructing a standard OWL ontology, and using the main process control module to uniformly arrange all steps such as data preprocessing, entity extraction, relationship identification, cross-modal fusion, ontology construction and visualization output, end-to-end automated processing from Excel maintenance logs to OWL ontology files can be achieved without manual intervention.

[0014] (5) Solve the problems of poor knowledge reusability, low standardization and lack of traceability: Generate standard OWL ontology files that can be directly imported into mainstream ontology editing tools and fault inference engines, and automatically save model training performance indicators to support performance analysis and continuous optimization.

[0015] (II) The technical solution adopted by the present invention to solve its technical problem In a first aspect, the present invention provides a method for constructing an end-to-end fault knowledge ontology based on multimodal fusion and deep learning, comprising: Acquire text data from the maintenance logs of the industrial equipment and the sensor operating data of the industrial equipment; The maintenance log text data is preprocessed to obtain standardized maintenance log text; the preprocessing includes: checking and completing the necessary fields of the maintenance log text data, the necessary fields including at least the component field; and inferring and completing the missing symptom information based on the preset mapping relationship between fault mode keywords and symptoms; The standardized maintenance log text is input into the entity recognition model to extract four types of entities: fault mode, fault level, occurrence frequency, and solution. The entity recognition model extracts semantic features of the text based on a pre-trained language model and performs consistency constraint decoding on the label sequence through a conditional random field layer. The faulty component entity is obtained by mapping the component field, and the symptom entity is formed by the inferred and completed symptom information. Together with the four types of entities, they constitute six core entities: faulty component, fault mode, symptom, solution, fault level, and occurrence frequency. The standardized maintenance log text and the extracted entities are input into the relation extraction model to identify the semantic relationships between entity pairs. The text modal features of the standardized maintenance log text and the sensor modal features of the sensor operation data are extracted. A cross-modal graph neural network is used to fuse the text modal features and the sensor modal features. The credibility of the triples composed of entities and semantic relations is scored to select credible knowledge triples. Based on the preset fault knowledge ontology structure, the six types of core entities, the semantic relations, and the trusted knowledge triples are converted into ontology instances that conform to the OWL specification, generating and outputting standardized OWL ontology files. The preprocessing, entity extraction, relationship recognition, cross-modal fusion, and ontology conversion are uniformly arranged by the main process control module to achieve end-to-end automated processing from maintenance log text input to OWL ontology file output.

[0016] Furthermore, the entity recognition model is a BERT-CRF model: the BERT encoder extracts contextual semantic features; the linear layer maps the contextual semantic features to the emission scores of 9 types of BIO tags; the CRF layer learns the tag transition probabilities and decodes and outputs the optimal tag sequence through the Viterbi algorithm; the 9 types of BIO tags include non-entity tags, as well as start tags and internal tags for four types of entities: fault mode, fault level, occurrence frequency, and solution.

[0017] Furthermore, the input features of the relation extraction model are composed of four parts: a global semantic vector of the text, a head entity average pooling semantic feature vector, a tail entity average pooling semantic feature vector, and an element-wise difference vector between the head and tail entities. Specifically, the [CLS] vector of the pre-trained language model is used as the global semantic vector of the text. Average pooling is performed on the tokens corresponding to the head and tail entity positions in the output sequence of the pre-trained language model to obtain the head entity average pooling semantic feature vector and the tail entity average pooling semantic feature vector. The element-wise difference vector is calculated between the two. The fused feature formed by concatenating the four features is input into a fully connected classification layer to output a relation category. The relation category includes at least hasLevel, hasFrequency, and hasSolution.

[0018] Furthermore, the cross-modal fusion specifically includes: mapping text modal features and sensor modal features to a unified hidden dimension through a linear mapping layer; constructing a heterogeneous graph using entity nodes corresponding to the text modality and data nodes corresponding to the sensor modality; performing message passing and feature aggregation on the heterogeneous graph through at least two layers of graph convolutional networks; dynamically calculating the fusion weights of the text modality and sensor modality through a modal attention mechanism to obtain fused features; and scoring the triples composed of head entities, relations, and tail entities through a fully connected scoring layer to select credible knowledge triples with scores higher than a preset threshold.

[0019] Furthermore, the preset fault knowledge ontology structure includes six core concepts (Part, FailureMode, Symptom, Solution, FailureLevel, Frequency) and five semantic relations (hasFailureMode, hasSymptom, hasLevel, hasFrequency, hasSolution).

[0020] Furthermore, the ontology output also includes: automatically converting entities into OWL instances and replacing spaces in entity names with underscores; generating a layered visualization image based on the OWL ontology: divided into four levels according to components, failure modes, symptoms and solutions, failure levels and occurrence frequencies, with the horizontal coordinate of nodes in each level fixed to the level number and maintaining a fixed layer spacing, the vertical coordinate of nodes within each level being evenly distributed to avoid node overlap, different node styles being adopted for different levels, and the layered visualization image being output at a preset high resolution.

[0021] Furthermore, it also includes the step of training the entity recognition model: adopting a hierarchical learning rate strategy, using a first learning rate for the pre-trained language model layer, and a second learning rate higher than the first learning rate for the conditional random field layer and the linear layer; using linear preheating scheduling to gradually increase the learning rate in the early stage of training; evaluating the F1 score on the validation set after each round of training, saving only the model parameters with the highest F1 score, and writing the evaluation metrics during the training process into the metric file.

[0022] Secondly, this invention provides an end-to-end fault knowledge ontology construction system based on multimodal fusion and deep learning, comprising: The data processing module is configured to acquire maintenance log text data and sensor operation data of industrial equipment, and preprocess the maintenance log text data to obtain standardized maintenance log text. The preprocessing includes checking and completing necessary fields, which at least include component fields, and inferring and completing missing symptom information based on a preset mapping relationship between fault mode keywords and symptoms. The entity and relation extraction module includes an entity recognition model and a relation extraction model. The entity recognition model is configured to extract four types of entities: fault mode, fault level, occurrence frequency, and solution. The entity recognition model extracts text semantic features based on a pre-trained language model and decodes the label sequence with consistency constraints through a conditional random field layer. The relation extraction model is configured to identify the semantic relationships between entity pairs. The cross-modal knowledge fusion module is configured to extract text modal features and sensor modal features, use a cross-modal graph neural network to fuse the two modal features, and score the credibility of triples composed of entities and semantic relations to select credible knowledge triples. The fault ontology modeling module is configured to convert the six core entities, the semantic relations, and the trusted knowledge triples into ontology instances conforming to the OWL specification based on a preset fault knowledge ontology structure, and generate and output a standardized OWL ontology file; the six core entities are fault component entities mapped from the component fields, symptom entities composed of the inferred and completed symptom information, and the four types of entities, namely fault components, fault modes, symptoms, solutions, fault levels, and occurrence frequencies. The main process control module is configured to uniformly orchestrate the data processing module, the entity and relation extraction module, the cross-modal knowledge fusion module, and the fault ontology modeling module, so as to realize end-to-end automated processing from maintenance log text input to OWL ontology file output.

[0023] Furthermore, the system also includes a model training and evaluation module, configured to train the entity recognition model using a hierarchical learning rate strategy, and employ linear preheating scheduling, validation set F1 selection and saving, and index file recording.

[0024] Furthermore, the system also includes a configuration management module, configured to uniformly manage model configuration, training configuration, and path configuration.

[0025] (III) Beneficial Effects Compared with the prior art, the present invention has the following beneficial effects: 1. Full-process automation: Through unified orchestration by the main process control module, end-to-end processing from Excel maintenance logs to OWL ontology files is achieved without manual intervention in intermediate steps; under the exemplary experimental conditions, the construction efficiency is improved by more than 80%.

[0026] 2. Multimodal knowledge fusion: By fusing two heterogeneous modalities, maintenance log text and sensor operation data, through a cross-modal graph neural network, the shortcomings of existing technologies that only utilize a single text modality are overcome; under the exemplary experimental conditions, knowledge integrity is improved by 40%.

[0027] 3. High accuracy in knowledge extraction: The BERT-CRF entity recognition model combines the deep semantic understanding capability of the pre-trained language model with the sequence label consistency constraint of the conditional random field, and with the training strategy of hierarchical learning rate, linear warm-up and F1 selection of validation set, it significantly improves the recognition accuracy of faulty entities; the relation extraction model of four-element feature concatenation can accurately identify three types of core semantic relations.

[0028] 4. High compatibility and standardization: The generated OWL ontology conforms to international standards and can be directly imported into mainstream ontology editing tools and fault reasoning engines. It supports logical reasoning and semantic querying, enabling standardized reuse of knowledge.

[0029] 5. Traceability: Performance metrics such as F1 score, Precision, and Recall from model training are automatically saved to a metrics file, facilitating performance analysis, model comparison, and continuous optimization.

[0030] 6. Industrial adaptability: Supports batch processing of Excel maintenance logs, conforms to enterprise data storage habits, and lowers the deployment threshold; intelligent symptom inference can automatically fill in missing symptom information in the logs, improving the completeness of knowledge.

[0031] 7. Visual and intuitive: The hierarchical ontology visualization method intuitively displays the knowledge structure with a hierarchical layout and differentiated node styles, making it easy for operation and maintenance personnel to understand and use. Attached Figure Description

[0032] Figure 1 The system architecture diagram provided for embodiments of the present invention; Figure 2 This is a structural diagram of the BERT-CRF entity recognition model provided in an embodiment of the present invention; Figure 3 This is a structural diagram of the relation extraction model provided in an embodiment of the present invention; Figure 4 This is a structural diagram of the cross-modal GNN fusion model provided in an embodiment of the present invention; Figure 5 A conceptual hierarchy diagram of the fault knowledge ontology provided in this embodiment of the invention; Figure 6 This is an end-to-end processing flowchart provided for an embodiment of the present invention. Detailed Implementation

[0033] To make the objectives, technical solutions, and advantages of this invention clearer, the specific embodiments of this invention will be described in further detail below with reference to the accompanying drawings. The following embodiments are for illustrative purposes only and are not intended to limit the scope of protection of this invention.

[0034] The end-to-end fault knowledge ontology construction system of this invention comprises seven core modules that collaboratively realize the complete process of "data input → knowledge extraction → fusion modeling → ontology output". The overall system architecture is as follows: Figure 1 As shown. The seven core modules are: configuration management module, data processing module, entity and relation extraction module, cross-modal knowledge fusion module, fault ontology modeling and visualization module, model training and evaluation module, and main process control module.

[0035] Example (I) Overall System Architecture like Figure 1 As shown, the main process control module unifies the configuration management, data processing, model training and evaluation, and ontology construction and output into four stages. The data processing module interfaces with Excel maintenance logs for input; the model training and evaluation module calls the entity and relation extraction module and the cross-modal knowledge fusion module; and the ontology construction and output stage is completed by the fault ontology modeling and visualization module. The system input is Excel maintenance logs, and the output is OWL ontology files, visualization images, and model indicator files.

[0036] (ii) Configuration Management Module The configuration management module centrally manages all system parameters, including: model configuration (name and path of the pre-trained language model, hidden layer dimensions, e.g., 768 dimensions, number of entity categories (9 BIO tags), number of relation categories (3 categories); training configuration (batch size, training epochs, stratified learning rate, with a lower learning rate for the pre-trained language model layer and a higher learning rate for the CRF and linear layers); and path configuration (Excel data input path, OWL ontology output path, visualization result saving path, model weight saving path, and evaluation metric saving path). This module ensures the consistency and configurability of parameters across all system components, facilitating rapid deployment and parameter tuning optimization in different scenarios.

[0037] (III) Data Processing Module The data processing module consists of two parts: dataset processing and Excel data processing.

[0038] (1) Dataset processing: The maintenance log text is segmented using a word segmenter that is compatible with the pre-trained language model. The segments are uniformly truncated or padded to a fixed length (e.g., 128 tokens) and include special tags [CLS] and [SEP]. A BIO tag sequence aligned with the segmentation results is generated and out-of-bounds protection is performed. A bidirectional mapping dictionary between tag IDs and tag names is provided to facilitate the restoration of the model output into readable entity categories.

[0039] (2) Excel data processing: Automatically read one or more Excel maintenance log files under the specified path and perform standardized preprocessing, including: data reading and null value handling, filling null values ​​with default values; intelligent symptom inference, automatically inferring and supplementing symptom information based on fault mode keywords, for example, when the fault information contains the keyword "leakage", automatically supplementing the symptom "system pressure drop", and when it contains the keyword "overheating", automatically supplementing the symptom "abnormal temperature rise", thereby solving the problem of missing symptom records in maintenance logs; data verification, checking whether necessary columns (components, fault information, fault level, occurrence frequency, cause and handling) exist, and if missing, reporting an error and providing example prompts to ensure data quality.

[0040] (iv) Entity and Relationship Extraction Module This module comprises two core sub-models: an entity recognition model and a relation extraction model. The entity recognition model extracts four types of entities: fault mode, fault level, frequency of occurrence, and solution. Combined with the faulty component entities obtained by mapping component fields from maintenance logs and the symptom entities obtained through intelligent symptom inference from the data processing module, a total of six core entity types are formed: faulty component, fault mode, symptom, solution, fault level, and frequency of occurrence, each corresponding one-to-one with the six core concepts in ontology modeling.

[0041] (1) BERT-CRF entity recognition model The BERT-CRF entity recognition model is used to extract four core entities from maintenance log text: Failure Mode, Failure Level, Frequency, and Solution. Its structure is as follows: Figure 2 As shown.

[0042] The BERT encoder extracts contextual semantic features of the input text (e.g., a 768-dimensional vector sequence); the linear layer maps the 768-dimensional semantic features to 9 dimensions, i.e., emission scores of 9 BIO tags; the CRF layer learns the tag transition probability matrix and imposes logical constraints on the prediction results (e.g., the "fault mode start" tag can only be followed by the "fault mode inside" tag or a non-entity tag, but not the "fault level inside" tag) to ensure sequence tag consistency; during the training phase, the CRF negative log-likelihood loss is calculated and the model parameters are optimized through backpropagation; during the inference phase, the CRF layer decodes and outputs the globally optimal tag sequence through the Viterbi algorithm.

[0043] The 9 categories of BIO tag system are defined as follows: 0-Non-entity (O), 1-Fault Mode Start (B-FM), 2-Inside Fault Mode (I-FM), 3-Fault Level Start (B-FL), 4-Inside Fault Level (I-FL), 5-Occurrence Frequency Start (B-FQ), 6-Inside Occurrence Frequency (I-FQ), 7-Solution Start (B-SOL), 8-Inside Solution (I-SOL).

[0044] Taking the input text "Severe leakage of the seal of pump No. 3, occurring monthly, and the problem is solved by replacing the seal ring" as an example, the model outputs the label sequence "OO B-FM I-FM B-FL O B-FQ I-FQ O B-SOL I-SOL I-SOL O", and the corresponding extracted entities are: [Seal Leakage] Fault Mode, [Severe] Fault Level, [Occurring Monthly] Occurrence Frequency, and [Replacing the Seal Ring] Solution.

[0045] (2) Relation Extraction Model The relation extraction model is used to identify three core semantic relations between entity pairs: hasLevel (failure mode → failure level), hasFrequency (failure mode → occurrence frequency), and hasSolution (failure mode → solution). Its structure is as follows: Figure 3 As shown.

[0046] The model takes text, head entity position, and tail entity position as input. Its input features are composed of four parts: the [CLS] vector output by the pre-trained language model serves as the global semantic vector of the text; average pooling is performed on the tokens corresponding to the head entity positions in the sequence hidden layer output to obtain the head entity average pooled semantic feature vector; average pooling is performed on the tokens corresponding to the tail entity positions to obtain the tail entity average pooled semantic feature vector; and the element-wise difference between the head and tail entity average pooled semantic feature vectors is calculated to obtain the element-wise difference vector. The fused feature formed by concatenating the above four parts is input into a fully connected classification layer, which outputs the relation category. The combination of global semantic features, local head and tail entity features, and entity difference features enables the model to grasp both the overall semantics of the sentence and the relative semantic differences between entity pairs, achieving accurate relation classification.

[0047] (v) Cross-modal knowledge fusion module The cross-modal knowledge fusion module is used to fuse fault knowledge from text modalities (maintenance log text features) and sensor modalities (equipment operation data features) to solve the problem of multimodal data heterogeneity. Its structure is as follows: Figure 4 As shown, it includes four steps: modality encoding, GNN propagation, modality attention fusion, and triplet scoring. (1) Modal coding: By using mutually independent linear mapping layers, text features (e.g., 768 dimensions) and sensor features (e.g., 128 dimensions) are uniformly mapped to the hidden dimensions of GNN (e.g., 128 dimensions) to eliminate dimensional differences; (2) GNN propagation: A heterogeneous graph is constructed using entity nodes corresponding to the text modality and data nodes corresponding to the sensor modality. A two-layer graph convolutional network (GCN) is used to perform message passing and feature aggregation on the heterogeneous graph to learn the correlation features between nodes of different modalities. The first layer of GCN is followed by a ReLU activation function. (3) Modal attention fusion: The fusion weights of text modality and sensor modality are dynamically calculated through the attention mechanism, and the contribution of each modality is adaptively adjusted according to different fault scenarios; (4) Triple scoring: The fused (head entity, relation, tail entity) triples are scored by the fully connected layer, and the trusted knowledge triples with scores higher than the preset threshold are selected for subsequent ontology construction.

[0048] Through the above four-step fusion, the fault feature information (such as abnormal temperature, increased vibration, etc.) contained in the sensor operation data is incorporated into the fault knowledge, which is mutually verified and supplemented with the fault entities and relationships extracted from the text, significantly improving the completeness of the knowledge.

[0049] (vi) Fault ontology modeling and visualization module The fault ontology modeling and visualization module constructs a standardized OWL ontology from the extracted and fused fault knowledge, and achieves hierarchical visualization. The ontology concept hierarchy is as follows: Figure 5 As shown.

[0050] (1) Ontology structure design: Preset 6 core concepts (OWL Classes): Part, FailureMode, Symptom, Solution, FailureLevel, Frequency; Preset 5 relationships (OWL Object Properties): hasFailureMode (Part→FailureMode), hasSymptom (FailureMode→Symptom), hasLevel (FailureMode→FailureLevel), hasFrequency (FailureMode→Frequency), hasSolution (FailureMode→Solution).

[0051] (2) Entity Addition: The six core entities—faulty component, fault mode, symptom, solution, fault level, and frequency of occurrence—are automatically converted into OWL instances (Individual). Faulty component instances are mapped from the component field of the maintenance log data; symptom instances are inferred intelligently from the symptom data processing module; and fault mode, fault level, frequency of occurrence, and solution instances are extracted from the maintenance log text by the entity recognition model. Each of the six types of instances is then attached to its corresponding core concept class. Spaces in the instance names are replaced with underscores to conform to OWL syntax (e.g., "sealing leak" is converted to "sealing_leak"), and object attribute assertions are established based on the identified semantic relationships.

[0052] (3) Layered visualization: Divide into 4 levels according to "component → failure mode → symptom / solution → level / frequency". Fix the x-coordinate as the level number (the layer spacing is 5 units, for example), and the y-coordinate is evenly distributed in each level to avoid node overlap. Use differentiated node styles (for example, components are blue as the largest node, failure modes are red as the second largest node, and other categories are smaller nodes) and output visualization images with high DPI (for example, 800) to intuitively display the knowledge structure.

[0053] (vii) Model Training and Evaluation Module (1) Entity recognition training: A hierarchical optimizer is used. A lower learning rate is used for the pre-trained language model layer to avoid catastrophic forgetting, and a higher learning rate is used for the CRF layer and the linear classification layer to accelerate the convergence of new parameters. A linear warmup schedule is used to gradually increase the learning rate in the early stage of training to avoid training oscillation caused by excessive parameter updates in the early stage. After each round of training, the F1 score is evaluated on the validation set, and only the best model with the highest F1 score is saved. Various metrics during the training process are automatically written to the metric file (e.g., entity_metrics.txt).

[0054] (2) Entity recognition evaluation: Filter out invalid labels (padding labels, non-entity labels O), calculate weighted Precision, Recall, and F1 values ​​only for valid entity labels; generate human-readable classification reports and map label IDs to readable names (e.g., 1 → "failure mode B").

[0055] (viii) Main process control module The main flow control module implements complete end-to-end flow orchestration, with specific steps as follows: Figure 6 As shown: ① Initialization: Check GPU availability and load the word segmenter (preferably using the local model); ② Data Loading: Read Excel maintenance logs, handle null values, perform symptom inference, and generate standardized logs; ③ Data Splitting: Split the training and validation sets in a ratio of, for example, 8:2; ④ Model Training: Train the BERT-CRF entity recognition model and output the best F1 score, initialize the relation extraction model and cross-modal fusion model; ⑤ Knowledge Extraction: Use the trained model to extract entities and relations from the logs; ⑥ Ontology Construction: Add entities and relations to the OWL ontology and save the .owl file; ⑦ Visualization Generation: Generate ontology visualization images according to a hierarchical layout; ⑧ Result Output: Print and output a list of generated files, including the best model weight file, performance index file, OWL ontology file, and visualization image file.

[0056] (ix) Effect verification Under exemplary experimental conditions, the entire process of maintenance log data for a certain industrial equipment was verified: the system requires no manual intervention from inputting the maintenance logs from Excel to outputting the OWL ontology file, improving the construction efficiency by more than 80% compared to manual compilation; after introducing sensor modalities and performing cross-modal fusion, the completeness of fault knowledge is improved by 40%; the generated standard OWL ontology file can be directly imported into mainstream ontology editing tools, supporting logical reasoning and semantic query; the F1, Precision, Recall and other indicators of model training are automatically saved to the indicator file, facilitating performance analysis and continuous optimization.

[0057] It should be understood that the specific models, dimensional values, number of layers, interlayer spacing, resolution, etc., given in the above embodiments are all exemplary descriptions. Those skilled in the art can make various equivalent substitutions or modifications without departing from the concept of this invention. For example, the pre-trained language model can be replaced by other pre-trained language models such as RoBERTa and ALBERT; entity recognition can also use lightweight structures such as BiLSTM-CRF to replace BERT-CRF to reduce computational resource requirements; cross-modal fusion can use the Transformer cross-attention mechanism to replace GCN, or use simplified mechanisms such as feature concatenation and gated fusion; knowledge representation can use RDF to replace OWL, or store knowledge in a graph database. The above equivalent substitutions and modifications are all within the protection scope of this invention.

[0058] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A method for constructing an end-to-end fault knowledge ontology based on multimodal fusion and deep learning, characterized in that, include: Acquire text data from the maintenance logs of the industrial equipment and the sensor operating data of the industrial equipment; The maintenance log text data is preprocessed to obtain standardized maintenance log text; The preprocessing includes: checking and completing the necessary fields of the maintenance log text data, the necessary fields including at least the component field; and inferring and completing the missing symptom information based on the preset mapping relationship between fault mode keywords and symptoms. The standardized maintenance log text is input into the entity recognition model to extract four types of entities: fault mode, fault level, occurrence frequency, and solution. The entity recognition model extracts semantic features of the text based on a pre-trained language model and performs consistency constraint decoding on the label sequence through a conditional random field layer. The faulty component entity is obtained by mapping the component field, and the symptom entity is formed by the inferred and completed symptom information. Together with the four types of entities, they constitute six core entities: faulty component, fault mode, symptom, solution, fault level, and occurrence frequency. The standardized maintenance log text and the extracted entities are input into the relation extraction model to identify the semantic relationships between entity pairs. The text modal features of the standardized maintenance log text and the sensor modal features of the sensor operation data are extracted. A cross-modal graph neural network is used to fuse the text modal features and the sensor modal features. The credibility of the triples composed of entities and semantic relations is scored to select credible knowledge triples. Based on the preset fault knowledge ontology structure, the six types of core entities, the semantic relations, and the trusted knowledge triples are converted into ontology instances that conform to the OWL specification, generating and outputting standardized OWL ontology files. The preprocessing, entity extraction, relationship recognition, cross-modal fusion, and ontology conversion are uniformly arranged by the main process control module to achieve end-to-end automated processing from maintenance log text input to OWL ontology file output.

2. The method according to claim 1, characterized in that, The entity recognition model is a BERT-CRF model: the BERT encoder extracts the contextual semantic features of the standardized maintenance log text; the linear layer maps the contextual semantic features to the emission scores of 9 types of BIO tags; The CRF layer learns the label transition probability and decodes and outputs the optimal label sequence through the Viterbi algorithm; the nine types of BIO labels include non-entity labels, as well as start labels and internal labels for four types of entities: fault mode, fault level, occurrence frequency, and solution.

3. The method according to claim 1, characterized in that, The input features of the relation extraction model are composed of four parts: a global semantic vector of the text, a head entity average pooling semantic feature vector, a tail entity average pooling semantic feature vector, and an element-wise difference vector between the head and tail entities. The relation extraction model is implemented based on a pre-trained language model. The [CLS] vector is used as the global semantic vector of the text. The tokens corresponding to the head entity positions in the output sequence of the pre-trained language model are average pooled to obtain the head entity average pooling semantic feature vector, and the tokens corresponding to the tail entity positions are average pooled to obtain the tail entity average pooling semantic feature vector. The element-wise difference vector is calculated between the head entity average pooling semantic feature vector and the tail entity average pooling semantic feature vector. The fused features formed by concatenating these four parts are input into a fully connected classification layer, which outputs relation categories. These relation categories include at least the hasLevel relationship between fault mode and fault level, the hasFrequency relationship between fault mode and occurrence frequency, and the hasSolution relationship between fault mode and solution.

4. The method according to claim 1, characterized in that, The process of fusing the text modal features and the sensor modal features using a cross-modal graph neural network specifically includes: The text modal features and the sensor modal features are mapped to a unified hidden dimension through a linear mapping layer; A heterogeneous graph is constructed using entity nodes corresponding to text modalities and data nodes corresponding to sensor modalities. Message passing and feature aggregation are performed on the heterogeneous graph using at least two layers of graph convolutional networks; The fusion weights of text modality and sensor modality are dynamically calculated using a modal attention mechanism to obtain fusion features; The triples consisting of head entity, relation and tail entity are scored by a fully connected scoring layer, and trusted knowledge triples with scores higher than a preset threshold are selected.

5. The method according to claim 1, characterized in that, The preset fault knowledge ontology structure includes six core concepts and five semantic relationships. The six core concepts are: Part, FailureMode, Symptom, Solution, FailureLevel, and Frequency. The five semantic relationships are: the hasFailureMode relationship between Part and FailureMode, the hasSymptom relationship between FailureMode and Symptom, the hasLevel relationship between FailureMode and FailureLevel, the hasFrequency relationship between FailureMode and Frequency, and the hasSolution relationship between FailureMode and Solution.

6. The method according to claim 5, characterized in that, The generation and output of standardized OWL ontology files also includes: The entity is automatically converted into an OWL instance, and spaces in the entity name are replaced with underscores to conform to OWL syntax. Based on the OWL ontology, a layered visualization image is generated: it is divided into four levels according to components, failure modes, symptoms and solutions, failure levels and occurrence frequency. The horizontal coordinate of the nodes in each level is fixed to the level number of that level and a fixed layer spacing is maintained. The vertical coordinate of the nodes in each level is evenly distributed to avoid node overlap. Different levels adopt different node styles, and the layered visualization image is output at a preset high resolution.

7. The method according to claim 1, characterized in that, It also includes the step of training the entity recognition model: A hierarchical learning rate strategy is adopted, with a first learning rate applied to the pre-trained language model layer and a second learning rate higher than the first learning rate applied to the conditional random field layer and the linear layer. Linear warm-up scheduling is used to gradually increase the learning rate in the early stages of training; After each training round, the F1 score is evaluated on the validation set. Only the model parameters with the highest F1 score are saved, and the evaluation metrics during the training process are written to the metric file.

8. An end-to-end fault knowledge ontology construction system based on multimodal fusion and deep learning, characterized in that, include: The data processing module is configured to acquire maintenance log text data of industrial equipment and sensor operation data of the industrial equipment, and preprocess the maintenance log text data to obtain standardized maintenance log text; the preprocessing includes checking and completing necessary fields, the necessary fields including at least component fields, and inferring and completing missing symptom information based on a preset mapping relationship between fault mode keywords and symptoms; The entity and relation extraction module includes an entity recognition model and a relation extraction model. The entity recognition model is configured to extract four types of entities from the standardized maintenance log text: fault mode, fault level, occurrence frequency, and solution. The entity recognition model extracts text semantic features based on a pre-trained language model and performs consistency constraint decoding on the label sequence through a conditional random field layer. The relation extraction model is configured to identify the semantic relationships between entity pairs. The cross-modal knowledge fusion module is configured to extract the text modal features of the standardized maintenance log text and the sensor modal features of the sensor operation data, fuse the two modal features using a cross-modal graph neural network, and score the credibility of the triples composed of entities and semantic relations to select credible knowledge triples. The fault ontology modeling module is configured to convert the six core entities, the semantic relations, and the trusted knowledge triples into ontology instances conforming to the OWL specification based on a preset fault knowledge ontology structure, and generate and output a standardized OWL ontology file; the six core entities are fault component entities mapped from the component fields, symptom entities composed of the inferred and completed symptom information, and the four types of entities, namely fault components, fault modes, symptoms, solutions, fault levels, and occurrence frequencies. The main process control module is configured to uniformly orchestrate the data processing module, the entity and relation extraction module, the cross-modal knowledge fusion module, and the fault ontology modeling module, so as to realize end-to-end automated processing from maintenance log text input to OWL ontology file output.

9. The system according to claim 8, characterized in that, It also includes a model training and evaluation module, configured to train the entity recognition model using a hierarchical learning rate strategy: a first learning rate is used for the pre-trained language model layer, and a second learning rate higher than the first learning rate is used for the conditional random field layer and the linear layer. A linear warm-up scheduling method is used to gradually increase the learning rate in the early stages of training. After each training round, the F1 score is evaluated on the validation set, and only the model parameters with the highest F1 score are saved. The evaluation metrics during the training process are written to the metric file.

10. The system according to claim 8, characterized in that, It also includes a configuration management module, configured to uniformly manage model configuration, training configuration, and path configuration; the model configuration includes pre-trained language model parameters, hidden layer dimension, number of entity categories, and number of relation categories; the training configuration includes batch size, training epochs, and stratified learning rate; the path configuration includes maintenance log data path, OWL ontology output path, visualization result saving path, and model metric saving path.

Images