A Method for Constructing a Multimodal Fault Knowledge Graph for Belt Conveyors

Abstract

This invention discloses a method for constructing a multimodal fault knowledge graph for belt conveyors. First, an ontology is constructed and entity relationships are predefined. Fault text is annotated using the BIO data annotation format to generate a dataset. The Albert-BiLSTM-CRF model is then used to extract entities from the dataset to obtain structured data. A fault text knowledge graph for belt conveyors is constructed by combining the predefined entity relationships. Based on the entity information in the fault text knowledge graph, corresponding images and their URLs are retrieved from web pages or open-source image libraries to construct a fault image knowledge graph. Finally, the fault text knowledge graph and the image knowledge graph are merged to obtain the multimodal fault knowledge graph. This invention analyzes and diagnoses fault phenomena or causes through three application methods: fault information retrieval, fault information question answering, and a comprehensive view of fault causal relationships, obtaining corresponding fault handling solutions, thereby realizing the fault auxiliary diagnosis function of belt conveyors.

Description

Technical Field

[0001] This invention relates to the field of fault diagnosis and multimodal knowledge graph construction and application of belt conveyor systems, specifically to a method for constructing a multimodal fault knowledge graph for belt conveyors. Background Technology

[0002] Belt conveyors are large, complex mechanical material transport machines. Due to their operation under high loads in complex and harsh environments for extended periods, they are prone to various malfunctions. If a malfunction is not addressed promptly, it can lead to significant economic losses and even threaten personal safety. To ensure the safe and reliable operation of belt conveyors, it is essential to quickly and accurately locate the fault and develop a reasonable repair plan.

[0003] Commonly used knowledge-driven and data-driven fault diagnosis methods cannot effectively utilize large amounts of referable historical fault data and expert knowledge. Furthermore, their results have poor interpretability and cannot provide convenient decision-making solutions in practical applications. Knowledge graphs can fully utilize historical fault data, extract entities and relationships between entities from unstructured data, and store these entities and their relationships in a directed graph database. The main applications of knowledge graphs include intelligent question answering and intelligent recommendation, and they are widely used in fields such as medicine and agriculture. However, in the field of fault diagnosis, the application of knowledge graphs is still in the theoretical research stage. Moreover, there is currently no research specifically focusing on belt conveyor faults. Summary of the Invention

[0004] Purpose of the Invention: Addressing the problem that current fault diagnosis methods for belt conveyors are mostly data-driven, resulting in weak interpretability of diagnostic results and insufficient utilization of a large amount of unstructured prior data, this invention proposes a method for constructing a multimodal fault knowledge graph for belt conveyors. This method fully utilizes existing structured data, extracts entities from unstructured data including text and image data, constructs a multimodal fault knowledge graph for belt conveyors, and achieves fault diagnosis of belt conveyor systems through fault information retrieval, fault information question answering, and a comprehensive view of fault causal relationships.

[0005] Technical Solution: To achieve the objectives of this invention, the technical solution adopted is: a method for constructing a multimodal fault knowledge graph for a belt conveyor, comprising the following steps:

[0006] S1: Determine the entity types and relationship types existing in the multimodal fault knowledge graph of the belt conveyor; construct the ontology of the multimodal fault knowledge graph of the belt conveyor in a top-down manner. The ontology construction model includes six entity types: fault type, fault cause, fault location, fault phenomenon, handling solution, and image description.

[0007] S2: Select the collected text corpus of belt conveyor failures, divide it into training set, test set and validation set according to the set ratio, and perform BIO entity annotation on the corpus dataset;

[0008] S3: Train a named entity recognition model based on Albert-BiLSTM-CRF using the labeled training set. The input of the model is the collected text corpus of belt conveyor faults. Test the basic performance of the model on the test set and validation set.

[0009] S4: Use the trained model to extract entities from the document-level fault corpus to obtain structured fault entity data.

[0010] S5: Based on the ontology design scheme of S1, the relationship types between text entities are designed as five types: cause, occur, phenomenon, processing scheme, and imgof.

[0011] S6: Based on the entity extraction results of S4, obtain the fault image corresponding to the entity;

[0012] S7: Based on the extracted entities and defined relationships, construct a text-based fault knowledge graph for belt conveyors using the Neo4j graph database;

[0013] S8: Take the image data obtained in S6, use a URL address converter to obtain the URL addresses of the images in batches, and use the Neo4j graph database to access the URL addresses to build a knowledge graph of image faults for the belt conveyor.

[0014] S9: Merge the text knowledge graph obtained in S7 and the image knowledge graph obtained in S8 into a multimodal fault knowledge graph;

[0015] S10: Utilize the constructed multimodal fault knowledge graph to diagnose faults in belt conveyor systems in the form of fault information retrieval, fault information question answering, and a complete picture of fault causal relationships.

[0016] Furthermore, by adding information about the location and entity type of each character in the corpus:

[0017] The fault dataset contains fault entity types. The label B-FAT indicates that the character is the first character of a fault mode entity, with B being an abbreviation for Begin; the label I-FAT indicates that the character is not the first character of a fault mode entity, with I being an abbreviation for In; and the label O indicates that the character is not in the entity, with O being an abbreviation for Outside.

[0018] Furthermore, the structure of the named entity recognition model based on Albert-BiLSTM-CRF is as follows:

[0019] The Albert-BiLSTM-CRF model consists of three layers. The Albert layer generates word vectors from the input text through pre-training, and uses the word vectors as the input to the next BiLSTM layer. Through bidirectional training of BiLSTM, the model extracts feature information by using contextual information. Finally, the CRF layer determines the global optimal solution and obtains the best label sequence.

[0020] The Albert model decomposes the original word embedding matrix V*M into two matrices through parameter factorization, maps words to a low-dimensional embedding space E, and then projects them to a high-dimensional hidden space M. This reduces the number of parameters in the word embedding from O(V*M) to O(V*E+E*M).

[0021] Furthermore, the basic performance of the model was tested on the test and validation sets, with accuracy, recall, and F1 score as the calculation criteria. The calculation formulas are as follows:

[0022]

[0023] In the formula, P is precision, R is recall, TP represents the number of positive samples correctly predicted as positive (i.e., the number of characters whose labels are actually drawn), FP represents the number of negative samples incorrectly predicted as positive (i.e., the number of characters whose labels are not actually drawn), and FN represents the number of positive samples incorrectly predicted as negative (i.e., the number of characters whose labels are actually drawn but are not actually drawn). The final evaluation index F1 value is calculated from P and R.

[0024] Furthermore, the multimodal fault knowledge graph is integrated and merged using the following method:

[0025] When performing image search, text entities are used as the standard, and there is a one-to-one correspondence between image entities and text entities. First, the entity nodes for which the text-image entity relationship needs to be established are queried using the MATCH query statement of the Neo4j graph database. Then, the relationship between text nodes and image nodes is constructed.

[0026] Furthermore, by utilizing the constructed multimodal fault knowledge graph, fault diagnosis of the belt conveyor system is performed in the form of fault information retrieval, fault information question answering, and a complete picture of fault causal relationships.

[0027] The fault information retrieval method retrieves all relevant information about a fault by searching for a specific fault name: fault symptoms, location of occurrence, common causes, and solutions.

[0028] The fault information Q&A function allows users to retrieve the causes of faults by asking questions.

[0029] The complete picture of the causal relationship of faults is presented in the form of a knowledge graph, showing the causal relationship of all faults and their causes, and the relationship between fault nodes and cause nodes is set as prior probability.

[0030] Beneficial effects: Compared with the prior art, the technical solution of the present invention has the following beneficial technical effects:

[0031] (1) The present invention uses the Albert-BiLSTM-CRF model for named entity recognition tasks, which is more suitable for lightweight fault datasets in belt conveyor fault systems, and effectively improves the accuracy of entity extraction.

[0032] (2) This invention introduces image data to enhance the fault description of belt conveyors and uses the form of "entity-relationship-entity" triples to express the fault data of belt conveyors, which more intuitively displays the fault information of belt conveyors and forms a multi-dimensional and systematic belt conveyor fault knowledge network.

[0033] (3) This invention proposes a general process for constructing and applying a multimodal fault knowledge graph for belt conveyors by integrating text knowledge graphs and image knowledge graphs into a multimodal fault knowledge graph. It also provides the implementation methods for each step and demonstrates the feasibility of knowledge graphs in the field of belt conveyor fault diagnosis by combining existing structured data of belt conveyors with unstructured text data and image data.

[0034] (4) This invention proposes three application forms for the multimodal fault knowledge graph of belt conveyors: fault information retrieval, fault information question and answer, and fault causal relationship overview. Attached Figure Description

[0035] Figure 1 This invention describes the process of constructing and applying a multimodal fault knowledge graph for belt conveyors.

[0036] Figure 2 This is the result of constructing the knowledge graph ontology of this invention;

[0037] Figure 3 This is a schematic diagram of the Albert-BiLSTM-CRF model structure of the present invention;

[0038] Figure 4 This refers to the annotation results of the document data in this invention;

[0039] Figure 5 This is the result of building the named entity recognition model of this invention;

[0040] Figure 6 The effect of this invention's model on the test set;

[0041] Figure 7This demonstrates the effectiveness of the validation set on the model of this invention.

[0042] Figure 8 This is an example of the text knowledge graph construction result of this invention;

[0043] Figure 9 This is the result of constructing an example image knowledge graph for this invention;

[0044] Figure 10 This invention provides a multimodal fault knowledge graph.

[0045] Figure 11 This is the interface for retrieving fault-related information in this invention;

[0046] Figure 12 This is the interface showing the overall causal chain of the fault in this invention;

[0047] Figure 13 This is the fault causal relationship query interface for the present invention. Detailed Implementation

[0048] The technical solution of the present invention will be further described below with reference to the accompanying drawings and embodiments.

[0049] Figure 1 To configure the construction and application process of a multimodal fault knowledge graph for belt conveyors, the present invention provides a method for constructing a multimodal fault knowledge graph for belt conveyors, which specifically includes the following steps:

[0050] S1: Utilizing expert knowledge and existing structured data, determine the possible entity and relationship types in the multimodal fault knowledge graph of the belt conveyor. Construct the ontology of the multimodal fault knowledge graph of the belt conveyor using a top-down approach. The ontology construction model is as follows: Figure 2 As shown, it includes six major entity types: "fault type", "fault cause", "fault location", "fault phenomenon", "handling solution" and "image description".

[0051] S2: Select the collected text corpus of belt conveyor failures, divide it into training set, test set and validation set in a ratio of 8:1:1, and perform BIO entity annotation on the corpus dataset.

[0052] Specifically, this method adds a label to each character in the corpus to indicate its position and entity type. For example, if the fault dataset contains the entity type "fault", then the label "B-FAT" represents that the character is the first character of a fault mode entity, with B being an abbreviation for "Begin"; the label "I-FAT" represents that the character is not the first character of a fault mode entity, with I being an abbreviation for "In"; and the label "O" represents that the character is not in an entity, with O being an abbreviation for "Outside".

[0053] S3: Train a named entity recognition model based on Albert-BiLSTM-CRF using the labeled training set data. The model implementation structure is as follows:

[0054] First, the input text sequence is encoded, and each word is converted into a corresponding word vector. The word vectors are further encoded using an Albert layer to obtain the context-dependent representation of each word. The encoded word vectors are then input into a BiLSTM to obtain the hidden state representation of the bidirectional LSTM. The output of the BiLSTM is used as the feature sequence, and a CRF layer is used to decode the labels to obtain the optimal label sequence. The Albert layer is a lightweight pre-trained model that employs a parameter-sharing strategy, reducing the number of parameters and improving training efficiency.

[0055] Figure 3 This is a structural diagram of the Albert-BiLSTM-CRF model. The model consists of three layers. The Albert layer generates word vectors from the input text through pre-training. The word vectors are then used as the input to the next BiLSTM layer. Through bidirectional training of the BiLSTM, contextual information is used to extract feature information. Finally, the CRF layer determines the global optimal solution and obtains the best label sequence.

[0056] The BERT model uses an attention mechanism to effectively capture the relationships between sentence contexts. However, due to the large number of parameters in the BERT model and the long training time, the lightweight Albert model is used instead.

[0057] The Albert model primarily uses parameter factorization to decompose the original word embedding matrix V*M into two smaller matrices, mapping words to a low-dimensional embedding space E and then projecting them to a high-dimensional hidden space M. This reduces the number of parameters in the word embedding from O(V*M) to O(V*E + E*M), with the most significant reduction occurring when M is much larger than E. The second approach is parameter sharing. While each transformer layer in the Albert model is independent, the Albert model shares all parameters across all layers, further drastically reducing the number of parameters and effectively improving model stability.

[0058] The basic performance of the model was tested on the test and validation sets, and the calculation criteria were accuracy, recall, and F1 score, as shown in the following formulas:

[0059]

[0060] In the formula, P represents precision; R represents recall; TP represents the number of correctly predicted positive samples (i.e., the number of characters actually drawn with the labeled label); FP represents the number of incorrectly predicted negative samples (i.e., the number of characters actually drawn with a label other than the labeled label); and FN represents the number of incorrectly predicted positive samples (i.e., the number of characters labeled with a certain label but not actually drawn with that label). The final evaluation metric, F1 score, is calculated from P and R.

[0061] S4: Use the trained model to extract entities from the document-level fault corpus to obtain structured fault entity data.

[0062] S5: Based on the ontology design scheme of S1, the relationship types between text entities are designed as five types: "cause", "occurs", "phenomenon", "processing scheme" and "imgof".

[0063] S6: Based on the entity extraction results of S4, obtain the fault image corresponding to the entity from the webpage or public dataset.

[0064] For example, if the extracted entity for "Bearing damage often occurs on the reducer" is "Bearing damage" as the fault entity and "Reducer" as the location entity, then it is necessary to find and obtain the images of "Bearing damage" and "Reducer" and save them as .jpg files in the main body folder for use by S8.

[0065] S7: Based on the extracted entities and defined relationships, construct a text-based fault knowledge graph for belt conveyors using the Neo4j graph database.

[0066] S8: Take the image data obtained in S6, use a URL converter to batch obtain URLs prefixed with http: / / ms006.2.u8s.ru / ms006 / img / image / img / , and use the Neo4j graph database to access the URLs to build a knowledge graph of image faults for the belt conveyor.

[0067] S9: Merge the text knowledge graph obtained in S7 and the image knowledge graph obtained in S8 into a multimodal fault knowledge graph. The merging process is as follows:

[0068] When performing image searches, text entities are used as the standard, and there is a one-to-one correspondence between image entities and text entities. First, the Neo4j graph database uses the MATCH query statement to query the entity nodes for which the text-image entity relationship needs to be established. Then, the relationship between text nodes and image nodes is constructed as follows:

[0069] “MATCH

[0070] (p26:Fault{Name:“Roller Bearing Deformation”}),(Roller Bearing Deformation:Fault{Name:“Roller Bearing Deformation”})CREATE

[0071] (p26)-[:imgof]->(Drum bearing deformation)

[0072] RETURN*;”

[0073] S10: Utilize the constructed multimodal fault knowledge graph to diagnose faults in belt conveyor systems in the form of fault information retrieval, fault information question answering, and a complete picture of fault causal relationships.

[0074] The fault information retrieval function allows users to search for a specific fault name and obtain all relevant information about that fault, including fault symptoms, location, common causes, and solutions. The fault information question-and-answer function allows users to search for the causes of a fault by asking questions, such as "What caused the tear?". The comprehensive fault causal relationship function uses a knowledge graph to display the causal relationships between all faults and their causes, and the relationship between fault nodes and cause nodes is set as prior probabilities, which can more intuitively show the root cause of the fault.

[0075] Specific implementation examples:

[0076] 1. Ontology Construction

[0077] Based on existing structured data and expert experience, five entity types are designed: fault type, fault cause, fault phenomenon, fault location, handling solution, and image description; and five relation types are designed: cause, occurs, handling solution, phenomenon is, and imgof.

[0078] 2. Dataset Annotation Description: The dataset used in this case study is derived from excerpts from relevant literature. Individual documents contain, but do not fully contain, fault types, fault causes, fault locations, fault symptoms, and solutions. Some document content is shown below:

[0079] "When the redirecting roller and the drive roller are working normally, the noise is very small. Abnormal noise usually indicates that the bearing is damaged. A rattling sound will be heard at the bearing housing. In this case, the bearing needs to be replaced."

[0080] This document contains the location of the fault, the type of fault, the symptoms of the fault, and the solution, as shown in the table below.

[0081] Table 1 Entity Types and Entities

[0082]

[0083] The data results in this document are labeled as follows: Figure 4 As shown.

[0084] 3. Model Training

[0085] The belt conveyor fault document is divided into a training set of 2402 entities, a test set of 290 entities, and a validation set of 281 entities.

[0086] The Albert-BiLSTM-CRF named entity recognition model was trained using the labeled training set text. The model construction results are as follows: Figure 5 As shown in Table 2.

[0087] Table 2 Named Entity Recognition Model Parameter Settings

[0088] Parameter name Parameter value batch size 32 epoch 128 Learning rate 0.1

[0089] The model takes all characters in the dataset as input and outputs an M-dimensional vector corresponding to each character. Based on the entity label of each character, the corresponding entity is extracted. During model testing, the labeled data from the test set is extracted and compared with the labeled entities. The model performance is evaluated based on the comparison results. The model test and validation results are as follows: Figure 6 and Figure 7 As shown in the figure, the model identified 290 entities on the test set with an accuracy of 97.59%, a recall of 96.90%, and an F1 score of 97.23%; and identified 281 entities on the validation set with an accuracy of 93.96%, a recall of 91.81%, and an F1 score of 92.84%.

[0090] 4. Entity extraction

[0091] The model was used to extract the following from the fault document: "The noise level of the redirecting roller and drive roller is very low when they are working normally. Abnormal noise is usually caused by bearing damage, with a rattling sound at the bearing housing. In this case, the bearing should be replaced." The resulting entities were: fault entity—bearing damage; location entities—redirecting roller and drive roller; fault phenomenon entity—rattling sound at the bearing housing; and solution entity—replacing the bearing.

[0092] 5. Construction of text knowledge graphs

[0093] Based on the entity extraction results and the determined relationship types, a text knowledge graph is constructed, and the result is as follows: Figure 8 .

[0094] 6. Construction of Image Knowledge Graph

[0095] Based on the extracted entities, descriptive images of the redirecting roller, drive roller, and bearing damage need to be obtained from web pages or open-source image databases and saved as .jpg files in a folder. The image URLs should be obtained using a tool, in the format: http: / / ms006.2.u8s.ru / img / image / img1 / +filename. An image knowledge graph should then be constructed in the Neo4j graph database. The result is shown below. Figure 9 .

[0096] 7. Fusion of Text Knowledge Graph and Image Knowledge Graph: Using the construction statements mentioned in S9, the two graphs are fused. The fusion result is as follows: Figure 10 .

[0097] 8. Applications based on multimodal knowledge graphs

[0098] Once the knowledge graph is constructed, users can perform functions such as searching and querying based on the knowledge graph in the application interface of the belt conveyor multimodal fault knowledge graph.

[0099] The fault-related information retrieval function allows users to search for information describing a specific fault, such as the fault symptoms, location of the fault, common causes, solutions, and images describing the fault. In addition, it can display the nodes and relationships connected to the fault in the knowledge graph, such as... Figure 11 As shown.

[0100] The interface displaying the overall fault causal chain visualizes the entire fault knowledge graph, such as... Figure 12 As shown, the difference between this graph and the Neo4j graph database is that the relationships in this graph show the probabilities of occurrence between adjacent nodes, which helps to deduce the root cause of the failure.

[0101] Figure 13 This is a fault cause-and-effect relationship query interface. This interface supports searching for the causes of a certain fault, displaying them in the form of a graph, and retrieving and outputting information about related entities through a knowledge base.

Claims

1. A method for constructing a multimodal fault knowledge graph for a belt conveyor, characterized in that, Includes the following steps: S1: Determine the entity types and relationship types existing in the multimodal fault knowledge graph of the belt conveyor; construct the ontology of the multimodal fault knowledge graph of the belt conveyor in a top-down manner. The ontology construction model includes six entity types: fault type, fault cause, fault location, fault phenomenon, handling solution, and image description. S2: Select the collected text corpus of belt conveyor failures, divide it into training set, test set and validation set according to the set ratio, and perform BIO entity annotation on the corpus dataset; S3: Train a named entity recognition model based on Albert-BiLSTM-CRF using the labeled training set. The input of the model is the collected text corpus of belt conveyor faults. Test the basic performance of the model on the test set and validation set. S4: Use the trained model to extract entities from the document-level fault corpus to obtain structured fault entity data. S5: Based on the ontology design scheme of S1, the relationship types between text entities are designed as five types: cause, occur, phenomenon, processing scheme, and imgof. S6: Based on the entity extraction results of S4, obtain the fault image corresponding to the entity; S7: Based on the extracted entities and defined relationships, construct a text-based fault knowledge graph for belt conveyors using the Neo4j graph database; S8: Take the image data obtained in S6, use a URL address converter to obtain the URL addresses of the images in batches, and use the Neo4j graph database to access the URL addresses to build a knowledge graph of image faults for the belt conveyor. S9: Merge the text knowledge graph obtained in S7 and the image knowledge graph obtained in S8 into a multimodal fault knowledge graph; S10: Utilize the constructed multimodal fault knowledge graph to diagnose faults in belt conveyor systems in the form of fault information retrieval, fault information question answering, and a complete picture of fault causal relationships.

2. The method for constructing a multimodal fault knowledge graph for a belt conveyor according to claim 1, characterized in that, By adding information about the position and type of the entity to each character in the corpus: The fault dataset contains fault entity types. The label B-FAT indicates that the character is the first character of a fault mode entity, with B being an abbreviation for Begin; the label I-FAT indicates that the character is not the first character of a fault mode entity, with I being an abbreviation for In; and the label O indicates that the character is not in the entity, with O being an abbreviation for Outside.

3. The method for constructing a multimodal fault knowledge graph for a belt conveyor according to claim 1, characterized in that, The structure of the named entity recognition model based on Albert-BiLSTM-CRF is as follows: The Albert-BiLSTM-CRF model consists of three layers. The Albert layer generates word vectors from the input text through pre-training, and uses the word vectors as the input to the next BiLSTM layer. Through bidirectional training of BiLSTM, the model extracts feature information by using contextual information. Finally, the CRF layer determines the global optimal solution and obtains the best label sequence. The Albert model decomposes the original word embedding matrix V*M into two matrices through parameter factorization, maps words to a low-dimensional embedding space E, and then projects them to a high-dimensional hidden space M. This reduces the number of parameters in the word embedding from O(V*M) to O(V*E+E*M).

4. The method for constructing a multimodal fault knowledge graph for a belt conveyor according to claim 1, characterized in that, The basic performance of the model was tested on the test and validation sets, and the calculation criteria were accuracy, recall, and F1 score, as shown in the following formulas: In the formula, P is precision, R is recall, TP represents the number of positive samples correctly predicted as positive (i.e., the number of characters whose labels are actually drawn), FP represents the number of negative samples incorrectly predicted as positive (i.e., the number of characters whose labels are not actually drawn), and FN represents the number of positive samples incorrectly predicted as negative (i.e., the number of characters whose labels are actually drawn but are not actually drawn). The final evaluation index F1 value is calculated from P and R.

5. The method for constructing a multimodal fault knowledge graph for a belt conveyor according to claim 1, characterized in that, The method for fusing and merging multimodal fault knowledge graphs is as follows: When performing image search, text entities are used as the standard, and there is a one-to-one correspondence between image entities and text entities. First, the entity nodes for which the text-image entity relationship needs to be established are queried using the MATCH query statement of the Neo4j graph database. Then, the relationship between text nodes and image nodes is constructed.

6. The fault diagnosis method for a belt conveyor with a multimodal fault knowledge graph according to any one of claims 1-5, characterized in that, By utilizing a constructed multimodal fault knowledge graph, fault diagnosis of belt conveyor systems is performed in the form of fault information retrieval, fault information question-and-answer, and a comprehensive view of fault causal relationships. The fault information retrieval method retrieves all relevant information about a fault by searching for a specific fault name: fault symptoms, location of occurrence, common causes, and solutions. The fault information Q&A function allows users to retrieve the causes of faults by asking questions. The complete picture of the causal relationship of faults is presented in the form of a knowledge graph, showing the causal relationship of all faults and their causes, and the relationship between fault nodes and cause nodes is set as prior probability.

Images