Steel production line equipment diagnosis method based on self-learning entity relationship joint extraction

Abstract

The present invention relates to a method for diagnosing steel production line equipment faults based on a self-learning knowledge graph, comprising: obtaining steel production line equipment data and preprocessing it; constructing a self-learning entity relationship joint extraction model based on an encoder-decoder structure; extracting human-machine-object triplet data from human, information system and physical system using the self-learning entity relationship joint extraction model; constructing a self-learning knowledge graph of steel production line equipment faults based on the human-machine-object triplet data, and performing visual display; and realizing the evolution of the steel production line equipment fault knowledge graph based on a graph neural network and reinforcement learning. Compared with the prior art, the present invention can accurately and effectively provide richer fault diagnosis information, thereby improving fault diagnosis results and providing effective fault solutions.

Description

Technical Field

[0001] The present invention relates to the field of knowledge graph technology, and in particular to a steel production line equipment diagnosis method based on self-learning entity relationship joint extraction. Background Art

[0002] Traditional equipment fault diagnosis in the steel industry is often based on physical systems and people. Physical systems are various production equipment and various signal collection devices. When a production equipment fails, maintenance personnel perceive the signals collected by the signal collection equipment based on their own experience, data-driven equipment maintenance inspection system, existing maintenance records, internal equipment information, etc., analyze them, and then determine the fault location and cause. Obviously, fault decision knowledge is mainly based on expert experience and mechanism models, and the knowledge source is single. This method cannot analyze the fault location and specific cause well.

[0003] With the development of science and technology, steel industry equipment is becoming more and more intelligent and complex. In the context of big data, the growth rate of data volume is gradually accelerating. Therefore, the knowledge used for fault diagnosis has the characteristics of complex structure, wide variety, and strong coupling, which makes traditional fault diagnosis techniques such as multivariate statistical analysis no longer applicable. At the same time, with the progress of disciplines such as machine learning, artificial intelligence, pattern recognition, and control theory, the combination of fault diagnosis technology and various types of knowledge has made a qualitative leap. Knowledge graph technology has been introduced into the field of fault diagnosis. The process of knowledge graph construction involves several key steps: data acquisition, knowledge extraction, knowledge representation, quality assessment, etc. In the data acquisition stage, we have obtained the information we need. However, this information may exist in the form of text or data and cannot be directly used to build a knowledge graph. We need to extract knowledge from these data that can build a knowledge graph and construct the minimum element triple of the knowledge graph. This process is entity relationship extraction. The core task of building a knowledge graph is to identify and parse the relevant entities and their semantic relationships in the text data, and then represent the extracted entity pairs and their relationships in the form of triple semantic relationships. In the industrial field, fault knowledge graph diagnosis systems based on manual and semi-automatic construction have shortcomings such as sparse internal relationships, poor interpretability, and low accuracy of machine learning methods. Summary of the invention

[0004] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and to provide a steel production line equipment fault diagnosis method based on self-learning knowledge graph, so as to perform real-time, rapid and accurate diagnosis of steel production line equipment faults and provide appropriate solutions.

[0005] The purpose of the present invention can be achieved by the following technical solutions:

[0006] A method for diagnosing faults of steel production line equipment based on a self-learning knowledge graph comprises the following steps:

[0007] S1: Obtain steel production line equipment data;

[0008] S2: Build a self-learning entity-relationship joint extraction model. First, annotate the entity relationships in the steel production line equipment data. Then, use the encoder layer to convert each annotated token into a vector representation. Use the decoder layer to extract relationships and entities from the vector representation to construct triples. Then, train the encoder and decoder layers through joint training.

[0009] S3: extracting human-machine-object triplet data from humans, information systems, and physical systems using the self-learning entity relationship joint extraction model;

[0010] S4: Construct a knowledge graph of equipment failures in steel production lines based on the human-machine-object triplet data and display it visually.

[0011] Furthermore, the method further comprises the following steps:

[0012] S5: Evolution of the knowledge graph of equipment failures in steel production lines.

[0013] Furthermore, step S5 specifically includes the following sub-steps:

[0014] S51: Introducing reinforcement learning to mine implicit relationships in the knowledge graph of steel production line equipment failures and correcting erroneous triples;

[0015] The reinforcement learning predicts the implicit knowledge that has not yet been stored based on the explicit knowledge already in the steel production line equipment fault knowledge graph;

[0016] The prediction process of the reinforcement learning includes constructing an external environment and a reinforcement learning agent, wherein the external environment is used to specify the dynamics of the interaction between the agent and the indicator graph, and the external environment is modeled as defining a tuple <S, A, P, R> to represent a Markov decision process, where S is a continuous state space, A = {a1, a2, ..., a n} represents the set of all available actions, P(S t+1 =s′|S t =s,A t =a) is the transition probability matrix, R(s,a) is the reward function for each (s,a) pair;

[0017] The reinforcement learning agent includes: θ (s,a)=p(a|s;θ) transfers the state vector to the random policy and updates the neural network parameters θ using stochastic gradient descent;

[0018] The specific action is: given an entity pair (e s ,e t ), from the source entity e s Initially, the agent uses the Policy network to select the most promising relationship at each step of the expansion path until it reaches the target entity e t ,The selected relationship is the action, and the action space is defined as all the relationships in the knowledge graph of equipment failure in steel production line;

[0019] The state is specifically: capturing the semantic information of entities and relationships in the knowledge graph of steel production line equipment failure. After taking action, the agent will move from one entity to another. The state is used to capture the position of the agent in the knowledge graph. The vector calculation formula of the state is:

[0020] s t =(e t ,e tartget -e t )

[0021] Among them, e t represents the embedding of the current entity node at step t, e tartget represents the embedding of the target entity, in the initial state e t =e source ;

[0022] The rewards include global accuracy rewards, path efficiency rewards and path diversity rewards. GLOBAL The calculation expression is:

[0023]

[0024] The path efficiency reward r EFFICIENCY The calculation expression is:

[0025]

[0026] Here, a path p is defined as a relation sequence r1→r2→…→r n , length is the calculation function of path length;

[0027] The path diversity reward r DIVERSITY The calculation expression is:

[0028]

[0029] In the formula, F is the total number of paths;

[0030] The Policy network specifically uses a fully connected neural network to parameterize the Policy function π(s;θ) and converts the state vector S into a probability distribution of all possible actions.

[0031] Furthermore, the training process of the reinforcement learning agent includes:

[0032] The supervised policy of random breadth-first search on both sides is used for training. After inserting the path found by random breadth-first search on both sides, it is used to update the approximate gradient of the policy network. The update expression of the approximate gradient of the policy network is:

[0033]

[0034] Where J(θ) is the expected total reward of an interval. For each relation, the reasoning of each entity pair is regarded as an interval, r t belongs to path p;

[0035] Use the reward function to retrain the supervised Policy network. The gradient update expression of the Policy network is:

[0036]

[0037] R total =λ1r GLOBAL +λ2r GLoBAL +λ3r DIVERSITY

[0038] Among them, R total is a linear combination of the defined reward functions.

[0039] Furthermore, step S5 specifically includes the following sub-steps:

[0040] S52: Introduce graph neural network to calculate the embedding vector of new entities based on the limited auxiliary knowledge provided in the test phase;

[0041] The graph neural network includes a propagation model and an output model. The propagation model is used to determine how to propagate information from a node to its neighboring nodes. The graph neural network broadcasts the information of a node to more distant nodes by repeatedly applying the propagation model. The output model defines an objective function based on nodes and edges represented by vectors for a given task.

[0042] The graph neural network uses different propagation model parameters according to the time step n, and the transfer function of the propagation model at each time step n is The calculation expression is:

[0043]

[0044]

[0045]

[0046]

[0047]

[0048] Where e∈ε(G) is an entity, is the d-dimensional representation vector of e. In a knowledge graph G, the head entity neighborhood N h (e) = {(h,r,e)|(h,r,e)∈G}, tail neighborhood represents a transformation function that depends on the head / tail entity and the time step, P represents the sum pooling function, which maps a set of vectors to a vector. BN represents batch normalization;

[0049] The calculation expression of the objective function is:

[0050]

[0051]

[0052]

[0053]

[0054]

[0055] In the formula, Describes the complexity of the relationship, r * Indicates that at position l * The relationship that connects the most entities, Indicates that the relationship is at position l * The number of relations with the largest number of entities at [x] + = max(0,x), (h,r,t) represents a positive triplet, (h′,r,t′) represents one negative triplet for each positive triplet, and τ is the margin separating positive and negative triples.

[0056] Furthermore, step S2 specifically includes the following sub-steps:

[0057] S21: Use the translation model to translate Chinese sentences in the steel production line equipment data into English sentences, and use the BIEOS annotation scheme to annotate the entity relationships in the sentences;

[0058] S22: Construct an encoder layer, which includes an embedding layer and a context word representation layer. The embedding layer uses the BERT pre-trained language model as a shared feature encoding to model, converts the context of the word into a vector representation, and uses CNN to extract the character features of the data; the context word representation layer introduces a BiLSTM with a hole convolution to capture context information;

[0059] S23: construct a decoder layer, the decoder layer including a named entity recognition module and a multi-relation classifier, the named entity recognition module is used to detect entities with relationships in the text, the multi-relation classifier is used to identify the type of relationship contained in the text, and the decoder layer performs joint extraction based on the recognized entities and relationships;

[0060] S24: Use joint training method to train the multi-relation classifier and named entity recognition module.

[0061] Furthermore, step S22 specifically includes the following sub-steps:

[0062] S221: Given a sentence consisting of n words Among them, w t Represents the tth word in a sentence of length n. The BERT pre-trained language model is used as a shared feature encoder to convert each token in the sentence into a vector representation. The word embedding layer is used to express its semantic and grammatical meaning, and we get v = [v1, v2, …, v n ], where v t ∈R d Represents the d-dimensional word vector embedded in the tth word in the sentence. The embedding layer randomly initializes the embedding matrix and updates the weight parameters as the model is trained;

[0063] S222: Character-level representation of data using CNN To extract, the extraction expression is:

[0064]

[0065] The final representation of a word is the concatenation of the word-level and character-level representations. The final representation of a word is expressed as:

[0066]

[0067] Where X∈R n*dw , dw is the annotation embedding dimension, v t It is the word w t word embedding, Yes t Character-based representation of ;

[0068] S223: Introduce BiLSTM with dilated convolution as the context word representation layer to effectively capture context information; the LSTM memory block in the BiLSTM encoding layer is used to generate a new word according to the previous hidden vector h t-1 , the previous unit vector c t-1 and the current input word embedding x t Calculate the current hidden vector h t ;

[0069] The forward LSTM layer of the BiLSTM Encode the input sequence from x1 to x n , backward LSTM layer Transform the input sequence from x n Encoded to x1. Then we connect and And represents the final encoding information of the word The final sentence is expressed as

[0070] The semantic unit representation output by the 3-layer dilated convolution is combined with the word semantic information output by the Bi-LSTM as the output of the encoder. The vector expression generated at each time step is:

[0071] m t =[h t ; g t ]

[0072] In the formula, h t is the vector generated by Bi-LSTM at time step t, g t is the vector of semantic unit information extracted by the dilated convolution at the tth word;

[0073] Finally, the sentence representation is obtained

[0074] Furthermore, step S23 specifically includes the following sub-steps:

[0075] S231. We regard entity extraction as a sequence labeling task, detect all entities that may have relationships in the text, correctly identify them, and assign an entity label to each word, thus achieving preliminary entity extraction.

[0076] Because the correlation between neighborhood labels and the best label chain for joint decoding are beneficial for sequence labeling models, we use CRF for joint decoding instead of decoding each label independently. is the input sequence score, which is obtained from the position-aware sentence representation m t generate:

[0077] z t =Wu m t

[0078] in, is the label score of the tth word, N t is the number of different labels. t,j is regarded as the score of the jth tag at position t. We define the decoding score as:

[0079]

[0080] Where A is the transfer matrix, for example, A i,j represents the transition score from label i to label j. Then we get the conditional probability of all possible label sequences y, defined as follows:

[0081]

[0082] Among them, Y z represents the set of possible label sequences of Z. During training, we maximize the training set {(Z i ,y i )} log-likelihood of the correct label sequence:

[0083]

[0084] Decoding is to search for the label sequence that achieves the maximum score:

[0085]

[0086] The best label sequence y * It can be calculated using the Viterbi algorithm.

[0087] CRF decoding has two disadvantages: (1) The entity recognition results are not absolutely correct because they are predicted by the model during inference. Errors from entity labels may propagate to the relation classification branch and hurt performance. (2) The CRF decoding process is based on the Viterbi algorithm, which contains a non-differentiable argmax operation. To address this issue, we propose soft label embedding, which takes logits as input to preserve the probability of each entity type. Assuming N is the logits dimension, i.e., the number of entity types, and y is the label embedding matrix, the soft label embedding of the tth word is defined as follows:

[0088]

[0089] Get the final soft label embedding By learning soft label embeddings and using entity labels as input to the relation classification layer, the information between entity recognition and relation extraction can be effectively transferred.

[0090] S232. We regard the relation prediction task as a multi-label classification task, aiming to identify the relation types contained in the text. Inspired by the idea of ​​multi-label classification, we use a classifier chain for relation prediction. In order to train a better relation classifier to improve classification accuracy, the output vector M of the encoder layer and the soft label embedding H in NER are fused to construct the relation layer

[0091] δ=Norm(Concat(H,M))

[0092] Among them, Concat represents the concatenation operator, and Norm represents the normalization operation.

[0093] Then apply convolution operation and max pooling operation:

[0094] β=Conv(δ)

[0095] ρ = relu(max(β))

[0096] Among them, β∈R m×(n-l+1) is the feature output by the convolution operation, m is the number of filters, n is the length of the text, and l is the size of the convolution filter. First, a maximum pooling operation is applied to the feature β, and then the text embedding ρ∈R is obtained using the relu activation m , which is regarded as the local feature vector of the text.

[0097] The binary classifier for the jth relation type is as follows

[0098]

[0099]

[0100] In which, the above formula applies the linear layer to generate the hidden layer state R j ∈R d . is a learnable weight matrix. In the following formula, another linear layer with a softmax activation function is used to predict the probability distribution of whether the text contains the jth relation type, is the weight matrix. The hidden layer state R j is considered as the relation embedding of the jth relation type. If the text contains the jth relation type, R j Will be fed to the variable-length entity pair predictor to help entity pair recognition. In the classifier chain, the task is transformed into m label classification tasks, where the input vector of the first classifier is The input vector of the second classifier is δ and the output vector o of the first classifier c1The input vector of the mth classifier is:

[0101] p cm =[δ;o c1 ;…;o cm-1 ]

[0102] S233. The last part of the decoder is the joint extraction of entity relations. Prior to this process, two parallel processes, relation prediction and entity prediction, detect all relations present in the text and all entities related to the relations in the text, respectively. In the joint extraction process, we select two entities as target entity pairs and assign target relations to entity pairs so that they form a triplet. We use an attention model to perform joint extraction of entities and relations.

[0103]

[0104]

[0105] Among them, ci represents the output vector of the i-th target relation classifier, G t-1 represents the global embedding vector at the time step t-1 entity prediction, m i Represents the context vector produced by the encoding layer. represents the label sequence of the i-th relation. α represents the attention weight, Represents the context vector. Finally, the label sequence corresponding to the i-th relationship is obtained through the sofmax function

[0106] Furthermore, the model joint training process in step S24 specifically includes the following steps:

[0107] The joint loss function of the model is defined as:

[0108] L local =λ·L rel +(1-λ)·L ent

[0109] Among them, L rel represents the cross entropy loss for relation classification, L ent represents the cross entropy loss of named entity recognition predictor, and λ∈R is a hyperparameter used to balance the multi-relation classifier and named entity recognition predictor.

[0110] Furthermore, in step S1, the method for acquiring the steel production line equipment data includes online data crawling, offline data collection and preprocessing of the acquired data.

[0111] Compared with the prior art, the present invention has the following advantages:

[0112] (1) The present invention first obtains the steel production line equipment data and performs preprocessing, then constructs a self-learning entity relationship joint extraction model and performs training, uses the self-learning entity relationship joint extraction model to extract triples, and then displays the triples in Neo4j to realize the visualization of the steel production line equipment fault knowledge graph. Because the initially constructed steel production line equipment fault knowledge graph is incomplete, it is necessary to further complete the knowledge graph. In this process, GraphNNs are introduced to solve the problem of new entities outside the knowledge graph, and reinforcement learning is introduced to complete and update the missing relationships in the knowledge graph through reasoning. After multiple trainings, the fault knowledge graph is updated and evolved into a knowledge graph with richer information. The information-rich fault knowledge graph greatly improves the accuracy of steel production line equipment fault diagnosis.

[0113] (2) The present invention introduces reinforcement learning to fully mine the implicit relationships in the knowledge graph and corrects erroneous triples to a certain extent; it introduces graph neural networks GraphNNs to calculate the embedding vectors of new entities based on the limited auxiliary knowledge provided in the test phase, solving the problem of new entities outside the graph. The model was evaluated on a steel production line equipment failure dataset. The experimental results show that the steel production line equipment failure knowledge graph can accurately and effectively provide richer fault diagnosis information, thereby improving fault diagnosis results and providing effective fault solutions. BRIEF DESCRIPTION OF THE DRAWINGS

[0114] Figure 1 A schematic flow chart of a steel production line equipment diagnosis method based on self-learning entity relationship joint extraction provided in an embodiment of the present invention;

[0115] Figure 2 A schematic diagram of the structure of a self-learning entity relationship joint extraction model provided in an embodiment of the present invention;

[0116] Figure 3 A schematic diagram of a BiLSTM with dilated convolution provided in an embodiment of the present invention;

[0117] Figure 4 This is a schematic diagram of the results of S41 in the embodiment;

[0118] Figure 5 This is a schematic diagram of the results of S51 in the embodiment;

[0119] Figure 6 It is a schematic diagram of the results of S52 in the embodiment. DETAILED DESCRIPTION

[0120] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0121] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0122] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0123] Example 1

[0124] like Figure 1 As shown, this embodiment provides a method for diagnosing faults of steel production line equipment based on a self-learning knowledge graph, comprising the following steps:

[0125] S1: Obtain steel production line equipment data;

[0126] S2: Build a self-learning entity-relationship joint extraction model. First, annotate the entity relationships in the steel production line equipment data. Then, use the encoder layer to convert each annotated token into a vector representation. Use the decoder layer to extract relationships and entities from the vector representation to construct triples. Then, train the encoder and decoder layers through joint training.

[0127] S3: extracting human-machine-object triplet data from humans, information systems, and physical systems using the self-learning entity relationship joint extraction model;

[0128] S4: Construct a knowledge graph of equipment failures in steel production lines based on the human-machine-object triple data and display it visually;

[0129] S5. Evolution of the knowledge graph of steel production line equipment failures.

[0130] In this embodiment, data related to equipment failures in a steel production line are selected to construct a fault KG.

[0131] The step S1 specifically includes the following steps:

[0132] S11. Online data crawling: crawling online Internet data, such as encyclopedia websites and search logs retained by search engines, using Python-based web crawler technology

[0133] S12. Offline data collection: Collect a large amount of offline data related to production equipment maintenance generated during the steel production process, including file records and table records, mainly involving information such as faulty equipment, fault causes, fault phenomena, and solutions.

[0134] S13. Data preprocessing: The data crawled from the Internet and the data obtained from the steel production line are not standardized, so the data must be preprocessed first. Finally, the steel production line equipment failure data set is sorted to contain 19,630 simple sentences, which are manually annotated and sorted from the text information.

[0135] like Figure 2 As shown, the step S2 of constructing a self-learning entity relationship joint extraction model specifically includes the following steps:

[0136] S21. Use the translation model to translate the Chinese sentences into English sentences. Use the translation model to translate the Chinese sentences into English sentences, and then use the BIEOS annotation scheme to annotate the entity relationships in the sentences to solve the triple overlap problem.

[0137] S22. Construct the encoder layer: The encoder layer contains an embedding layer, a context word representation layer, and a relation attention layer with a gating mechanism. For the embedding layer, in order to better capture and summarize the semantics of a given sentence, the BERT pre-trained language model is used as a shared feature encoding to model the context of the word and convert it into a vector representation. Since out-of-vocabulary (OOV) words are common for entities, we also use CNN to extract character features of the data and improve the performance of the model through character-level features. For the context word representation layer, the introduction of BiLSTM with dilated convolution can effectively capture context information and solve the problem of long-term dependency restrictions.

[0138] Furthermore, the process of constructing the encoder layer in step S22 specifically includes the following steps:

[0139] S221. Given a sentence consisting of n words Among them, w t To represent the tth word in a sentence of length n, we use the BERT pre-trained language model as a shared feature encoder to convert each token in the sentence into a vector representation, and express its semantic and grammatical meaning through the word embedding layer, and obtain v = [v1, v2, …, v n ], where v t ∈R dRepresents the d-dimensional word vector embedded in the tth word in the sentence. The embedding layer randomly initializes the embedding matrix and updates the weight parameters as the model is trained.

[0140] S222. Since out-of-vocabulary words are common for entities, we use CNN to represent the data at the character level Character-level representation can effectively capture the morphological information of words and enhance the vocabulary through character-level information, thereby improving the performance of the model.

[0141]

[0142] The final representation of a word is the concatenation of the word-level and character-level representations:

[0143]

[0144] Where X∈R n*dw , dw is the annotation embedding dimension. t It is the word w t word embedding. Yes t A character-based representation of .

[0145] S223, such as Figure 3 As shown in the figure, BiLSTM with hole convolution is introduced as the context word representation layer to effectively capture context information and solve long-term dependencies. The LSTM memory block in the Bi-LSTM encoding layer is used to generate a new word based on the previous hidden vector h t-1 , the previous unit vector c t-1 and the current input word embedding x t Calculate the current hidden vector h t , the specific operation definition is as follows:

[0146] i t =σ(W xi x t +W hi h t-1 +w ci c t-1 +b i ) (3)

[0147] f t =σ(W xf x t +W hf h t-1 +W cf c ct-1 +b f ) (4)

[0148] g t =tanh(W xc xt +W h ch t-1 +W cc c t-1 +b c ) (5)

[0149] c t =i t g t +f t c t-1 (6)

[0150] o t =o(W xo x t +W ho h t-1 +W co c t +b0) (7)

[0151] h t =o t tanh(c t ) (8)

[0152] Where W and b represent the weight matrix and bias vector respectively, σ represents the logic function, i t represents the input gate, f t represents the forget gate and o t represents the output gates, which control the proportion of information to be forgotten and passed to the next time step. t Represents a memory unit that can maintain the previous state and remember the characteristics of the current input word.

[0153] Forward LSTM layer Encode the input sequence from x1 to x n , the backward LSTM layer transforms the input sequence from x n Encoded to x1. Then we connect and And represents the final encoding information of the word In this way, we can effectively utilize past and future features within a specific time range to obtain the final sentence representation It encodes the semantic information of the context. For multi-label classification problems, each category corresponding to different relations in the same sentence has a semantic unit that constitutes the semantic meaning of the entire text. Therefore, such a representation is not enough to decode the n tag sequences generated by our tagging scheme. Therefore, in addition to using Bi-LSTM to capture word-level semantic information, the extraction of semantic unit information is also very important. The dilated convolution can expand the receptive field without losing information, so the output of each convolution contains a wide range of information, which is very beneficial for capturing semantic units. Dilated convolution is often used in text processing with long sequence information dependencies. In this model, the dilated convolution and BiLSTM are combined to obtain the context extraction module Dconv-BiLSTM. The semantic unit representation output by the 3-layer dilated convolution is combined with the word semantic information output by the Bi-LSTM as the output of the encoder. The vector generated at each time step is:

[0154] m t =[h t ; g t ] (9)

[0155] Among them, h t is the vector generated by Bi-LSTM at time step t, g t is the vector of semantic unit information extracted by the dilated convolution at the tth word. The final sentence representation is It encodes rich contextual semantic information.

[0156] S23. Construct the decoder layer: The decoder layer consists of a named entity recognition module NER and a multi-relation classifier. The last part of the decoder is the joint extraction of entity relations. Before this process, two parallel processes, relation prediction and entity prediction, detect all relations present in the text and all entities related to the relations in the text respectively. Two entities are selected as target entity pairs and the target relation is assigned to the entity pair so that they can form a triple, and the attention model is used to perform the joint extraction of entities and relations.

[0157] Furthermore, the process of constructing the decoder layer in step S23 specifically includes the following steps:

[0158] S231. We regard entity extraction as a sequence labeling task, detect all entities that may have relationships in the text, correctly identify them, and assign an entity label to each word, thus achieving preliminary entity extraction.

[0159] Because the correlation between neighborhood labels and the best label chain for joint decoding are beneficial for sequence labeling models, we use CRF for joint decoding instead of decoding each label independently. We believe is the input sequence score, which is obtained from the position-aware sentence representation mt generate:

[0160] z t =W u m t (10)

[0161] in, is the label score of the tth word, N t is the number of different labels. t,j is regarded as the score of the jth tag at position t. The decoding score is defined as:

[0162]

[0163] Where A is the transfer matrix, for example, A i,j represents the transition score from label i to label j. Then we get the conditional probability of all possible label sequences y, defined as follows:

[0164]

[0165] Among them, Y z represents the set of possible label sequences of Z. During training, we maximize the training set {(Z i ,y i )} log-likelihood of the correct label sequence:

[0166]

[0167] Decoding is to search for the label sequence that achieves the maximum score:

[0168]

[0169] The best label sequence y * It can be calculated using the Viterbi algorithm.

[0170] CRF decoding has two disadvantages: (1) The entity recognition results are not absolutely correct because they are predicted by the model during inference. Errors from entity labels may propagate to the relation classification branch and hurt performance. (2) The CRF decoding process is based on the Viterbi algorithm, which contains a non-differentiable argmax operation. To address this issue, we propose soft label embedding, which takes logits as input to preserve the probability of each entity type. Assuming N is the logits dimension, i.e., the number of entity types, and y is the label embedding matrix, the soft label embedding of the tth word is defined as follows:

[0171]

[0172] Get the final soft label embedding By learning soft label embeddings and using entity labels as input to the relation classification layer, the information between entity recognition and relation extraction can be effectively transferred.

[0173] S232. We regard the relation prediction task as a multi-label classification task, aiming to identify the relation types contained in the text. Inspired by the idea of ​​multi-label classification, we use a classifier chain for relation prediction. In order to train a better relation classifier to improve classification accuracy, the output vector M of the encoder layer and the soft label embedding H in NER are fused to construct the relation layer

[0174] δ=Norm(Concat(G,M)) (16)

[0175] Among them, represents the connection operator and represents the normalization operation.

[0176] Then apply convolution operation and max pooling operation:

[0177] β=Conv(δ)

[0178] ρ = relu(max(β))

[0179] Among them, β∈R m×(n-l+1) is the feature output by the convolution operation, m is the number of filters, n is the length of the text, and l is the size of the convolution filter. First, a maximum pooling operation is applied to the feature β, and then the text embedding ρ∈R is obtained using the relu activation m , which is regarded as the local feature vector of the text.

[0180] The binary classifier for the jth relation type is as follows:

[0181]

[0182]

[0183] Applying the linear layer in Equation 19 produces the hidden layer state R j ∈R d . is a learnable weight matrix. In Formula 20, another linear layer with a softmax activation function is used to predict the probability distribution of whether the text contains the jth relation type, is the weight matrix. The hidden layer state R j is considered as the relation embedding of the jth relation type. If the text contains the jth relation type, R j Will be fed to the variable-length entity pair predictor to help entity pair recognition. In the classifier chain, the task is transformed into m label classification tasks, where the input vector of the first classifier is The input vector of the second classifier is δ and the output vector o of the first classifier c1 The input vector of the mth classifier is:

[0184] p cm =[δ;o c1 ;…;o cm-1 ] (twenty one)

[0185] S233. The last part of the decoder is the joint extraction of entity relations. Prior to this process, two parallel processes, relation prediction and entity prediction, detect all relations present in the text and all entities related to the relations in the text, respectively. In the joint extraction process, we select two entities as target entity pairs and assign target relations to entity pairs so that they form a triplet. We use an attention model to perform joint extraction of entities and relations.

[0186]

[0187] α=sofmax(β i ) (twenty three)

[0188]

[0189]

[0190] Among them, ci represents the output vector of the i-th target relation classifier, G t-1 represents the global embedding vector at the time step t-1 entity prediction, m i Represents the context vector produced by the encoding layer. represents the label sequence of the i-th relation. α represents the attention weight, Represents the context vector. Finally, the label sequence corresponding to the i-th relationship is obtained through the sofmax function

[0191] S24. Model training: During model training, BERT is fine-tuned; dynamic negative sampling is used to improve model performance and robustness, where the negative samples of entities and relations are 30 times the basic facts in each sentence. With this policy, our model maintains a more balanced data distribution on the training data; gradient clipping, parameter averaging, and L2 regularization are used to update model parameters; dropout is applied to the embedding layer, the final hidden layer of entity detection, and the final hidden layer of relation classification. The loss function of the model can be calculated by comparing the model output with the actual annotation results. The cross entropy loss function is used to define the loss of the multi-relation classifier and the named entity recognition predictor respectively. Because they share the same code clause input, the multi-relation classifier and the named entity recognition predictor are trained in a joint training manner.

[0192] Furthermore, the model joint training process in step S24 specifically includes the following steps:

[0193] S241. The joint loss function of the model is defined as:

[0194] L local =λ·L rel +(1-λ)·L ent (26)

[0195] Among them, L rel represents the cross entropy loss for relation classification, L ent represents the cross entropy loss of named entity recognition predictor, and λ∈R is a hyperparameter used to balance the multi-relation classifier and named entity recognition predictor. Since the performance of relation classification is usually worse than that of entity prediction, we rel Applying a larger weight score aims to make the model focus more on relation classification, λ>0.5.

[0196] Furthermore, the step S3 specifically includes the following steps:

[0197] S31. Use the entity relationship joint extraction model to extract human-machine-object triple data from people, information systems and physical systems.

[0198] Furthermore, the step S4 specifically includes the following steps:

[0199] S41. Visualization of knowledge graph of steel production line equipment failure: Store and represent the human-machine-object triples in the Neo4j graph database to visualize the knowledge graph of steel production line equipment failure. The partial schematic diagram is shown in the figure. Figure 4 shown.

[0200] Furthermore, the step S5 specifically includes the following steps:

[0201] S51. Introduce reinforcement learning to fully explore the implicit relationships in the knowledge graph and correct erroneous triples to a certain extent.

[0202] like Figure 5 As shown in the figure, after the introduction of reinforcement learning, the missing relationships in the fault knowledge graph are completed.

[0203] Furthermore, the step S51 introduces reinforcement learning to fully explore the implicit relationships in the knowledge graph and correct the erroneous triples to a certain extent, specifically including the following steps:

[0204] The knowledge graph is relatively sparse, and the hidden relationships between a large number of entities have not been fully explored. Knowledge graph completion technology based on reinforcement learning (RL) adopts the method of knowledge reasoning, using the explicit knowledge already in the knowledge graph to predict the implicit knowledge that has not yet been stored in the graph, and gradually completes the knowledge graph. Reinforcement learning consists of two parts: (1) The first part is the external environment E, which specifies the dynamics of the interaction between the agent and the knowledge graph. The environment is modeled as a Markov decision process (MDP). Define a tuple <S, A, P, R> to represent MDP, where S is a continuous state space and A = {a1, a2, …, a n} represents the set of all available actions, P(S t+1 =s′|S t =s,A t =a) is the transition probability matrix, and R(s,a) is the reward function for each (s,a) pair. (2) The second part of the system is the reinforcement learning agent, represented by the policy network π θ (s,a) = p(a|s;θ) It transfers the state vector to the random policy. The neural network parameters θ are updated using stochastic gradient descent.

[0205] S511, Action: Given an entity pair with a relation r (e s ,e t ), we hope that the agent can find the most informative path linking these entity pairs. s Initially, the agent uses the Policy network to select the most promising relationship at each step of the expansion path until it reaches the target entity e t In order to keep the output dimension of the Policy network consistent, the action space is defined as all relations in the knowledge graph.

[0206] S512, State: Translation-based embedding TransH is used to capture the semantic information of entities and relations in the knowledge graph. Each state captures the position of the agent in the knowledge graph. After taking an action, the agent moves from one entity to another. The two are connected by the action (relation) that the agent just took. The state vector at step t is calculated as follows:

[0207] s t =(e t ,e tartget -e t ) (27)

[0208] Among them, e t represents the embedding of the current entity node, e tartget Represents the embedding of the target entity. In the initial state e t =e source .

[0209] S513. Reward: To encourage the agent to find the predicted path, our reward function includes the following scoring criteria:

[0210] 1) Global Accuracy: The first reward function we added to our reinforcement learning model is defined as follows:

[0211]

[0212] If the agent reaches the goal after a sequence of actions, it receives an offline positive reward of +1.

[0213] 2) Path efficiency: The efficiency reward is defined as follows:

[0214]

[0215] Here, a path p is defined as a relation sequence r1→r2→…→r n .

[0216] 3) Path Diversity: We train the agent to find paths using positive samples from each relation. These training samples (e source ,e target ) have similar state representations in vector space. The agent tends to find paths with similar syntax and semantics. Often some of these paths may be related and therefore contain redundant information. To encourage the agent to find different paths, we define a diversity reward function using the cosine similarity between the current path and existing paths:

[0217]

[0218] in, Represents the relationship chain r1→r2→…→r n Path embedding.

[0219] S514, Policy Network: We use a fully connected neural network to parameterize the policy function π(s; θ), which transforms the state vector S into a probability distribution over all possible actions. The neural network consists of two hidden layers, each followed by a ReLU nonlinear layer. The output layer is normalized using the softmax function.

[0220] S515, training process: Use the supervised policy of random breadth-first search (BFS) on both sides to train the model. After inserting the path found by BFS, it is used to update the approximate gradient of the policy network. The calculation process is as follows:

[0221]

[0222] where J(θ) is the expected total reward of an interval, and for each relation, the reasoning of each entity pair is considered as an interval. t belongs to path p. In practice, AdamOptimizer with L2 regularization is used to update θ.

[0223] To find the reasoning path controlled by the reward function, we use the reward function to retrain the supervised Policy network. After each interval, the Policy network is updated with the following gradient:

[0224]

[0225] R total =λ1r GLOBAL +λ2r GLOBAL +λ3r DIVERSITY (33)

[0226] Among them, R total is a linear combination of the defined reward functions.

[0227] S52, the introduction of graph neural networks GraphNNs calculates the embedding vectors of new entities based on the limited auxiliary knowledge provided in the test phase, solving the problem of new entities outside the graph, such as Figure 6 shown.

[0228] Furthermore, the step S52 introduces the graph neural network GraphNNs to solve the problem of new entities outside the knowledge graph, which specifically includes the following steps:

[0229] Since the system does not input all entities into the network during training, the model does not have the embeddings of these entities, and therefore cannot predict the relationships between these entities in the constructed fault knowledge graph. Although this problem can be solved by retraining the embeddings using the relationship triplets containing the OOKB entities, retraining often consumes too much time and resources.

[0230] Graph-NN is a neural network defined on a graph structure, which provides a way to encode nodes and edges into vectors, as this is more suitable for knowledge graph completion. Graph neural networks consist of two models, a propagation model and an output model. The propagation model determines how to propagate information from a node to its neighboring nodes. By repeatedly applying this propagation model, we can broadcast the information of a node to more distant nodes, that is, each node can receive more information. In the model, stacked Graph-NNs are applied to knowledge graphs. The output model defines the objective function based on the nodes and edges represented by vectors for a given task.

[0231] Let ε be a set of entities, R be a set of relations, and G be a knowledge graph. Define a fact or relation triple as (h, r, t), where h, t∈ε and r∈R. The propagation process in stacked Graph-NN uses different model parameters depending on the time step n. The transfer function at each time step n is Defined by the following equation

[0232]

[0233]

[0234]

[0235]

[0236]

[0237] Among them, e∈ε(G) is an entity, is the d-dimensional representation vector of e. In a knowledge graph G, the head entity neighborhood N h (e) = {(h,r,e)|(h,r,e)∈G}, tail neighbor N t (e)={(e,r,t)|e,r,t)∈G}. also, represents a transformation function that depends on the head / tail entity and the time step, used to modify / transform the vectors of adjacent nodes to merge them into the current vector v e In the example, P represents the sum pooling function, which maps a set of vectors to a single vector. BN represents batch normalization.

[0238] Heterogeneity and imbalance are two typical characteristics of knowledge graphs. Heterogeneity means that some entities have simple relationships, while others have complex relationships connecting various entities. Imbalance means that the number of positive and negative sides connected by many relationships is very unequal. For relationships between entities of different difficulty, TranSpare uses matrices with different sparsity (different number of parameters) to represent them. It helps prevent underfitting of complex relationships or overfitting of simple relationships. Therefore, TranSpare is used as an embedding-based model. The scoring function is defined as follows.

[0239]

[0240]

[0241]

[0242]

[0243] in, Describes the complexity of the relationship. * Indicates that at position l * The relationship that connects the most entities. Indicates that the relationship is at position l * The number of relationships with the largest number of entities.

[0244] The Absolute-Margin objective function L based on TranSpare is used as the output model.

[0245] L=∑ (h,r,t)∈Δ ∑ (h′,r,t′)∈Δ′ [τ+f r (h,t)-f r (h′,t′)] + (43)

[0246] Where [x] + = max(0,x), (h,r,t) represents a positive triplet, (h′,r,t′) represents one negative triplet for each positive triplet, and τ is the margin separating positive and negative triples. This objective function considers positive and negative triples separately in the first and second terms. The score of positive triples will be optimized towards zero, while the score of negative triples will be at least τ. This objective function is easy to optimize and can give good results.

[0247] According to the above process, first obtain the steel production line equipment data and preprocess it, then build a self-learning entity relationship joint extraction model and train it, use the self-learning entity relationship joint extraction model to extract triples, and then display the triples in Neo4j to realize the visualization of the steel production line equipment fault knowledge graph. Because the initially constructed steel production line equipment fault knowledge graph is incomplete, it is necessary to further complete the knowledge graph. In this process, GraphNNs are introduced to solve the problem of new entities outside the knowledge graph, and reinforcement learning is introduced to complete the missing relationships in the knowledge graph through reasoning. After multiple trainings, the fault knowledge graph is updated and evolved into a knowledge graph with richer information, such as Figure 6 As shown in Figure 2, the information-rich fault knowledge graph greatly improves the accuracy of fault diagnosis of steel production line equipment.

[0248] The preferred specific embodiments of the present invention are described in detail above. It should be understood that a person skilled in the art can make many modifications and changes based on the concept of the present invention without creative work. Therefore, any technical solution that can be obtained by a person skilled in the art through logical analysis, reasoning or limited experiments based on the concept of the present invention on the basis of the prior art should be within the scope of protection determined by the claims.

Claims

1. A method for diagnosing faults in steel production line equipment based on self-learning knowledge graph, characterized in that: The following steps are involved: S1: Obtain steel production line equipment data; S2: Build a self-learning entity-relationship joint extraction model. First, annotate the entity relationships in the steel production line equipment data. Then, use the encoder layer to convert each annotated token into a vector representation. Use the decoder layer to decode the relationship and entity from the vector representation and extract the triples. Then, train the encoder and decoder layers through joint training. S3: extracting human-machine-object triplet data from humans, information systems, and physical systems using the self-learning entity relationship joint extraction model; S4: Construct a knowledge graph of equipment failures in steel production lines based on the human-machine-object triple data and display it visually; Step S2 specifically includes the following sub-steps: S21: Use the translation model to translate Chinese sentences in the steel production line equipment data into English sentences, and use the BIEOS annotation scheme to annotate the entity relationships in the sentences; S22: Construct an encoder layer, which includes an embedding layer and a context word representation layer. The embedding layer uses the BERT pre-trained language model as a shared feature encoding to model, converts the context of the word into a vector representation, and uses CNN to extract the character features of the data; the context word representation layer introduces a BiLSTM with a hole convolution to capture context information; S23: construct a decoder layer, the decoder layer including a named entity recognition module and a multi-relation classifier, the named entity recognition module is used to detect entities with relationships in the text, the multi-relation classifier is used to identify the type of relationship contained in the text, and the decoder layer performs joint extraction based on the recognized entities and relationships; S24: train the multi-relation classifier and named entity recognition module using joint training; Step S22 specifically includes the following sub-steps: S221: Given a sentence consisting of n words Among them, w t Represents the tth word in a sentence of length n. The BERT pre-trained language model is used as a shared feature encoding to convert each token in the sentence into a vector representation. The word embedding layer is used to express its semantic and grammatical meaning, and we get v = [v1, v2, …, b n ], where b t ∈R d Represents the d-dimensional word vector embedded in the t-th word in the sentence; the embedding layer randomly initializes the embedding matrix and updates the weight parameters as the model is trained; S222: Character-level representation of data using CNN To extract, the extraction expression is: The final representation of a word is the concatenation of the word-level and character-level representations. The final representation of a word is expressed as: Where X∈R n*dw , dw is the annotation embedding dimension, v t It is the word w t word embedding, Yes t Character-based representation of ; S223: Introduce BiLSTM with dilated convolution as the context word representation layer to effectively capture context information; the LSTM memory block in the BiLSTM encoding layer is used to generate a new word according to the previous hidden vector h t-1 , the previous unit vector c t-1 and the current input word embedding x t Calculate the current hidden vector h t ; The forward LSTM layer of the BiLSTM Encode the input sequence from x1 to x n , backward LSTM layer Transform the input sequence from x n Encoded to x1; then we connect and And represents the final encoding information of the word The final sentence is expressed as The semantic unit representation output by the 3-layer dilated convolution is combined with the word semantic information output by the Bi-LSTM as the output of the encoder. The vector expression generated at each time step is: m t =[h t ;g t ] In the formula, h t is the vector generated by Bi-LSTM at time step t, g t is the vector of semantic unit information extracted by the dilated convolution at the tth word; Finally, the sentence representation is obtained Step S23 specifically includes the following sub-steps: S231: Entity extraction is regarded as a sequence labeling task, all possible entities with relationships in the text are detected, an entity label is assigned to each word, and preliminary entity extraction is achieved; Use CRF for joint decoding, As the input sequence score, the input sequence score is derived from the position-aware sentence representation m t Generate, z t The generative expression for is: z t =W u m t in, is the label score of the tth word, N t is the number of different labels; Z t,j is regarded as the score of the jth tag at position t. For the tag sequence Define the decoding score as: Among them, A is the transfer matrix, A i,j represents the transfer score from label i to label j; The conditional probability expression of all possible label sequences y is: Where Y z Represents the set of possible label sequences of Z; During the training process, the decoder maximizes the training set {(Z i ,y i )}, the log-likelihood of the correct label sequence is calculated as: Decoding is to search for the label sequence with the maximum score. The calculation expression of decoding is: Set the soft label embedding to retain the probability of each entity type with logits as input. Assuming N is the logits dimension, i.e. the number of entity types, and y is the label embedding matrix, the soft label embedding of the tth word is defined as: Get the final soft label embedding By learning soft label embeddings; S232: Treat the relationship prediction task as a multi-label classification task, use a classifier chain for relationship prediction, and fuse the output vector M of the encoder layer and the soft label embedding H in NER to construct the relationship layer δ=Norm(Concat(H,M)) Among them, Concat represents the concatenation operator, and Norm represents the normalization operation; Apply convolution operations and max pooling operations: β=Conv(δ) ρ = relu(max(β)) Among them, β∈R m×(n-l+1) is the feature output by the convolution operation, m is the number of filters, n is the length of the text, and l is the size of the convolution filter; First, a maximum pooling operation is applied to the feature β, and then the text embedding ρ∈R is obtained using the relu activation m , text embedding is regarded as the local feature vector of text; The expression of the binary classifier of the j-th relationship type is: in, is a learnable weight matrix, is the weight matrix; the hidden layer state R j is considered as the relation embedding of the jth relation type; if a text contains the jth relation type, R j Will be fed to the variable-length entity pair predictor to help entity pair recognition; in the classifier chain, the task is transformed into m label classification tasks, where the input vector of the first classifier is The input vector of the second classifier is δ and the output vector o of the first classifier c1 The connection of , and so on; the input vector of the mth classifier is: p cm =[δ;o c1 ;…;o cm-1 ] S233: Perform joint extraction of entity relations, select two entities as target entity pairs, and assign target relations to the entity pairs so that they can form a triplet; use an attention model to perform joint extraction of entities and relations, and the expression of the attention model is: α=sofmax(β i ) Among them, ci represents the output vector of the i-th target relation classifier, G t-1 represents the global embedding vector at the time step t-1 entity prediction, m i Represents the context vector produced by the encoding layer; represents the label sequence of the i-th relation; α represents the attention weight, Represents the context vector; finally, the label sequence corresponding to the i-th relationship is obtained through the sofmax function 2. A method for diagnosing faults in steel production line equipment based on self-learning knowledge graph according to claim 1, characterized in that: The method further comprises the following steps: S5: Evolution of the knowledge graph of equipment failures in steel production lines.

3. A method for diagnosing faults in steel production line equipment based on self-learning knowledge graph according to claim 2, characterized in that: Step S5 specifically includes the following sub-steps: S51: Introducing reinforcement learning to mine implicit relationships in the knowledge graph of steel production line equipment failures and correcting erroneous triples; The reinforcement learning predicts the implicit knowledge that has not yet been stored based on the explicit knowledge already in the steel production line equipment fault knowledge graph; The prediction process of the reinforcement learning includes constructing an external environment and a reinforcement learning agent, wherein the external environment is used to specify the dynamics of the interaction between the agent and the indicator graph, and the external environment is modeled as defining a tuple<S,A,P,R> To represent the Markov decision process, where S is a continuous state space, A = {a1, a2, …, a n } represents the set of all available actions, P(S t+1 =s ′ |S t =s,A t =a) is the transition probability matrix, R(s,a) is the reward function for each (s,a) pair; The reinforcement learning agent includes: θ (s,a)=p(a|s;θ) transfers the state vector to the random policy and updates the neural network parameters θ using stochastic gradient descent; The specific action is: given an entity pair (e s ,e t ), from the source entity e s Initially, the agent uses the Policy network to select the most promising relationship at each step of the expansion path until it reaches the target entity e t ,The selected relationship is the action, and the action space is defined as all the relationships in the knowledge graph of equipment failure in steel production line; The state is specifically: capturing the semantic information of entities and relationships in the knowledge graph of steel production line equipment failure. After taking action, the agent will move from one entity to another. The state is used to capture the position of the agent in the knowledge graph. The vector calculation formula of the state is: s t =(e t ,e tartget -e t ) Among them, e t represents the embedding of the current entity node at step t, e tartget Represents the embedding of the target entity, in the initial state e t =e source ; The rewards include global accuracy rewards, path efficiency rewards and path diversity rewards. GLOBAL The calculation expression is: The path efficiency reward r EFFICIENCY The calculation expression is: Here, a path p is defined as a relation sequence r1→r2→…→r n , Length is the calculation function of the path length; The path diversity reward r DiVERSITY The calculation expression is: In the formula, F is the total number of paths; The Policy network specifically uses a fully connected neural network to parameterize the Policy function π(s;θ) and converts the state vector S into a probability distribution of all possible actions.

4. A method for diagnosing faults in steel production line equipment based on self-learning knowledge graph according to claim 3, characterized in that: The training process of the reinforcement learning agent includes: The supervised policy of random breadth-first search on both sides is used for training. After inserting the path found by random breadth-first search on both sides, it is used to update the approximate gradient of the policy network. The update expression of the approximate gradient of the policy network is: Where J(θ) is the expected total reward of an interval. For each relation, the reasoning of each entity pair is regarded as an interval, r t belongs to path p; Use the reward function to retrain the supervised Policy network. The gradient update expression of the Policy network is: R total =λ1r GLOBAL +λ2r GLOBAL +λ3r DIVERSITY Among them, R total is a linear combination of the defined reward functions.

5. A method for diagnosing faults in steel production line equipment based on self-learning knowledge graph according to claim 1, characterized in that: The model joint training process in step S24 specifically includes the following steps: The joint loss function of the model is defined as: L local =λ·L rel +(1-λ)·L ent Among them, L rel represents the cross entropy loss for relation classification, L ent represents the cross entropy loss of named entity recognition predictor, and λ∈R is a hyperparameter used to balance the multi-relation classifier and named entity recognition predictor.

6. A method for diagnosing faults in steel production line equipment based on self-learning knowledge graph according to claim 1, characterized in that: In step S1, the method for acquiring steel production line equipment data includes online data crawling, offline data collection and preprocessing of the acquired data.

Images