Intelligent metering device maintenance knowledge base management system

Abstract

The present application relates to a kind of intelligent metering equipment maintenance knowledge base management system, specifically to artificial intelligence field, the scheme is structured by extracting image, voice and text, mapping heterogeneous representation to shared semantic space and clustering as event cluster, then calling multi-modal big model generates the structured knowledge entry containing fault phenomenon, reason, measure and evidence link, finally executes entity alignment, relationship completion and conflict resolution, and then newly added knowledge is hung to graph node;From this, the automatic conversion from fragmented multi-modal data to executable maintenance knowledge is realized, the implicit experience of front line is effectively captured and deposited, and the loss of knowledge caused by personnel flow is prevented;At the same time, the content of knowledge base is continuously enriched, so that operation and maintenance personnel can conveniently obtain precise diagnosis suggestions with evidence chain, significantly reduce the repeat fault rate, shorten the growth cycle of new employees, and improve the practicability and vitality of knowledge base.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence, and more specifically, to an intelligent metering equipment maintenance knowledge base management system. Background Technology

[0002] In the energy management departments of large manufacturing enterprises, various smart metering devices, including smart meters, flow meters, pressure sensors, and temperature transmitters, are widely deployed to monitor the energy consumption and operating status of production lines, power workshops, and utilities in real time. Engineering teams face numerous equipment malfunctions during daily maintenance, and their communication and collaboration heavily rely on instant messaging groups, emails, conference calls, and internal work order systems. During these exchanges, engineers frequently upload photos of equipment taken on-site, showcasing visual anomalies such as component corrosion, aging wiring, or surface deformation; they also draw handwritten wiring diagrams or sticky notes annotating key parameters. Record abnormal audio noises emitted by the equipment during operation, such as the friction sound of motor bearings and electromagnetic vibration noise; at the same time, supplement them with conversational natural language descriptions, such as "this capacitor feels a bit hot to the touch", "similar to the fault on Line 3 last time", "tightening that screw will fix it for a while"; this multimodal information from front-line practice carries a lot of implicit maintenance experience and is the core knowledge asset for diagnosing difficult faults, formulating rapid repair strategies and preventing similar problems from recurring; however, this knowledge is stored in a fragmented form in different media, lacks semantic connection between them, and is highly dependent on personal memory and experience inheritance.

[0003] Currently, the processing of this type of unstructured information mainly relies on keyword retrieval, basic text classification, and optical character recognition technology. This can only extract text from images or perform simple text categorization, failing to deeply understand the visual features in images, such as the degree of component corrosion, circuit color markings, or solder joint shapes; it cannot analyze abnormal frequency components in audio signals and their correspondence with specific fault modes; and it cannot integrate multimodal information for deep cross-media reasoning, such as automatically associating a voice describing a "buzzing sound" with a photo showing a bulging capacitor and determining it as a precursor to filter capacitor failure. Furthermore, natural language is generally prone to... The colloquial expressions, ambiguous references, and omitting of context further exacerbate the difficulty of machine understanding. Due to the lack of an effective multimodal knowledge extraction mechanism, a large amount of valuable practical experience is lost with the departure of senior employees, resulting in a high rate of repeated failures, a long growth cycle for newly hired engineers, and knowledge base content gradually becoming outdated and rigid due to the lack of vivid case support, leading to low willingness among maintenance personnel to use it. Therefore, there is an urgent need for a multimodal knowledge extraction engine that can automatically capture, understand, and associate unstructured information from multiple channels, extract structured and executable maintenance knowledge from it, and integrate it into the knowledge base, so as to achieve the systematic accumulation and sustainable reuse of tacit experience. Summary of the Invention

[0004] This invention addresses the technical problems existing in the prior art by providing an intelligent metering equipment maintenance knowledge base management system. It solves the problems mentioned in the background art through a multimodal parsing module, a cross-modal alignment module, a joint reasoning module, and a knowledge graph completion module.

[0005] The technical solution of this invention to solve the above-mentioned technical problems is as follows: A smart metering equipment maintenance knowledge base management system, specifically comprising: The multimodal parsing module generates structured labels for images in the received raw data by calling a visual language model; it performs automatic speech recognition and large language model summarization to generate speech summaries and key elements for speech; and it performs named entity recognition and relation extraction to generate phenomenon-cause-measure triples for text; all of which are collectively referred to as primary structured representation. Cross-modal alignment module: Maps the primary structured representation to a shared semantic space via a pre-trained encoder, and performs density clustering on the generated semantic vectors to group them into the same event cluster; Joint Reasoning Module: For each event cluster, it invokes multimodal large model joint reasoning to generate structured knowledge entries containing fault phenomena, causes, measures, confidence levels, and evidence links, and marks candidate new knowledge; Knowledge graph completion module: It compares structured knowledge entries with pre-stored knowledge graphs, performs entity alignment, relationship completion and conflict resolution, and attaches the newly added structured knowledge entries and their evidence links to the knowledge graph nodes in the form of attributes. It takes effect after review.

[0006] In a preferred embodiment, the specific operation of generating structured labels for images by calling a visual language model in the multimodal parsing module is as follows: Receive raw data and output candidate structured labels and their confidence scores for the input image using a visual language model; Calculate the semantic saliency factor between each candidate structured label and the pre-built device domain ontology. The semantic saliency factor is defined as the normalized mutual information between the candidate structured label and the domain ontology. Multiply the confidence of the candidate structured label by the semantic saliency factor to obtain the comprehensive score. The top K candidate structured labels with the highest comprehensive scores are selected as the final structured labels, where K is a preset positive integer.

[0007] In a preferred embodiment, the specific operations for performing automatic speech recognition and large language model summarization to generate speech summaries and key elements are as follows: Automatic speech recognition converts audio into text, and large language models generate summaries from the text; For each word in the summary, a position-frequency weighting factor is calculated. The position-frequency weighting factor is obtained by combining the word's position score in the summary, the word's frequency of occurrence in the summary, and the inverse document frequency. The position score is assigned different values ​​according to the sentence position of the word, and the inverse document frequency is based on the statistics of the entire historical summary corpus. The top M words in the position-frequency weighted factor ranking are selected as key elements, where M is a preset positive integer; Structured tags, speech summaries and key elements, and phenomenon-cause-measure triples together constitute the primary structured representation, in which the phenomenon-cause-measure triples are generated by named entity recognition and relation extraction of the text.

[0008] In a preferred embodiment, the specific operation of mapping the primary structured representation to the shared semantic space via a pre-trained encoder in the cross-modal alignment module is as follows: Each structured label in the structured label set is encoded into a label vector by a pre-trained encoder, each word in the speech summary and key element list is encoded into a word vector, and each triple in the phenomenon-cause-measure triple set is encoded into a triple vector. Intramodal aggregation is performed on the structured label set, speech summary and key element list, and phenomenon-cause-measure triple set in the primary structured representation, and the mean of all encoded vectors in each modality is taken as the aggregation vector of the modality. Calculate the modality normalization factor for each modality aggregation vector. The modality normalization factor is the result of applying a logarithmic function with the natural constant e to the Euclidean norm of the modality aggregation vector plus one, and then dividing it by the sum of all modal logarithmic results. Multiply each modality aggregation vector by its corresponding modality normalization factor to obtain the normalized vector; The normalized vector is input into a shared linear projection layer. The parameters of the linear projection layer are obtained through multimodal contrastive learning pre-training, and the output is a projection vector.

[0009] In a preferred embodiment, the specific operation of performing density clustering on the generated semantic vectors to classify them into the same event cluster is as follows: Calculate the cross-modal semantic distance between the projection vector of each primary structured representation instance and the projection vectors of other instances. The cross-modal semantic distance is equal to one minus the cosine similarity between the two projection vectors, plus a penalty term. The penalty term is positive when the modality sets contained in the two instances are different, and zero otherwise. The modality set refers to the presence of three types of modal data in the instance: structured labels, speech summaries and key elements, and phenomenon-cause-measure triples. An adaptive density peak clustering algorithm is used to determine cluster centers. For each instance, a local density is calculated. The local density is equal to the sum of the squares of the cross-modal semantic distances of all other instances to this instance, which is a negative exponential function. The truncated distance is determined by the first local minimum point of the kernel density estimation curve of all distance values. For each instance, calculate the minimum distance, which is equal to the minimum distance to the instance among all instances with a density higher than that instance; Cluster centers are selected from instances whose product of local density and minimum distance is greater than the mean of the products of all instances plus twice the standard deviation. The remaining instances are assigned to the clusters of their nearest and densest neighbors, and each cluster is an event cluster.

[0010] In a preferred embodiment, the specific operation of invoking multimodal large model joint inference for each event cluster in the joint inference module is as follows: Extract the set of structured labels, speech summaries, and key element lists of all primary structured representation instances within the event cluster; each primary structured representation instance contains evidence links pointing to the original data; The information richness of each mode is calculated. The information richness is obtained by mapping the ratio of the number of elements in the mode to the number of elements in the maximum mode to the interval of zero to one using the logistic function. Calculate the arithmetic mean of the confidence scores of all primary structured representation instances within an event cluster on the corresponding modality, and use it as the modality confidence score; Combined with modal confidence, the complementarity weight of each modality is calculated. The complementarity weight is equal to the product of the modality's information richness and modality confidence, divided by the sum of all modality products. The original data of each modality are concatenated into a multimodal prompt text according to the complementary weight ratio; Multimodal prompt text is input into a large multimodal model, and semantic consistency constraints are introduced during the model generation process. The semantic consistency constraints are measured by calculating the average cosine similarity between the three fields of fault phenomenon, fault cause, and maintenance measures output by the model and the modal projection vectors of each instance in the event cluster. The consistency loss is added as a penalty term to the objective function of the model generation to ensure that the output is semantically consistent with the multimodal information in the cluster. After the model outputs, the fields of fault phenomenon, fault cause, and maintenance measures are parsed out. The average consistency score of all instances is used as the confidence level. At the same time, the evidence links of all instances within the event cluster are merged, and the fault phenomenon, fault cause, maintenance measures, confidence level, and merged evidence links are combined into structured knowledge items.

[0011] In a preferred embodiment, the specific operation of marking candidate new knowledge is as follows: The three fields of fault phenomenon, fault cause, and maintenance measures are parsed from the structured knowledge entries and mapped into semantic vectors through a pre-trained encoder. Calculate the maximum cosine similarity between the semantic vector of each field and the vectors of all entity nodes in the pre-stored knowledge graph; Define a semantic novelty measure, which is equal to the average of the maximum cosine similarity of each of the three fields minus one of the three fields. If the semantic novelty measure is greater than the preset novelty threshold, the entire structured knowledge entry will be marked as candidate new knowledge and a novelty score will be added. Otherwise, it is considered known knowledge and is only used to update the statistical information in the knowledge graph.

[0012] In a preferred embodiment, the specific operation for entity alignment in the knowledge graph completion module is as follows: Each entity is extracted from the fields of fault phenomenon, fault cause, and maintenance measures of the structured knowledge entries, and the entity is mapped into a semantic vector through a pre-trained encoder; Simultaneously, hyperbolic embedding representations of all entity nodes are extracted from the pre-stored knowledge graph, and the hyperbolic embedding representations are obtained by training a Poincaré sphere model. Calculate the hyperbolic distance between the semantic vector and each hyperbolic embedding representation. The hyperbolic distance is calculated based on the inverse hyperbolic cosine function and the Euclidean norm. Calculate the graph kernel structure similarity between entities and each knowledge graph node. The graph kernel structure similarity is calculated based on the structural similarity of local subgraphs using Wefler-Lehman subtree kernels. The alignment score is obtained by combining the hyperbolic distance with the graph kernel structure similarity. The alignment score is equal to the inverse of the hyperbolic distance multiplied by a negative structure penalty coefficient with the natural constant e as the base, multiplied by an exponential function of the graph kernel structure similarity. Knowledge graph nodes with alignment scores greater than the preset entity alignment threshold are selected as matching objects. If no match is found, a new node is created for the entity and its hyperbolic embedding is initialized.

[0013] In a preferred embodiment, the specific operation for performing relation completion is as follows: Extract relation triples from structured knowledge entries that have been aligned with entities. Relation triples include a head entity, a tail entity, and a relation type, where the head entity and tail entity are entities from the structured knowledge entries. The graph attention network is used to calculate the score of each relation triplet to be completed. The graph attention network aggregates the path information of all paths connected to the tail entity in the neighborhood of the head entity, and transforms the hyperbolic embedding of the head entity, the hyperbolic embedding of the tail entity, and the relation embedding through a learnable weight matrix. The score is then obtained through the attention vector and the non-linear activation function. If the score is greater than the preset relation completion threshold, the relation triple will be added to the knowledge graph as a candidate new edge. Otherwise, discard the relation triples and return the lower-scoring relations among the candidate new knowledge marked by the joint reasoning module to the manual review queue.

[0014] In a preferred embodiment, the specific operations for conflict resolution are as follows: When a relation in a structured knowledge entry conflicts with an existing relation in the knowledge graph, the credibility of the evidence for the structured knowledge entry is calculated. The credibility of the evidence is equal to the authority weight of all evidence links multiplied by a negative time decay coefficient with the natural constant e as the base, multiplied by the average of the exponential function of the difference between the time the evidence was generated and the current time. The Bayesian update formula is used to fuse the confidence of structured knowledge entries with the confidence of existing knowledge in the knowledge graph. The confidence of existing knowledge in the knowledge graph is the confidence attribute value of the relation stored in the knowledge graph. The Bayesian update formula divides the product of the confidence of structured knowledge entries and the credibility of evidence by the product of the confidence of structured knowledge entries and the credibility of evidence, adds one minus the product of the confidence of structured knowledge entries and the credibility of evidence as the fusion weight, and then multiplies it by the confidence of existing knowledge and the confidence of structured knowledge entries respectively, and sums them to obtain the updated confidence. The updated confidence level is used as the final confidence level of the relationship in the knowledge graph, and the conflict history is recorded. After entity alignment, relationship completion, and conflict resolution, newly added structured knowledge entries and their evidence links will be attached to knowledge graph nodes as attributes and will take effect after review.

[0015] The beneficial effects of this invention are as follows: By performing structured extraction of images, speech, and text, heterogeneous representations are mapped to a shared semantic space and clustered into event clusters. Then, a multimodal large model is invoked to generate structured knowledge entries containing fault phenomena, causes, measures, and evidence links. Finally, after performing entity alignment, relationship completion, and conflict resolution, the newly added knowledge is mounted to the graph nodes. This achieves the automated transformation from fragmented multimodal data to executable maintenance knowledge, effectively capturing and accumulating implicit front-line experience, and preventing knowledge loss due to personnel turnover. At the same time, the knowledge base content is continuously enriched, enabling maintenance personnel to easily obtain accurate diagnostic suggestions with evidence chains, significantly reducing the rate of repeated faults, shortening the growth cycle of new employees, and enhancing the practicality and vitality of the knowledge base. Attached Figure Description

[0016] Figure 1 This is a flowchart of the method of the present invention; Figure 2 This is a block diagram of the system structure of the present invention. Detailed Implementation

[0017] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0018] In the description of this application, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the stated features. In the description of this application, "multiple" means two or more, unless otherwise explicitly specified.

[0019] In the description of this application, the term "for example" is used to mean "used as an example, illustration, or description." Any embodiment described as "for example" in this application is not necessarily to be construed as being more preferred or advantageous than other embodiments. The following description is provided to enable any person skilled in the art to make and use the invention. Details are set forth in the following description for purposes of explanation. It should be understood that those skilled in the art will recognize that the invention can be made without using these specific details. In other instances, well-known structures and processes will not be described in detail to avoid obscuring the description of the invention with unnecessary detail. Therefore, the invention is not intended to be limited to the embodiments shown, but is consistent with the broadest scope of the principles and features disclosed in this application.

[0020] Example 1: This example provides the following... Figures 1-2 The intelligent metering equipment maintenance knowledge base management system shown here specifically includes: The multimodal parsing module generates structured labels for images in the received raw data by calling a visual language model; it performs automatic speech recognition and large language model summarization to generate speech summaries and key elements for speech; and it performs named entity recognition and relation extraction to generate phenomenon-cause-measure triples for text; all of which are collectively referred to as primary structured representation. Cross-modal alignment module: Maps the primary structured representation to a shared semantic space via a pre-trained encoder, and performs density clustering on the generated semantic vectors to group them into the same event cluster; Joint Reasoning Module: For each event cluster, it invokes multimodal large model joint reasoning to generate structured knowledge entries containing fault phenomena, causes, measures, confidence levels, and evidence links, and marks candidate new knowledge; Knowledge graph completion module: It compares structured knowledge entries with pre-stored knowledge graphs, performs entity alignment, relationship completion and conflict resolution, and attaches the newly added structured knowledge entries and their evidence links to the knowledge graph nodes in the form of attributes. It takes effect after review.

[0021] In this embodiment, it is specifically necessary to explain the following operation in the multimodal parsing module: The operation of generating structured labels from images using a visual language model is as follows: The system receives raw data and outputs candidate structured labels and their confidence scores for the input images using a visual language model. The visual language model is pre-trained, and its training process involves: collecting an image dataset containing device parts and defect types, labeling each image with a corresponding structured label; inputting the images into a visual encoder to extract visual features, and inputting the label text into a text encoder to extract text features; using a contrastive learning loss function to narrow the vector distance between matching image-text pairs and widen the vector distance between mismatched pairs; the training iterations are 500,000, the batch size is 256, the learning rate is 0.0001, and an adaptive moment estimation optimizer is used; after training, the model outputs the probability value of each candidate label as a confidence score for the input image. The semantic saliency factor between each candidate structured label and the pre-built device domain ontology is calculated. The semantic saliency factor is defined as the normalized mutual information between the candidate structured label and the domain ontology. The calculation process of normalized mutual information is as follows: the frequency of occurrence of each concept in the candidate structured label and the domain ontology in the historical maintenance records is counted to construct a co-occurrence matrix; the information entropy of the candidate label is calculated, which is equal to the sum of the logarithms of the probability of occurrence of the negative candidate label multiplied by the probability of occurrence of the candidate label; the mutual information between the candidate label and the domain ontology is calculated, which is equal to the probability of co-occurrence of the candidate label and the domain ontology concept multiplied by the logarithm of the probability of co-occurrence of the candidate label and the domain ontology concept multiplied by the product of the probability of occurrence of the candidate label and the probability of occurrence of the domain ontology concept, and the sum is applied to all concepts; the mutual information is divided by the information entropy of the candidate label to obtain the normalized mutual information; the pre-built device domain ontology is constructed by collecting professional terms from equipment manuals, maintenance guides, and fault case libraries, using ontology editing tools, and includes concepts such as equipment model, component name, defect type, measurement parameters, and their hierarchical and attribute relationships; The confidence score of the candidate structured labels is multiplied by the semantic saliency factor to obtain the overall score. The top K candidate structured labels with the highest comprehensive scores are selected as the final structured labels, where K is a preset positive integer; the preset value of K is five, that is, the top five candidate labels with the highest comprehensive scores are retained as the final structured labels to ensure sufficient information and avoid redundancy. The specific operations for performing automatic speech recognition and large language model summarization to generate speech summaries and key elements are as follows: Automatic speech recognition converts audio to text, and a large language model generates summaries from the text. Automatic speech recognition employs an end-to-end convolutional recurrent neural network model, with log-Melogram spectral features as input and character sequences as output. The training data consists of 100,000 hours of Chinese speech data, including different accents and noisy environments. Decoding utilizes a beam search algorithm with a beam width of 10 and language model weights of 0.5. The large language model employs a generative pre-trained converter architecture with 7 billion parameters. After pre-training on the Chinese corpus, it is fine-tuned using a summarization task dataset. During summarization generation, the maximum generation length is set to 128 tokens, the temperature parameter is 0.7, and the top k-sampling is 50. For each word in the abstract, a position-frequency weighted factor is calculated. This factor is a composite of the word's position score, its frequency of occurrence, and its inverse document frequency (IVF). The position score is assigned a different value based on the word's position within the sentence, and the IVF is based on statistics from the entire historical abstract corpus. The specific calculation process for the position-frequency weighted factor is as follows: First, determine the word's position within the sentence of the abstract. If the word appears in the first sentence, its position score is assigned as 1.0; if the word appears in the last sentence, its position score is assigned as 0.0. The position score is assigned a value of 0.8; if the word appears in other sentences, the position score is assigned a value of 0.5; next, the total number of times the word appears in the summary is counted as the word frequency; then, the inverse document frequency is calculated based on the full historical summary corpus. The inverse document frequency is equal to the logarithm of the quotient of the total number of documents divided by the number of documents containing the word, where the total number of documents is the total number of historical summaries, and the number of documents containing the word is the number of summaries in which the word has appeared; finally, the position score is incremented by one, multiplied by 0.5, multiplied by the logarithm of the word frequency plus one, and multiplied by the inverse document frequency to obtain the position-frequency weighting factor; Select the top M words by position-frequency weighting factor as key elements, where M is a preset positive integer; the preset value of M is ten, that is, retain the top ten words by weighting factor as key elements to ensure that the number of key elements is moderate and covers the main information. Structured tags, speech summaries and key elements, and phenomenon-cause-measure triples together constitute the primary structured representation. The phenomenon-cause-measure triples are generated by named entity recognition and relation extraction from the text. Named entity recognition and relation extraction are implemented using a pre-trained language model based on a bidirectional encoder-transformer architecture, which is fine-tuned on the labeled work order text corpus. The training corpus contains 100,000 work order records, each labeled with entity type (phenomenon, cause, measure) and the relationship between entities. During fine-tuning, the cross-entropy loss function is used, the learning rate is 0.00002, the batch size is 32, and the number of training epochs is 10. The model outputs entity boundaries and categories to the input text, and outputs the relationship type between entity pairs, ultimately forming the phenomenon-cause-measure triples.

[0022] In this embodiment, it is specifically necessary to explain the operation of mapping the primary structured representation to the shared semantic space via the pre-trained encoder in the cross-modal alignment module as follows: The pre-trained encoder encodes each structured label in the structured label set to obtain a label vector, each word in the speech summary and key element list to obtain a word vector, and each triple in the phenomenon-cause-measure triple set to obtain a triple vector. The pre-trained encoder adopts a bidirectional encoder model based on a converter architecture. Its training process is as follows: a large-scale general Chinese corpus is collected, including news, encyclopedia, technical documents, etc., totaling 100 billion tokens; a masked language model training task is used, randomly masking 15% of the tokens, and the model predicts the masked tokens; the batch size during training is 4096, the learning rate is 0.0001, cosine annealing learning rate scheduling is used, the optimizer is an adaptive moment estimation optimizer, and the training steps are one million steps; after training, the encoder maps any input text to a dense vector of fixed dimension, with a vector dimension of 768. Intramodal aggregation was performed on the structured tag set, speech summary and key element list, and phenomenon-cause-measure triple set in the primary structured representation. The mean of all encoded vectors within each modality was taken as the aggregation vector of the modality. The specific calculation process of intramodal aggregation is as follows: For the structured tag set, assuming it contains five structured tags, each tag is encoded to obtain a tag vector. These five tag vectors are summed element by element and divided by five to obtain the aggregation vector of the image modality. For the speech summary and key element list, assuming it contains ten key elements, the word vectors corresponding to each element are summed element by element and divided by ten to obtain the aggregation vector of the speech modality. For the phenomenon-cause-measure triple set, assuming it contains three triples, each triple concatenates the entity name and relation name into a string. The encoder obtains a triple vector, and these three vectors are summed element by element and divided by three to obtain the aggregation vector of the text modality. The modality normalization factor for each modal aggregation vector is calculated. The modality normalization factor is the result of applying a logarithmic function (base e) to the Euclidean norm of the modal aggregation vector plus one, then dividing by the sum of all modal logarithmic results. The calculation process is as follows: First, calculate the Euclidean norm of each modal aggregation vector by squaring each component, summing the results, and then taking the square root. Then, add one to this norm and take the logarithm (base e). Next, sum the logarithmic results of the three modes. Finally, divide the logarithmic result of each mode by the sum to obtain the normalization factor for that mode. This factor nonlinearly compresses the lengths of different modal vectors, giving shorter modes a greater relative weight and preventing a particular mode from dominating subsequent calculations due to excessively long vectors. Multiply each modality aggregation vector by its corresponding modality normalization factor to obtain the normalized vector; The normalized vectors are input into a shared linear projection layer. The parameters of the linear projection layer are obtained through multimodal contrastive learning pre-training, and the output is a projection vector. The training process of the linear projection layer is as follows: a large amount of multimodal pairing data is collected, and each sample contains image, speech, and text data describing the same equipment maintenance event; images are encoded into image vectors through a visual language model, speech is obtained into speech vectors through speech recognition and encoder, and text is obtained into text vectors through text encoder; these three vectors are input into the linear projection layer respectively to obtain projection vectors; a contrastive learning loss function is used to make the projection vectors of different modalities of the same event closer in space and the projection vectors of different events farther apart; the batch size during training is 128, the learning rate is 0.0001, the optimizer is an adaptive moment estimation optimizer, and the number of training rounds is 50; the weight matrix dimension of the projection layer is 768 rows and 768 columns, and the bias vector dimension is 768; after training, the projection layer maps the vectors of different modalities to a unified shared semantic space, in which semantically similar vectors are closer together; The specific operation of performing density clustering on the generated semantic vectors to group them into the same event cluster is as follows: Calculate the cross-modal semantic distance between the projected vector of each primary structured representation instance and the projected vectors of other instances. The cross-modal semantic distance is equal to one minus the cosine similarity between the two projected vectors, plus a penalty term. The penalty term is positive when the modality sets contained in the two instances are different, and zero otherwise. The modality set refers to the presence of three types of modal data in the instance: structured labels, speech summaries and key elements, and phenomenon-cause-measure triples. The criteria for determining the modality set are: for each primary structured representation instance, check its triples... The data is considered to be present and have a confidence level greater than 0.5. If some data is missing or its confidence level is less than or equal to 0.5, the modality is considered invalid. If the valid modality sets of two instances are completely identical, the penalty term is zero. If they are different, the penalty term is 0.3. The purpose of this penalty term is to prevent two semantically similar but modally different instances from being incorrectly clustered together due to modality missingness. The cosine similarity is calculated by calculating the dot product of the two projected vectors and then dividing by the product of the Euclidean norms of the two vectors. The result ranges from negative one to positive one. An adaptive density peak clustering algorithm is used to determine cluster centers. For each instance, a local density is calculated. The local density is equal to the sum of the negative exponential function of the squared cross-modal semantic distances between the instance and all other instances. The truncation distance is determined by the first local minimum point of the kernel density estimation curve of all distance values. The calculation process of the local density is as follows: for each instance, all other instances are traversed, the cross-modal semantic distance between them is calculated, the squared distance is negative, and then the exponent is calculated with the natural constant e as the base. All exponent results are added together to obtain the local density. The process of determining the cutoff distance is as follows: Sort all pairs of distance values ​​between instances from smallest to largest to generate a kernel density estimation curve. The kernel function is a Gaussian kernel, and the bandwidth is automatically estimated using Scott's rule. Find the first local minimum point of the curve, that is, the point where the derivative changes from negative to positive. The distance value corresponding to this point is the cutoff distance. The cutoff distance is used to control the decay rate of the Gaussian kernel function. Instances with a distance smaller than the cutoff distance contribute more to the local density, while instances with a distance greater than the cutoff distance contribute less rapidly. For each instance, calculate the minimum distance. The minimum distance is equal to the minimum distance between the instance and all instances with higher densities. The calculation process is as follows: For each instance, find all other instances with higher local densities than the instance, calculate the cross-modal semantic distance between these instances and the instance, and take the minimum value. If there are no other instances with higher densities than the instance, that is, the instance is the instance with the highest density, then set its minimum distance to the maximum value of the pairwise distances between all instances. Cluster centers are selected from instances whose product of local density and minimum distance is greater than the mean of all instances' products plus twice the standard deviation. The selection process for cluster centers is as follows: first, calculate the product of local density and minimum distance for each instance; then, calculate the mean and standard deviation of all instances' products; compare the product of each instance with the mean plus twice the standard deviation; if it is greater than this threshold, then mark the instance as a cluster center. This threshold ensures that only those points with high local density and large distances from other high-density instances can become cluster centers, thus automatically determining the number of clusters without the need for pre-specification. The remaining instances are assigned to the clusters of their nearest and densest neighbors, with each cluster being an event cluster. The instance assignment process is as follows: for each instance that is not a cluster center, among all instances with a higher density, find the instance with the smallest cross-modal semantic distance to that instance, and assign the cluster label of that instance to the current instance. After the assignment is completed, each cluster contains a cluster center and several ordinary instances. All instances in the cluster represent multimodal fragments describing the same maintenance event. The output of each event cluster includes a cluster identifier, the mean vector of the projection vectors of all instances in the cluster, and the original primary structured representation and modal confidence of each instance in the cluster. This information will be passed to the joint inference module for further processing.

[0023] In this embodiment, it is necessary to specifically explain the operation of calling the multimodal large model joint inference for each event cluster in the joint inference module as follows: Extract the set of structured labels, speech summaries and key element lists, and the set of phenomenon-cause-measure triples for all primary structured representation instances within the event cluster; each primary structured representation instance contains evidence links pointing to the original data; the evidence links are Uniform Resource Locators or internal identifiers pointing to the original image files, speech files and text work orders, used for subsequent tracing of the original data; The information richness of each modality is calculated. Information richness is obtained by mapping the ratio of the number of elements within a modality to the number of elements in the maximum modality using a logistic function, to the interval between zero and one. The specific calculation process for information richness is as follows: First, count the number of elements in each modality. For the image modality, the number of elements is the number of structured tags; for the speech modality, the number of elements is the number of key elements; and for the text modality, the number of elements is the number of phenomenon-cause-measure triples. Find the maximum number of elements among the three modalities. For each modality, calculate the ratio of its number of elements to the maximum value. Then, subtract 0.5 from this ratio, multiply by the kurtosis coefficient of 3.0, take the negative value, calculate the exponent with the natural constant e as the base, add one, and take the reciprocal to obtain the information richness. The closer this value is to one, the more abundant the information provided by the modality; the closer it is to zero, the less information is provided. The kurtosis coefficient of 3.0 is used to control the steepness of the mapping curve, ensuring that modalities with moderate richness obtain a suitable level of information richness. The modal confidence is calculated as the arithmetic mean of the confidence scores of all primary structured representation instances within an event cluster in the corresponding modality. The modal confidence is calculated as follows: for each modality, the confidence scores of all primary structured representation instances within the event cluster in that modality are collected, these confidence scores are added together, and then divided by the total number of instances to obtain the arithmetic mean as the modal confidence. For example, if an event cluster contains three instances, and their confidence scores in the image modality are 0.9, 0.8, and 0.7 respectively, then the confidence score of the image modality is 3.0 divided by 3, which equals 0.8. Based on modality confidence, the complementarity weight of each modality is calculated. The complementarity weight is equal to the product of the modality's information abundance and its confidence, divided by the sum of all modality products. The calculation process for the complementarity weight is as follows: first, multiply the information abundance of each modality by its confidence to obtain a weighted value for each modality; then, sum the weighted values ​​of the three modalities to obtain a total; finally, divide the weighted value of each modality by the total to obtain the complementarity weight of that modality. The sum of the complementarity weights is one, and a larger weight indicates that the modality should have a greater proportion in subsequent prompts. The raw data of each modality are concatenated into a multimodal prompt text according to the complementary weight ratio. The concatenation process of the multimodal prompt text is as follows: First, the raw data of each modality is converted into natural language descriptions; the description format of the image modality is "Image Labels: Label 1, Label 2, Label 3"; the description format of the speech modality is "Speech Summary: Summary Content; Key Elements: Element 1, Element 2, Element 3"; the description format of the text modality is "Work Order Record: Phenomenon 1, Cause 1, Measure 1; Phenomenon 2, Cause 2, Measure 2"; then, the level of detail of each modality description is adjusted according to the complementary weight: the modality with the highest weight retains the entire description, the modality with the lowest weight retains only the first three core pieces of information, and the modalities with intermediate weights retain half of the information; finally, a complete prompt string is concatenated in the order of "image-speech-text", and a task instruction is added at the beginning: "Please analyze the phenomenon, cause, and maintenance measures of the equipment failure based on the following multimodal information". Multimodal prompt text is input into a large multimodal model, and semantic consistency constraints are introduced during model generation. These constraints are measured by calculating the average cosine similarity between the model's output fields (fault phenomenon, fault cause, and maintenance measures) and the modal projection vectors of each instance within the event cluster. Consistency loss is added as a penalty term to the model's objective function to ensure semantic consistency between the output and the multimodal information within the cluster. The specific implementation of semantic consistency constraints is as follows: the large multimodal model employs a generative pre-trained converter architecture with 70 billion parameters. After pre-training on general text and image data, it is fine-tuned using multimodal question-and-answer data from the equipment maintenance domain. During model generation, in addition to the standard autoregressive loss, an extra consistency loss term is added. The calculation process for this loss term is as follows: For the fault phenomenon text currently generated by the model, it is mapped to a semantic vector through a pre-trained encoder; the image modal projection vector of each instance is extracted from the event cluster, and the cosine similarity between the fault phenomenon vector and each image projection vector is calculated, with the average value taken as the phenomenon consistency score; similarly, the average cosine similarity between the fault cause field and the speech modal projection vector of each instance is calculated, and the average cosine similarity between the maintenance measure field and the text modal projection vector of each instance is calculated; the three average values ​​are added together and divided by three to obtain the overall consistency score; the consistency loss is equal to one minus the overall consistency score; the final loss function is the generation loss plus the consistency loss multiplied by the consistency penalty coefficient of 0.5; the model optimizes this loss function through gradient descent, gradually adjusting the output during the generation process to keep the output semantically consistent with the multimodal information within the cluster; the calculation process for cosine similarity is as follows: calculate the dot product of two vectors, and then divide by the product of the Euclidean norms of the two vectors, with the result ranging from negative one to positive one; After the model outputs, it parses out the fields of fault phenomenon, fault cause, and maintenance measures. The average consistency score of all instances is used as the confidence level. At the same time, the evidence links of all instances within the event cluster are merged. The fault phenomenon, fault cause, maintenance measures, confidence level, and merged evidence links are combined into a structured knowledge entry. The format of the structured knowledge entry is: fault phenomenon field, fault cause field, maintenance measures field, confidence level value, and evidence link list. The confidence level is equal to the overall consistency score, with a value ranging from zero to one. The higher the value, the more consistent the output is with the information within the cluster. The evidence link list is the deduplicated and merged result of the evidence links of all instances within the event cluster. Each link includes the source type and address. The specific operation for marking candidate new knowledge is as follows: The fault phenomenon, fault cause, and maintenance measures are parsed from structured knowledge entries and mapped into semantic vectors by a pre-trained encoder. The pre-trained encoder uses a bidirectional encoder model based on a converter architecture. Its training process is as follows: a large-scale general Chinese corpus, including news, encyclopedias, and technical documents, totaling 100 billion tokens, is collected; a masked language model training task is used, randomly masking 15% of the tokens and having the model predict the masked tokens; the batch size during training is 4096, the learning rate is 0.0001, cosine annealing learning rate scheduling is used, the optimizer is an adaptive moment estimation optimizer, and the training steps are one million steps; after training, the encoder maps any input text into a dense vector of fixed dimension, with a vector dimension of 768; the three fields of fault phenomenon, fault cause, and maintenance measures are input into the encoder to obtain three semantic vectors. Calculate the maximum cosine similarity between the semantic vector of each field and all entity node vectors in the pre-stored knowledge graph. Each entity node in the knowledge graph is pre-mapped to a node vector using the same pre-trained encoder and stored in a vector database. For the semantic vector of each field, iterate through all entity node vectors in the knowledge graph, calculate the cosine similarity, and take the maximum value. The cosine similarity is calculated by taking the dot product of the two vectors and then dividing by the product of the Euclidean norms of the two vectors. A semantic novelty metric is defined as follows: the semantic novelty metric is equal to the average of the maximum cosine similarity scores of the three fields, each minus one. The calculation process of the semantic novelty metric is as follows: for the fault phenomenon field, calculate one minus its maximum cosine similarity to obtain the phenomenon novelty score; similarly, obtain the cause novelty score and the measure novelty score; add these three scores together and divide by three to obtain the semantic novelty metric; the metric ranges from zero to one, and the larger the value, the greater the difference between the output knowledge and the existing knowledge graph, and the more likely it is to be new knowledge; If the semantic novelty metric is greater than the preset novelty threshold, the entire structured knowledge entry is marked as a candidate new knowledge and a novelty score is added. The preset novelty threshold is 0.4. That is, when the semantic novelty metric is greater than 0.4, the knowledge entry is considered to contain content not recorded in the existing graph and is marked as a candidate new knowledge. The novelty score is the numerical value of the semantic novelty metric, which is used for subsequent sorting and review priority. The marked knowledge entry will be sent to the knowledge graph completion module for entity alignment and conflict resolution, and finally officially added to the knowledge graph after manual review. Otherwise, it is considered known knowledge and is only used to update the statistical information in the knowledge graph. If the semantic novelty measure is less than or equal to 0.4, the knowledge entry is considered to be highly similar to the knowledge in the existing graph. No new nodes or relationships are added. Only the statistical attributes such as the frequency of occurrence and the most recent update time of the corresponding nodes and relationships in the graph are updated for subsequent recommendation and trend analysis.

[0024] In this embodiment, it is specifically necessary to explain the specific operation of entity alignment in the knowledge graph completion module as follows: Each entity is extracted from the fields of fault phenomenon, fault cause, and maintenance measures of structured knowledge entries. The entities are then mapped to semantic vectors by a pre-trained encoder. The pre-trained encoder adopts a bidirectional encoder model based on a converter architecture. Its training process is as follows: a large-scale general Chinese corpus is collected, including news, encyclopedias, technical documents, etc., totaling 100 billion tokens. A masked language model training task is used, randomly masking 15% of the tokens and having the model predict the masked tokens. During training, the batch size is 4096, the learning rate is 0.0001, cosine annealing learning rate scheduling is used, the optimizer is an adaptive moment estimation optimizer, and the number of training steps is one million. After training, the encoder maps any input text to a dense vector of fixed dimension, with a vector dimension of 768. Simultaneously, hyperbolic embedding representations of all entity nodes are extracted from the pre-stored knowledge graph. These hyperbolic embedding representations are obtained through training a Poincaré sphere model. The training process of the Poincaré sphere model is as follows: all entities in the knowledge graph are initialized as random vectors within a unit sphere; for each positive sample triple (head entity, relation, tail entity), the energy function is defined as the distance between the head entity and the tail entity in hyperbolic space plus the relation offset; a negative sampling strategy is adopted to randomly replace the head entity or the tail entity to generate negative samples; the marginal ranking loss function is used for optimization, with the marginal value set to 1.0; the learning rate during training is 0.001, the batch size is 512, and the number of training rounds is 200; after training, each entity obtains a hyperbolic embedding vector with a dimension of 768, located within a Poincaré sphere with a radius of 1. The hyperbolic distance between the semantic vector and each hyperbolic embedding representation is calculated based on the inverse hyperbolic cosine function and the Euclidean norm. The specific calculation process of the hyperbolic distance is as follows: First, calculate the Euclidean norm of the semantic vector and the hyperbolic embedding vector, which is the square root of the sum of the squares of each component; then calculate the Euclidean norm of the difference between the semantic vector and the hyperbolic embedding vector; next, calculate the norm square of twice the vector difference in the numerator, and the denominator is the square of the norm of the semantic vector minus the square of the norm of the hyperbolic embedding vector minus the square of the norm of the hyperbolic embedding vector; divide the numerator by the denominator and add one, then take the inverse hyperbolic cosine function to obtain the hyperbolic distance. This distance measures semantic similarity in hyperbolic space and is sensitive to hierarchical structure; the smaller the distance, the closer the semantics. The graph kernel structure similarity between an entity and each knowledge graph node is calculated. The graph kernel structure similarity is based on the Wefeller-Lehman subtree kernel, which calculates the structural similarity of local subgraphs. The calculation process of the Wefeller-Lehman subtree kernel is as follows: For each candidate graph node corresponding to an entity, a subgraph within its two-hop radius is extracted. The subgraph contains the node itself, first-order neighbor nodes, second-order neighbor nodes, and the edges between them. Each node in the subgraph is assigned an initial label (node ​​type). Then, the neighbor labels of each node are iteratively aggregated, sorted, and compressed into new labels. The number of iterations is set to two. The multiset of labels after two iterations is used as the feature representation of the subgraph. The intersection size of the two subgraph label multisets is divided by the union size to obtain the graph kernel structure similarity. This similarity value ranges from zero to one; the larger the value, the more similar the two nodes are in their local topology. The alignment score is obtained by combining the hyperbolic distance and the graph kernel structural similarity. The alignment score is equal to the inverse of the hyperbolic distance multiplied by a negative structural penalty coefficient with the natural constant e as the base, multiplied by an exponential function minus the graph kernel structural similarity. The calculation process of the alignment score is as follows: First, add one to the hyperbolic distance and take the inverse as the semantic similarity term; then calculate the graph kernel structural similarity, multiply it by the structural penalty coefficient of 2.0, take the negative value, and calculate the exponent with the natural constant e as the base to obtain the structural penalty term; multiply the semantic similarity term and the structural penalty term to obtain the alignment score; the score ranges from zero to one, and the higher the score, the greater the probability of the entity matching the graph node; Knowledge graph nodes with alignment scores greater than a preset entity alignment threshold are selected as matching objects. If no match is found, a new node is created for the entity and its hyperbolic embedding is initialized. The entity alignment threshold is preset to 0.6, meaning that a match is considered successful only when the alignment score is greater than 0.6. If multiple nodes have alignment scores greater than 0.6, the node with the highest score is selected as the matching object. If the alignment scores of all nodes are less than or equal to 0.6, a new node is created for the entity, and its semantic vector is projected onto a Poincaré sphere (i.e., divided by the vector norm to make its norm less than one) as the initial hyperbolic embedding. At the same time, the new node is added to the entity set of the knowledge graph. The specific steps for performing relation completion are as follows: Relation triples are extracted from structured knowledge entries that have completed entity alignment. Each relation triple includes a head entity, a tail entity, and a relation type. The head and tail entities are derived from entities within the structured knowledge entries. The extraction process involves identifying semantic relationships between entities from the fault phenomenon, fault cause, and maintenance measures fields of the structured knowledge entries. For example, if the fault phenomenon field contains "voltage fluctuation" and the fault cause field contains "poor grounding," then the relation triple (voltage fluctuation, cause, poor grounding) is extracted. Relation types are predefined and include "cause," "measures taken," and "accompanying phenomena," with each relation type corresponding to a unique relation identifier. A graph attention network is used to calculate the score for each relation triplet to be completed. The graph attention network aggregates all path information connected to the tail entity within the head entity's neighborhood. The head entity hyperbolic embedding, tail entity hyperbolic embedding, and relation embedding are transformed using learnable weight matrices, and then the score is obtained through an attention vector and a non-linear activation function. The structure and training process of the graph attention network are as follows: the network contains two graph attention layers, each with eight attention heads; the inputs are the head entity hyperbolic embedding, tail entity hyperbolic embedding, and relation embedding, all with a dimension of 768; the first layer transforms the head entity embedding using a weight matrix, and the tail entity embedding using another weight matrix. In the first layer, the relation embedding is transformed by the third weight matrix, the three transformed vectors are added together, and then passed through a modified linear unit activation function with leakage, the leakage slope is set to 0.2; then a scalar score is calculated through the attention vector; the second layer repeats the same operation, but with a different weight matrix; during network training, positive and negative samples are used for comparative learning, positive samples are correct triples already existing in the knowledge graph, and negative samples are triples generated by randomly replacing the head or tail entity, the loss function is the marginal ranking loss, the marginal value is set to 0.5, the learning rate is 0.001, and the number of training rounds is 100; after training, the network outputs a real number score for any triple, the higher the score, the more reasonable the triple; If the score is greater than the preset relation completion threshold, the relation triple will be added to the knowledge graph as a candidate new edge. Otherwise, the relation triples are discarded, and the relations with lower scores among the candidate new knowledge marked by the joint reasoning module are returned to the manual review queue. The relation completion threshold is preset to 0.5, meaning that a relation is considered worth adding to the graph only if its score is greater than 0.5. For relations with scores less than or equal to 0.5, if the relation comes from candidate new knowledge marked by the joint reasoning module, it is placed in the manual review queue, and domain experts determine whether it should be added. If it is not candidate new knowledge, it is discarded directly. The specific steps for conflict resolution are as follows: When a relation in a structured knowledge entry conflicts with an existing relation in the knowledge graph, the credibility of the evidence for that knowledge entry is calculated. The credibility of the evidence is equal to the authority weight of all evidence links multiplied by a negative time decay coefficient base e multiplied by the average of an exponential function of the difference between the evidence generation time and the current time. The calculation process for the credibility of the evidence is as follows: First, a list of all evidence links in the structured knowledge entry is obtained. Each evidence link includes the source type (e.g., ticket system, instant messaging, email, etc.) and a timestamp. The system presets an authority weight for each source type. The weight of a single system is 0.9, instant messaging is 0.6, email is 0.7, and teleconference is 0.5. For each evidence link, the difference between the current time and the time the evidence was generated in days is calculated, multiplied by a time decay coefficient of 0.01, and the negative value is used to calculate the exponent with the natural constant e as the base, thus obtaining the timeliness factor. The authority weight is multiplied by the timeliness factor to obtain the credibility component of the evidence. The credibility components of all evidence are summed and divided by the total number of evidence links to obtain the evidence credibility. This value ranges from zero to one, comprehensively reflecting the authority and timeliness of the evidence. This paper utilizes a Bayesian update formula to fuse the confidence scores of structured knowledge entries with the confidence scores of existing knowledge in the knowledge graph. The confidence score of the existing knowledge in the knowledge graph is the confidence attribute value of that relation stored in the knowledge graph. The Bayesian update formula calculates the fusion weight by dividing the product of the confidence score of the structured knowledge entry and the confidence score of the evidence by the product of the confidence score of the structured knowledge entry and the confidence score of the evidence, and then adding the sum of the product of the confidence score of the structured knowledge entry and the confidence score of the evidence. This weight is then multiplied by the confidence scores of the existing knowledge and the structured knowledge entry, respectively, and summed to obtain the updated confidence score. The Bayesian update formula is calculated as follows: First, the numerator is calculated, which is the confidence score of the structured knowledge item multiplied by the confidence score of the evidence. Then, the denominator is calculated, which is the numerator plus one minus the confidence score of the structured knowledge item multiplied by one minus the confidence score of the evidence. The fusion weight is obtained by dividing the numerator by the denominator. The fusion weight is between zero and one. The updated confidence score is equal to the fusion weight multiplied by the original knowledge confidence score plus one minus the fusion weight multiplied by the confidence score of the structured knowledge item. This formula ensures that when the confidence score of the evidence is high and the confidence score of the new knowledge is high, the new knowledge dominates; otherwise, the old knowledge is retained. The updated confidence level is used as the final confidence level of the relationship in the knowledge graph, and the conflict history is recorded. The conflict history includes the time when the conflict occurred, the old and new knowledge identifiers involved in the conflict, their original confidence levels, the credibility of the evidence, the updated confidence level, and the handling method (automatic fusion or manual intervention). This record is used for subsequent auditing and model tuning. After entity alignment, relation completion, and conflict resolution, newly added structured knowledge entries and their evidence links are attached to knowledge graph nodes as attributes and become effective after review. Newly added structured knowledge entries include newly created entity nodes, newly added relation edges, and updated confidence attributes. Evidence links are stored as node attributes in a list format, with each link containing source type, timestamp, and access address. All new content first enters a pending review state, reviewed by system administrators or domain experts through a visual interface. Once approved, it becomes effective and triggers a knowledge graph index update. Content that fails review is marked as rejected, and the reason is recorded for subsequent analysis.

[0025] It should be noted that the descriptions of each embodiment in the above embodiments have different focuses. For parts that are not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0026] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0027] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0028] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0029] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0030] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended technical solutions are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the invention.

[0031] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the invention's technical solutions and their equivalents, the invention also intends to include these modifications and variations.

Claims

1. A knowledge base management system for the maintenance of intelligent metering equipment, characterized in that, Specifically, it includes: The multimodal parsing module generates structured labels for images in the received raw data by calling a visual language model; it performs automatic speech recognition and large language model summarization to generate speech summaries and key elements for speech; and it performs named entity recognition and relation extraction to generate phenomenon-cause-measure triples for text; all of which are collectively referred to as primary structured representation. Cross-modal alignment module: Maps the primary structured representation to a shared semantic space via a pre-trained encoder, and performs density clustering on the generated semantic vectors to group them into the same event cluster; Joint Reasoning Module: For each event cluster, it invokes multimodal large model joint reasoning to generate structured knowledge entries containing fault phenomena, causes, measures, confidence levels, and evidence links, and marks candidate new knowledge; Knowledge graph completion module: It compares structured knowledge entries with pre-stored knowledge graphs, performs entity alignment, relationship completion, and conflict resolution, and attaches the newly added structured knowledge entries and their evidence links to the knowledge graph nodes in the form of attributes. After review, they take effect.

2. The intelligent metering equipment maintenance knowledge base management system according to claim 1, characterized in that: In the multimodal parsing module, the specific operation of generating structured labels for images using a visual language model is as follows: Receive raw data and output candidate structured labels and their confidence scores for the input image using a visual language model; Calculate the semantic saliency factor between each candidate structured label and the pre-built device domain ontology. The semantic saliency factor is defined as the normalized mutual information between the candidate structured label and the domain ontology. Multiply the confidence of the candidate structured label by the semantic saliency factor to obtain the comprehensive score. The top K candidate structured labels with the highest comprehensive scores are selected as the final structured labels, where K is a preset positive integer.

3. The intelligent metering equipment maintenance knowledge base management system according to claim 2, characterized in that: The specific operations for performing automatic speech recognition and large language model summarization to generate speech summaries and key elements are as follows: Automatic speech recognition converts audio into text, and large language models generate summaries from the text; For each word in the summary, a position-frequency weighting factor is calculated. The position-frequency weighting factor is obtained by combining the word's position score in the summary, the word's frequency of occurrence in the summary, and the inverse document frequency. The position score is assigned different values ​​according to the sentence position of the word, and the inverse document frequency is based on the statistics of the entire historical summary corpus. The top M words in the position-frequency weighted factor ranking are selected as key elements, where M is a preset positive integer; Structured tags, speech summaries and key elements, and phenomenon-cause-measure triples together constitute the primary structured representation, in which the phenomenon-cause-measure triples are generated by named entity recognition and relation extraction of the text.

4. The intelligent metering equipment maintenance knowledge base management system according to claim 3, characterized in that: In the cross-modal alignment module, the specific operation of mapping the primary structured representation to the shared semantic space via the pre-trained encoder is as follows: Each structured label in the structured label set is encoded into a label vector by a pre-trained encoder, each word in the speech summary and key element list is encoded into a word vector, and each triple in the phenomenon-cause-measure triple set is encoded into a triple vector. Intramodal aggregation is performed on the structured label set, speech summary and key element list, and phenomenon-cause-measure triple set in the primary structured representation, and the mean of all encoded vectors in each modality is taken as the aggregation vector of the modality. Calculate the modality normalization factor for each modality aggregation vector. The modality normalization factor is the result of applying a logarithmic function with the natural constant e to the Euclidean norm of the modality aggregation vector plus one, and then dividing it by the sum of all modal logarithmic results. Multiply each modality aggregation vector by its corresponding modality normalization factor to obtain the normalized vector; The normalized vector is input into a shared linear projection layer. The parameters of the linear projection layer are obtained through multimodal contrastive learning pre-training, and the output is a projection vector.

5. The intelligent metering equipment maintenance knowledge base management system according to claim 4, characterized in that: The specific operation of performing density clustering on the generated semantic vectors to group them into the same event cluster is as follows: Calculate the cross-modal semantic distance between the projection vector of each primary structured representation instance and the projection vectors of other instances. The cross-modal semantic distance is equal to one minus the cosine similarity between the two projection vectors, plus a penalty term. The penalty term is positive when the modality sets contained in the two instances are different, and zero otherwise. The modality set refers to the presence of three types of modal data in the instance: structured labels, speech summaries and key elements, and phenomenon-cause-measure triples. An adaptive density peak clustering algorithm is used to determine cluster centers. For each instance, a local density is calculated. The local density is equal to the sum of the squares of the cross-modal semantic distances of all other instances to this instance, which is a negative exponential function. The truncated distance is determined by the first local minimum point of the kernel density estimation curve of all distance values. For each instance, calculate the minimum distance, which is equal to the minimum distance to the instance among all instances with a density higher than that instance; Cluster centers are selected from instances whose product of local density and minimum distance is greater than the mean of the products of all instances plus twice the standard deviation. The remaining instances are assigned to the clusters of their nearest and densest neighbors, and each cluster is an event cluster.

6. The intelligent metering equipment maintenance knowledge base management system according to claim 5, characterized in that: In the joint reasoning module, the specific operation of calling the multimodal large model joint reasoning for each event cluster is as follows: Extract the set of structured labels, speech summaries, and key element lists of all primary structured representation instances within the event cluster; each primary structured representation instance contains evidence links pointing to the original data; The information richness of each mode is calculated. The information richness is obtained by mapping the ratio of the number of elements in the mode to the number of elements in the maximum mode to the interval of zero to one using the logistic function. Calculate the arithmetic mean of the confidence scores of all primary structured representation instances within an event cluster on the corresponding modality, and use it as the modality confidence score; Combined with modal confidence, the complementarity weight of each modality is calculated. The complementarity weight is equal to the product of the modality's information richness and modality confidence, divided by the sum of all modality products. The original data of each modality are concatenated into a multimodal prompt text according to the complementary weight ratio; Multimodal prompt text is input into a large multimodal model, and semantic consistency constraints are introduced during the model generation process. The semantic consistency constraints are measured by calculating the average cosine similarity between the three fields of fault phenomenon, fault cause, and maintenance measures output by the model and the modal projection vectors of each instance in the event cluster. The consistency loss is added as a penalty term to the objective function of the model generation to ensure that the output is semantically consistent with the multimodal information in the cluster. After the model outputs, the fields of fault phenomenon, fault cause, and maintenance measures are parsed out. The average consistency score of all instances is used as the confidence level. At the same time, the evidence links of all instances within the event cluster are merged, and the fault phenomenon, fault cause, maintenance measures, confidence level, and merged evidence links are combined into structured knowledge items.

7. The intelligent metering equipment maintenance knowledge base management system according to claim 6, characterized in that: The specific operation for marking candidate new knowledge is as follows: The three fields of fault phenomenon, fault cause, and maintenance measures are parsed from the structured knowledge entries and mapped into semantic vectors through a pre-trained encoder. Calculate the maximum cosine similarity between the semantic vector of each field and the vectors of all entity nodes in the pre-stored knowledge graph; Define a semantic novelty measure, which is equal to the average of the maximum cosine similarity of each of the three fields minus one of the three fields. If the semantic novelty measure is greater than the preset novelty threshold, the entire structured knowledge entry will be marked as candidate new knowledge and a novelty score will be added. Otherwise, it is considered known knowledge and is only used to update the statistical information in the knowledge graph.

8. The intelligent metering equipment maintenance knowledge base management system according to claim 7, characterized in that: In the knowledge graph completion module, the specific operation for entity alignment is as follows: Each entity is extracted from the fields of fault phenomenon, fault cause, and maintenance measures of the structured knowledge entries, and the entity is mapped into a semantic vector through a pre-trained encoder; Simultaneously, hyperbolic embedding representations of all entity nodes are extracted from the pre-stored knowledge graph, and the hyperbolic embedding representations are obtained by training a Poincaré sphere model. Calculate the hyperbolic distance between the semantic vector and each hyperbolic embedding representation. The hyperbolic distance is calculated based on the inverse hyperbolic cosine function and the Euclidean norm. Calculate the graph kernel structure similarity between entities and each knowledge graph node. The graph kernel structure similarity is calculated based on the structural similarity of local subgraphs using Wefler-Lehman subtree kernels. The alignment score is obtained by combining the hyperbolic distance with the graph kernel structure similarity. The alignment score is equal to the inverse of the hyperbolic distance multiplied by a negative structure penalty coefficient with the natural constant e as the base, multiplied by an exponential function of the graph kernel structure similarity. Knowledge graph nodes with alignment scores greater than the preset entity alignment threshold are selected as matching objects. If no match is found, a new node is created for the entity and its hyperbolic embedding is initialized.

9. The intelligent metering equipment maintenance knowledge base management system according to claim 8, characterized in that: The specific steps for performing relation completion are as follows: Extract relation triples from structured knowledge entries that have been aligned with entities. Relation triples include a head entity, a tail entity, and a relation type, where the head entity and tail entity are entities from the structured knowledge entries. The graph attention network is used to calculate the score of each relation triplet to be completed. The graph attention network aggregates the path information of all paths connected to the tail entity in the neighborhood of the head entity, and transforms the hyperbolic embedding of the head entity, the hyperbolic embedding of the tail entity, and the relation embedding through a learnable weight matrix. The score is then obtained through the attention vector and the non-linear activation function. If the score is greater than the preset relation completion threshold, the relation triple will be added to the knowledge graph as a candidate new edge. Otherwise, discard the relation triples and return the lower-scoring relations among the candidate new knowledge marked by the joint reasoning module to the manual review queue.

10. The intelligent metering equipment maintenance knowledge base management system according to claim 9, characterized in that: The specific steps for conflict resolution are as follows: When a relation in a structured knowledge entry conflicts with an existing relation in the knowledge graph, the credibility of the evidence for the structured knowledge entry is calculated. The credibility of the evidence is equal to the authority weight of all evidence links multiplied by a negative time decay coefficient with the natural constant e as the base, multiplied by the average of the exponential function of the difference between the time the evidence was generated and the current time. The Bayesian update formula is used to fuse the confidence of structured knowledge entries with the confidence of existing knowledge in the knowledge graph. The confidence of existing knowledge in the knowledge graph is the confidence attribute value of the relation stored in the knowledge graph. The Bayesian update formula divides the product of the confidence of structured knowledge entries and the credibility of evidence by the product of the confidence of structured knowledge entries and the credibility of evidence, adds one minus the product of the confidence of structured knowledge entries and the credibility of evidence as the fusion weight, and then multiplies it by the confidence of existing knowledge and the confidence of structured knowledge entries respectively, and sums them to obtain the updated confidence. The updated confidence level is used as the final confidence level of the relationship in the knowledge graph, and the conflict history is recorded. After entity alignment, relationship completion, and conflict resolution, newly added structured knowledge entries and their evidence links will be attached to knowledge graph nodes as attributes and will take effect after review.

Images