An algorithm for mechanical and electrical operation and maintenance multi-modal named entity recognition enhanced by domain knowledge

Abstract

The application relates to the cross technical field of natural language processing and intelligent operation and maintenance of electromechanical equipment, in particular to a field knowledge enhanced electromechanical operation and maintenance multi-modal named entity recognition algorithm, which comprises the following steps: S1, field knowledge enhanced multi-modal representation learning, a term dictionary is constructed, term embedding is obtained by using Word2Vec, text features of a CLIP are adaptively fused through a gate fusion unit, and image area and global features are extracted by using ResNet-152 in combination with an open vocabulary detector; S2, cross-modal alignment and fusion guided by a graph, a unified multi-modal scene graph containing intra-text, intra-image and cross-modal edges is constructed; S3, sequence decoding of joint boundary optimization, entity candidate spans are enumerated, and a comprehensive feature vector containing boundaries and internal semantics is constructed, a feedforward network classification and a conditional random field joint scoring are combined, and an optimal label sequence is decoded by improving a Viterbi algorithm. The application effectively improves the recognition accuracy and robustness of entities such as equipment models and fault types in the electromechanical operation and maintenance scene.

Description

Technical Field

[0001] This invention relates to the interdisciplinary field of natural language processing and intelligent operation and maintenance of electromechanical equipment, and in particular to a domain knowledge-enhanced multimodal named entity recognition algorithm for electromechanical operation and maintenance. Background Technology

[0002] As core physical assets of modern industry and infrastructure, the stable operation of electromechanical equipment directly affects production safety and economic efficiency. Traditional operation and maintenance management relies heavily on manual experience and text records, making it difficult to systematically integrate multimodal information such as fault images and operational status graphs, resulting in low knowledge discovery efficiency and delayed fault diagnosis. Multimodal knowledge graphs, by structurally representing entities such as equipment, faults, and components and their relationships, have become an inevitable choice for improving operation and maintenance cognitive capabilities. Named entity recognition is the foundation and key link in building multimodal knowledge graphs, and its recognition quality directly determines the effectiveness of subsequent relation extraction and knowledge fusion. Therefore, there is an urgent need for high-performance multimodal named entity recognition algorithms adapted to the professional scenario requirements of electromechanical operation and maintenance.

[0003] Currently, the mainstream implementation of multimodal named entity recognition mainly relies on vision-language pre-trained (VLP) models combined with traditional sequence decoding frameworks. Among these, the most widely used approach is based on general-purpose pre-trained models such as CLIP and BLIP-2, combined with cross-modal fusion strategies such as Unified Multimodal Transformer (UMT), Multimodal Graph Fusion (UMGF), and Matching and Alignment Framework (MAF), and further combined with sequence decoding modules such as Conditional Random Fields (CRF). A commonly used approach in existing technologies is a conventional multimodal named entity recognition algorithm that combines a general-purpose VLP model with simple cross-modal fusion and traditional word-level sequence decoding.

[0004] The specific implementation process of existing conventional technologies is as follows: First, in the multimodal feature extraction stage, the system inputs text and image data related to electromechanical maintenance into a general pre-trained text encoder and image encoder, respectively. On the text side, BERT, RoBERTa, or CLIP text encoders are typically used for contextual feature extraction, outputting word-level feature vectors. On the image side, convolutional neural networks such as ResNet are often used to extract global features, or object detectors such as Faster R-CNN are used to extract region features, outputting visual feature vectors. This stage does not perform feature enhancement processing for electromechanical terminology.

[0005] Secondly, in the cross-modal feature fusion stage, existing technologies mostly employ methods such as linear concatenation, global cross-modal attention, or simple heterogeneous graph fusion to fuse text features with image features. For example, UMT uses a shared Transformer encoder to perform global attention interaction between text tokens and image region tokens, while UMGF constructs heterogeneous graphs using text phrases and image objects as nodes and aggregates information through GCN. This stage is mostly coarse-grained fusion, failing to achieve fine-grained semantic alignment between text entities and specific regions of the image, and lacks explicit visual noise filtering mechanisms.

[0006] Finally, in the entity recognition decoding stage, the detection head typically uses the traditional CRF framework to predict word-level labels on the fused feature sequence, outputting entity categories and boundary information. During training, a word-level cross-entropy loss function is used to calculate the deviation between the predicted labels and the ground truth labels, and the network parameters are updated through backpropagation. This stage relies solely on local context for boundary decisions, which can easily lead to boundary segmentation errors for long-span, complex-structure entities in electromechanical maintenance scenarios (such as permanent magnet synchronous motor encoders).

[0007] Although the aforementioned multimodal named entity recognition methods based on general pre-trained models have shown good recognition results in general image and text scenarios, they have extremely obvious systemic limitations and technical defects when applied to specialized industrial scenarios such as electromechanical maintenance. These limitations are manifested in the following four core aspects: 1) The general pre-trained model has insufficient ability to represent electromechanical terms. A large number of professional compound terms in the operation and maintenance text (such as inverter IGBT module and bearing raceway pitting) cannot be fully modeled, resulting in low accuracy of professional entity recognition. This is because the training data of the general pre-trained model has a significant distribution offset from the electromechanical operation and maintenance field and does not incorporate domain professional knowledge.

[0008] 2) Cross-modal fusion is a coarse-grained alignment, which does not achieve precise entity-level association between text entities and specific pixel regions / sub-components in the image, and lacks an adaptive noise filtering mechanism. When the correlation between text and images is weak, irrelevant visual noise is easily introduced, which will damage the entity recognition effect. Because the existing fusion strategy does not take into account the weak semantic correlation between text and images in electromechanical operation and maintenance scenarios, it does not dynamically adjust the fusion ratio of visual information.

[0009] 3) Traditional word-level sequence decoding frameworks are easily affected by local context. For entities with long spans, nested or complex structures in electromechanical operation and maintenance, the boundary decision-making ability is weak, resulting in incorrect entity boundary segmentation and poor integrity. This is because the framework does not perform global feature modeling of entity span, but only relies on the local features of a single word for label prediction.

[0010] 4) Poor adaptability to scenarios with scarce labeled data. Existing methods mostly rely on large-scale labeled data for fine-tuning, but high-quality graphic and text labeled data is difficult to obtain in the field of electromechanical operation and maintenance, resulting in insufficient generalization ability of the model. Because the lack of labeled data is not compensated by injecting domain knowledge, the structured knowledge in the electromechanical field cannot be fully utilized.

[0011] In response to a series of shortcomings exposed by existing technologies in multimodal named entity recognition for electromechanical maintenance, it is urgent to solve systemic technical problems such as insufficient domain terminology representation in general pre-trained models, easy introduction of noise in cross-modal coarse-grained alignment, poor robustness of long entity boundary recognition, and low adaptability of scarce labeled data. Summary of the Invention

[0012] The purpose of this invention is to solve systemic technical problems such as insufficient representation of domain terminology in general pre-trained models, easy introduction of noise in cross-modal coarse-grained alignment, poor robustness of long entity boundary recognition, and low adaptability of scarce labeled data.

[0013] To achieve the above objectives, the present invention adopts the following technical solution: a domain knowledge-enhanced electromechanical maintenance multimodal named entity recognition algorithm, comprising the following steps: S1. Domain Knowledge Enhanced Multimodal Representation Learning: A terminology dictionary for the electromechanical domain is constructed. The Word2Vec model is trained on pure text corpora in the electromechanical domain to obtain term embedding vectors. The CLIP text encoder is used to encode the operation and maintenance text to obtain general text features. The term embedding vectors and general text features are adaptively fused through a gated fusion unit to generate a knowledge-enhanced text feature matrix. At the same time, a ResNet-152 backbone network is used to extract high-level convolutional feature maps of the image, and an open vocabulary object detector is combined to extract salient region features related to electromechanical concepts in the image to generate image feature representations. S2. Graph-guided cross-modal alignment and fusion: Using each word in the knowledge-enhanced text features obtained in S1 and each detected object in the image features as nodes, a unified multimodal scene graph containing text internal edges, image internal edges, and cross-modal edges is constructed. A heterogeneous graph attention network is applied to the scene graph for information propagation and aggregation to obtain a visually enhanced text representation. Then, an adaptive gating module is used to generate a gating value for each text word to dynamically adjust the fusion ratio of visual information, resulting in an adaptively fused feature sequence. S3. Sequence Decoding with Joint Boundary Optimization: Enumerate all possible entity candidate spans for the fused feature sequence obtained in S2, and construct a comprehensive feature vector for each candidate span that includes left boundary features, right boundary features, boundary interaction features, internal semantic max pooling features, and internal semantic average pooling features; input the comprehensive feature vector into a feedforward network for span classification to obtain the score of the span belonging to each entity type; simultaneously, use a conditional random field framework to obtain the emission score and label transition score of each word, and design a joint scoring function to integrate the span scores into the sequence scoring of the conditional random field; decode using an improved Viterbi algorithm to obtain the label sequence that maximizes the joint score, and output the entity type and boundary information.

[0014] Preferably, the specific method for constructing the electromechanical terminology dictionary in S1 is as follows: by combining electromechanical equipment manuals, fault case databases and historical maintenance work orders, high-frequency word extraction, phrase mining and expert review are used to construct a core entity terminology dictionary D covering equipment models, component names and fault phenomena.

[0015] Preferably, the calculation formula for adaptive fusion of the gated fusion unit in S1 is: ; ; in, for The Middle Features of each word Embed the terms into linearly projected vectors. For the Sigmoid function, and These are learnable parameters. This indicates element-wise multiplication.

[0016] Preferably, the open vocabulary target detector in S1 adopts GroundingDINO, inputs prompt words related to electromechanical operation and maintenance, obtains m salient regions in the image related to electromechanical concepts and their bounding boxes, applies RoI-Align to the ResNet-152 feature map to extract region features, and then projects them to a unified dimension through a linear layer.

[0017] Preferably, the construction of the unified multimodal scene graph in S2 includes: Internal text edges: Using dependency parsing, if there is a direct dependency relationship between two words, an edge is established; Image internal edges: If the intersection-union ratio of two object bounding boxes is greater than the first threshold or the feature cosine similarity is greater than the second threshold, an edge is established, and the global context node is connected to all other image object nodes; Cross-modal edges: The CLIP projector is enhanced by freezing knowledge to map text nodes and image nodes to a common aligned semantic space. Cosine similarity is calculated, and edges are established when the image node with the highest similarity is retained for each text node and the similarity is greater than a third threshold.

[0018] Preferably, in S2, the attention coefficient calculation of the heterogeneous graph attention network distinguishes edge types. For node i, the update process is as follows: ; ; in, It is a node In the figure The set of neighbors in the middle, An embedded representation of the edge type. and These are learnable parameters. This indicates a concatenation operation; after GAT aggregation, the text nodes incorporate the sentence's internal syntactic and semantic information and relevant visual object features, generating a visually enhanced text representation. .

[0019] Preferably, the formula for calculating the gate value λi generated by the adaptive gating module for each text word in S2 is as follows: ; in, It is the original knowledge that enhances text features. It is the maximum similarity between the text word and the best-matching image node. The final fusion feature is: ; For visually enhanced features, Original, knowledge-enhanced text features.

[0020] Preferably, the comprehensive feature vector constructed for each candidate span (s,e) in S3 is: ; It is a left boundary feature; It is a right boundary feature; It is element-wise multiplication; It is an internal semantic feature (max pooling); It is an internal semantic feature (average pooling).

[0021] Preferably, the joint scoring function in S3 is: ; It is the set of start and end positions of all entity spans parsed from the label sequence Y. It belongs to each entity type The span of the score (including "non-entity" type O).

[0022] Preferably, the algorithm integrates three stages—domain knowledge-enhanced multimodal representation learning, graph-guided cross-modal alignment and fusion, and joint boundary optimization sequence decoding—in an end-to-end manner, forming a progressive architecture of "domain knowledge enhancement - graph-guided alignment and fusion - joint optimization decoding." This architecture is used to extract equipment models, fault types, and maintenance component entities from electromechanical operation and maintenance multimodal data, providing entity support for the automated construction of electromechanical operation and maintenance multimodal knowledge graphs.

[0023] Compared with the prior art, the advantages and positive effects of the present invention are as follows: This invention proposes a novel multimodal named entity recognition algorithm network architecture that integrates core technologies such as domain knowledge enhancement, cross-modal fine-grained alignment and adaptive noise filtering, and joint boundary optimization decoding. This architecture enables high-precision and robust recognition of multimodal entities in electromechanical maintenance scenarios, thereby fundamentally improving the overall performance of the multimodal named entity recognition model in electromechanical maintenance professional scenarios and providing reliable entity extraction support for the construction of multimodal knowledge graphs.

[0024] This architecture is specifically designed to address the challenges of low accuracy and poor robustness in multimodal named entity recognition caused by strong modal heterogeneity, dense domain terminology, weak semantic association between text and graph, and scarce labeled data in electromechanical operation and maintenance scenarios. It can accurately extract core entities such as equipment model, fault type, and maintenance parts from multimodal text and graph data in electromechanical operation and maintenance, providing high-quality entity support for the automated construction of multimodal knowledge graphs in electromechanical operation and maintenance, and ultimately improving the knowledge organization and fault diagnosis efficiency of intelligent operation and maintenance of electromechanical equipment.

[0025] First, it significantly enhances the representation capabilities of domain-specific entities, resolving the issue of general models being incompatible with local conditions. By constructing a domain terminology dictionary and employing a dual-path knowledge injection strategy, it integrates electromechanical expertise into multimodal feature representations, greatly improving the recognition accuracy of specialized entities such as equipment models and maintenance parts, while maintaining good generalization ability even in scenarios with scarce labeled data.

[0026] Second, it achieves cross-modal fine-grained alignment and adaptive noise filtering to improve the effectiveness of image-text fusion. Based on a heterogeneous graph attention network with a unified multimodal scene graph, it realizes accurate semantic association between text entities and specific regions of the image. The adaptive gating unit dynamically adjusts the visual information fusion ratio, effectively solving the problems of weak semantic association between images and text and strong visual noise interference in electromechanical operation and maintenance scenarios, and significantly reducing the noise introduction during the fusion process.

[0027] Third, it enhances the robustness of boundary recognition for long-span, complex entities, ensuring entity integrity. The joint boundary optimization decoding framework provides global feature basis for entity boundary decisions through explicit span feature modeling, breaking through the local context limitations of traditional word-level decoding and effectively solving the problem of boundary segmentation errors for long complex entities in electromechanical maintenance.

[0028] Fourth, it has excellent overall recognition performance and is suitable for the actual needs of electromechanical maintenance industry. It performs particularly well in visually related entities such as equipment models and maintenance parts, providing high-quality entity extraction support for the automated construction of multimodal knowledge graphs for electromechanical maintenance, and has good engineering application value. Attached Figure Description

[0029] Figure 1 The overall architecture diagram of the electromechanical maintenance multimodal named entity recognition algorithm for knowledge enhancement in the field of this invention; Figure 2 A diagram of the ResNet-152 network structure involved in the electromechanical maintenance multimodal named entity recognition algorithm for enhancing knowledge in the field of this invention; Figure 3 A diagram of the GAT network structure involved in the electromechanical maintenance multimodal named entity recognition algorithm for enhancing knowledge in this invention field; Figure 4 A CRF framework structure diagram related to the electromechanical operation and maintenance multimodal named entity recognition algorithm for enhancing knowledge in this invention field; Figure 5 This is a schematic diagram of a fault image input for an embodiment of the present invention. Detailed Implementation

[0030] This invention strengthens the representation capability of professional entities by constructing a terminology dictionary in the electromechanical field and designing a domain knowledge enhancement strategy with dual-path knowledge injection. Based on a unified multimodal scene graph, it combines a heterogeneous graph attention network and an adaptive gating unit to achieve cross-modal fine-grained alignment and dynamic filtering of visual noise. Furthermore, it improves the robustness of long entity recognition through a joint boundary decoding framework that combines span feature modeling and conditional random field co-optimization. Thus, it constructs a high-precision domain knowledge-enhanced multimodal named entity recognition algorithm network architecture for electromechanical operation and maintenance scenarios.

[0031] This invention relates to a domain knowledge-enhanced multimodal named entity recognition algorithm for electromechanical maintenance. The overall network architecture of this algorithm follows the classic paradigm of "multimodal feature extraction - cross-modal feature fusion - entity sequence decoding". It also incorporates three deep, core innovations at the underlying level to address the specific scientific problems of named entity recognition in electromechanical maintenance (weak domain terminology representation, coarse cross-modal alignment, visual noise interference, and blurred boundaries of long entities).

[0032] Figure 1This is a diagram illustrating the overall architecture of the present invention. The algorithm network architecture of the present invention mainly consists of three parts: a domain knowledge enhancement representation module, a graph-guided cross-modal alignment module, and a joint boundary optimization decoding module. The input electromechanical maintenance image-text pair (maintenance text description + fault scene image) first enters the domain knowledge enhancement representation module, which generates knowledge-enhanced text features H. t Image features H guided by open vocabulary v Subsequently, multi-level multimodal features are fed into a graph-guided cross-modal alignment module to construct a unified multimodal scene graph and achieve fine-grained semantic association and adaptive noise filtering, generating cross-modal fusion features. Finally, a joint boundary optimization decoding module performs entity candidate span modeling and label prediction, outputting accurate entity type and boundary information. To thoroughly solve the above technical challenges, this architecture implements the following three key improvement steps: a domain knowledge-enhanced multimodal representation learning strategy, a graph-guided cross-modal alignment and fusion mechanism, and a joint boundary optimization sequence decoding framework.

[0033] 1. The domain knowledge-enhanced representation module adopts a multimodal representation learning strategy that enhances domain knowledge.

[0034] To address the issues of insufficient representation of electromechanical terminology and scarce labeled data in general pre-trained models, this invention constructs an electromechanical terminology dictionary and designs a dual-path knowledge injection strategy during the multimodal feature extraction stage. At the same time, it adopts open vocabulary target detection to achieve domain adaptation of image features, generating an initial multimodal representation rich in domain knowledge from both text and image, thus completely solving the problem of weak semantic capture ability of professional terms.

[0035] 1) Domain terminology dictionary construction and embedding learning

[0036] By combining data sources such as mechanical and electrical equipment manuals, fault case databases, and historical maintenance work orders, and through high-frequency word extraction, phrase mining, and expert review, a domain terminology dictionary covering core entities such as equipment models, component names, and fault phenomena has been constructed. The Skip-gram model of Word2Vec was used for unsupervised training on tens of gigabytes of cleaned plain text corpus in the field of electromechanical operation and maintenance, to obtain the results for each term. Distributed embedding vectors It fully captures the co-occurrence semantics and functional associations of terms in operation and maintenance scenarios, making up for the lack of semantics in the general model domain.

[0037] 2) Knowledge-enhanced text representation

[0038] Text encoder using CLIP For operation and maintenance text Encode the text to obtain an initial context-aware representation. , where d is the feature dimension of CLIP. For each word in the text , ,judge Does it belong to a terminology dictionary? If so, then retrieve its corresponding Word2Vec embedding. And through a learnable linear projection layer Mapping it to the same space as the CLIP feature yields... Otherwise Let it be the zero vector.

[0039] The gated fusion unit is designed to adaptively fuse the general semantics of the CLIP text encoder with the domain-specific semantics of Word2Vec. The calculation formula is as follows: (1) (2) in, for The Middle Features of each word The semicolon (;) is the sigmoid function, and the semicolon (;) is the vector concatenation operator. This represents element-wise multiplication. and These are learnable parameters. It enhances text features. Gating value The fusion ratio of general representation and term embedding is automatically adjusted to ultimately form a knowledge-enhanced text feature matrix. .

[0040] 3) Image representation guided by open vocabulary

[0041] For fault scene images, this invention sets up a CLIP image encoding branch and a ResNet-152 feature extraction branch. First, the CLIP image encoder is used to encode the fault scene image as a whole, extracting global image features that can characterize the overall semantic content of the image. Since the CLIP image encoder and the aforementioned CLIP text encoder establish a correspondence between image semantics and text semantics through visual-linguistic contrastive learning, the CLIP image encoding result can provide a global semantic foundation for subsequent cross-modal semantic similarity calculation and common semantic space mapping between text nodes and image nodes. However, the CLIP image encoding result mainly reflects the overall semantics of the fault scene image, and the spatial location and texture information of fine-grained fault features such as bearing pitting, component ablation, surface cracks, and local deformation are still insufficient. Therefore, this invention further utilizes ResNet-152 and an open-vocabulary object detector to extract global contextual features and salient region features of the fault scene image to supplement the CLIP global image semantic features.

[0042] For images ( I (Referring to any image), using a ResNet-152 pre-trained on ImageNet as the backbone network to extract high-level convolutional feature maps. Global context features are obtained through global average pooling. An open-vocabulary object detector, GroundingDINO, is introduced. Inputting electromechanical maintenance-related prompts (such as motor, bearing, burnmark), it extracts objects from the image. A salient region and its bounding box related to the electromechanical concept. For each detected region... The RoI-Align operation is applied to the feature map to extract features of a fixed-size region, and then processed through a linear layer. Projected to dimension To obtain regional features Ultimately, the image-side features are represented as a set of nodes. Each feature vector has a dimension of 1. This approach enables the adaptation of image features to electromechanical maintenance scenarios. Specifically, the global image features output by the CLIP image encoder focus on describing the overall semantics of the fault scene image and provide a basis for cross-modal comparison with textual semantics; the high-level convolutional feature maps and global contextual features output by ResNet-152 focus on preserving the spatial structure and local fault textures in the image; GroundingDINO and RoI-Align are used to locate and extract salient region features related to electromechanical concepts. These features collectively participate in image-text alignment and information aggregation during subsequent common alignment semantic space mapping and unified multimodal scene graph construction, thus balancing cross-modal semantic consistency and the ability to express local fault details.

[0043] 2. The graph-guided cross-modal alignment module adopts a graph-guided cross-modal alignment and fusion mechanism.

[0044] To address the issues of weak semantic association between text and graphics in electromechanical maintenance scenarios, the ease with which general fusion methods introduce visual noise, and the coarse alignment granularity, this invention constructs a unified multimodal scene graph in the cross-modal fusion stage. It utilizes a heterogeneous graph attention network to achieve structured and fine-grained association between text semantics and visual objects, and introduces an adaptive gating fusion unit to dynamically filter visual noise, thereby achieving accurate cross-modal feature fusion.

[0045] 1) Unified multimodal scene graph construction

[0046] Text features enhanced with knowledge output from the gated fusion unit Image detection of object features (image features) Each element in the graph is treated as a node, and a node set is constructed. Edge construction is divided into three categories to ensure intramodal semantic association and cross-modal semantic alignment: Text internal edges: Using Stanford CoreNLP tools to perform dependency parsing on the text, if words... With words If direct dependencies exist (such as subject-verb relation in nsubj, verb-object relation in dobj), establish internal edges within the text. To capture the grammatical and semantic relationships between words.

[0047] Image interior edges: Connections are established based on spatial and semantic relationships. Spatially, if the object bounding boxes intersect (OI) and (IoU) are equal in length, then the connection is established. Then establish an edge, and in terms of semantic relationship, if the cosine similarity of the object node features... Then establish an edge; global node It connects to all other object nodes, passing on the global context.

[0048] Cross-modal edges: Enhance the CLIP projector with frozen knowledge, mapping all text and image nodes to a common aligned semantic space, and compute each pair of text nodes. With image nodes cosine similarity For each text node, retain the image node with the highest similarity, only when... Establish cross-modal edges This ensures alignment reliability.

[0049] Ultimately, a unified multimodal scene diagram is formed. ,in This provides a structured foundation for cross-modal semantic reasoning.

[0050] 2) Semantic aggregation of heterogeneous graph attention networks

[0051] In a unified multimodal scene graph A two-layer improved heterogeneous graph attention network (GAT) is applied for information propagation and aggregation. The GAT's attention coefficient calculation considers edge type information, distinguishing between intra-text / intra-image / cross-modal edges, and learning the contribution weights of different types of neighbors, enabling the model to differentiate the importance of intra-modal and cross-modal neighbors. For example... Figure 3 The GAT network structure diagram shown below illustrates the nodes. (characteristics) The update process is as follows: (3) (4) in, It is a node In the figure The set of neighbors in the middle, An embedded representation of the edge type. and These are learnable parameters. This indicates a concatenation operation. After GAT aggregation, the text nodes incorporate the sentence's internal syntactic and semantic information with relevant visual object features, generating a visually enhanced text representation. .

[0052] 3) Adaptive Gated Fusion Unit

[0053] To address the issue that not all text words require the same level of visual assistance, an adaptive gating module is designed to generate a gating value for each text word. It precisely controls the fusion ratio of visual information and filters visual noise. For the first... For each text word, first calculate its relevant visual context. That is, the average of the features of all image nodes connected to the word through cross-modal edges in the common semantic space; if there are no cross-modal edges, then... The zero vector is then used. The text features, visual context, their interaction, and alignment confidence are then input into a small feedforward network to calculate the gating value. (5) in, It is the original knowledge that enhances text features. It is the maximum similarity between the text word and the best-matching image node. parameter, parameter, It is the average of the features of all image nodes connected to the word through cross-modal edges in the common semantic space. , It is the Sigmoid function. The final fusion feature is: (6) These are features enhanced by visual enhancement.

[0054] When text words have high-confidence visual counterparts Approaching 1, the model relies on visual enhancement features; when text words are abstract or unrelated to the image, Approaching 0, the model reverts to the original text features, effectively avoiding visual noise interference, and finally obtains the adaptively fused feature sequence. .

[0055] 3. A joint boundary optimization sequence decoding framework

[0056] To address the issue of poor robustness in identifying long-span, complex structural entities in electromechanical maintenance, this invention introduces a boundary optimization module in the entity decoding stage. Through explicit entity span feature modeling and CRF collaborative optimization, it overcomes the limitations of traditional word-level decoding and improves the recognition accuracy of complex entities.

[0057] 1) Enumeration of entity span and construction of features

[0058] Enumerate all possible entity candidate spans in the text ,in and span length A comprehensive feature vector is constructed for each candidate span, containing five key pieces of information: left boundary features. Right boundary features Boundary interaction features Internal semantic max pooling features Internal semantic average pooling features By combining the above features, a comprehensive span representation is obtained. This comprehensively characterizes the boundary and internal properties of candidate entities, as shown in the following formula: (7)

[0059] 2) Span classification and joint scoring

[0060] Will The input is classified using a feedforward network (FFN), with the span classifier employing an FFN to determine which entity type the span belongs to. (Including "non-entity" type O) score: (8) At the same time, the standard CRF framework is adopted to fuse features. The emission score of each word belonging to each tag is obtained through a linear layer. and define label transfer score Design a joint scoring function that integrates span scores into the sequence scoring of the CRF for candidate label sequences. Its score is: (9) in, For label sequence This function sets the start and end positions of all parsed entity spans. It awards an extra reward to the label sequences corresponding to high-scoring candidate spans, improving the accuracy of complex entity recognition.

[0061] 3) Training and Decoding

[0062] During the training phase, the method of maximizing the correct label sequence is adopted. Joint log-likelihood: (10) in It is the set of all possible label sequences. During the decoding phase, an improved Viterbi algorithm is used, adding an extra dimension during state transitions to record whether the current entity is inside the entity and its starting position. This determines whether the current path constitutes a high-scoring entity span. During dynamic programming, the entity's starting position and span score are recorded to find the path that results in a joint score. highest label sequence Output entity type and boundary information.

[0063] This joint decoding framework ensures the global consistency of the label sequence through CRF, and the boundary optimization module provides global span feature evidence for the boundary recognition of long entities and complex entities through explicit span classification, thereby achieving accurate extraction of complex entities.

[0064] This invention aims to achieve high-precision multimodal named entity recognition in electromechanical maintenance scenarios. Its various technical modules are interconnected and indispensable. The key aspects mainly cover the following three areas: 1) Domain knowledge enhancement representation based on terminology dictionary and gating fusion: The key to this step is how to construct a high-quality mechatronics domain terminology dictionary and adaptively fuse domain terminology embeddings with general pre-trained features through learnable gating fusion units. At the same time, open vocabulary object detection is used to achieve domain adaptation of image features, making up for the lack of domain semantics in general models.

[0065] 2) Cross-modal fine-grained fusion based on unified scene graph and adaptive gating: The key to this step is how to construct a unified multimodal scene graph that includes text, image and cross-modal edges, realize multi-hop semantic reasoning and fine-grained alignment through heterogeneous graph attention network, and design an adaptive gating unit based on image-text alignment confidence to dynamically filter irrelevant visual noise and achieve accurate cross-modal feature fusion.

[0066] 3) Joint boundary optimization decoding based on span feature modeling and CRF collaboration: The key to this step is how to enumerate candidate entity spans and construct comprehensive span features that include boundaries and internal semantics. A joint scoring function is designed to integrate the span classification score with the CRF sequence score. By improving the Viterbi algorithm, global boundary decision-making for long entities and composite entities is achieved, thereby improving the completeness and accuracy of entity recognition.

[0067] This invention presents a dual-path knowledge-enhanced multimodal representation method for the electromechanical field. It constructs an electromechanical terminology dictionary and learns domain terminology embeddings using Word2Vec. A gated fusion unit adaptively fuses general pre-trained text features and terminology embedding features. Simultaneously, it combines ResNet-152 and the open-vocabulary object detector GroundingDINO to extract global and regional image features adapted to the electromechanical scene. Based on a fine-grained alignment and adaptive noise filtering fusion mechanism of a unified multimodal scene graph, it uses knowledge-enhanced text words and image detection objects as nodes to construct a unified scene graph containing intra-text, intra-image, and cross-modal edges. A heterogeneous graph attention network achieves cross-modal multi-hop semantic aggregation, and an adaptive gated unit based on graph-text alignment confidence is introduced to dynamically adjust the visual information fusion ratio. For long entity recognition, a joint boundary optimization decoding framework combining span feature modeling and CRF collaboration is proposed. This framework enumerates candidate spans of text entities and constructs comprehensive span features that fuse boundaries and internal semantics. A feedforward network is used for span classification, and a joint scoring function integrates span scores and CRF emission / transition scores. An improved Viterbi algorithm is used to achieve globally optimal decoding of entity sequences. The end-to-end domain knowledge-enhanced electromechanical maintenance multimodal named entity recognition algorithm architecture integrates the above-mentioned representation, fusion, and decoding technology modules to form a three-stage progressive end-to-end algorithm of "domain knowledge enhancement - graph-guided alignment fusion - joint optimization decoding", which realizes high-precision extraction of core entities in electromechanical maintenance graphic and textual multimodal data.

[0068] One alternative to domain knowledge augmentation is Domain Adaptive Pre-training (DAPT), which uses a general language model to perform secondary pre-training on electromechanical domain corpora. Both methods aim to improve the model's semantic representation ability of electromechanical domain terms, compensating for the distributional offset between the general model and the domain context. However, these methods require large-scale pure text corpora in the electromechanical domain, and the pre-training process is computationally expensive and time-consuming. They cannot achieve lightweight knowledge injection like the terminology dictionary + gating fusion strategy used in this approach, and they do not perform domain augmentation on image features.

[0069] An alternative to fine-grained cross-modal alignment is to use hierarchical visual prefixes (HVPNet) to inject visual prefixes into each layer of the text encoder to achieve cross-modal alignment. Both methods aim to achieve fine-grained semantic associations between text and images, improving the accuracy of cross-modal fusion. While this method achieves multi-granularity alignment through hierarchical injection of visual information, it does not construct a structured scene graph, fails to capture the semantic relationships between text and image objects, and lacks an explicit visual noise filtering mechanism, making it prone to introducing noise when the text and image are weakly correlated.

[0070] An alternative to joint boundary optimization decoding is to use PointerNetwork to directly predict the start and end positions of entities. Both aim to solve the problem of long entity boundary recognition in traditional word-level decoding by explicitly modeling the entity span to improve boundary decision accuracy. However, PointerNetwork can only predict the start and end positions of entities and cannot globally model the internal semantic features of the entity span. Furthermore, it is not combined with CRF, making it difficult to guarantee the global consistency of the label sequence, and its ability to recognize nested and overlapping entities is insufficient.

[0071] Explanation of technical terms: Multimodal Named Entity Recognition (MNER): This technology automatically extracts predefined types of entities (such as device names and fault types) from multimodal data such as text and images. It is an interdisciplinary research direction of natural language processing and computer vision, and provides a foundation for the construction of multimodal knowledge graphs.

[0072] Domain knowledge enhancement: A technique that injects structured / unstructured knowledge of a specific domain (such as terminology dictionaries and fault maps) into the model's feature representation or training process to improve the model's understanding and modeling ability of domain-specific content.

[0073] Vision-Language Pre-training (VLP): This method involves pre-training on large-scale image-text datasets to learn how to map images and text to a unified semantic space, thus alleviating modal heterogeneity and the semantic gap. Typical models include CLIP and BLIP-2.

[0074] Heterogeneous Graph Attention Network (HGAT): An attention network designed for heterogeneous graphs with diverse node / edge types. It can distinguish the contribution weights of different types of nodes / edges, enabling accurate information aggregation and semantic reasoning on heterogeneous graphs.

[0075] Adaptive Gating Unit: A neural network module that dynamically adjusts the fusion ratio of different information sources based on data features. It controls the contribution of each information source by learning a gating value, thereby achieving noise filtering and adaptive fusion of information.

[0076] Conditional Random Field (CRF): A probability-based sequence labeling model that can optimize the global consistency of label sequences by leveraging the label dependencies of context. It is widely used in sequence labeling tasks such as named entity recognition.

[0077] Entity span: The range formed by the start and end positions of a named entity in the text, representing the boundary range of the entity in the text. Long-span entities refer to composite entities composed of multiple consecutive words.

[0078] Open vocabulary object detection: This is a target detection technique that breaks through the limitations of predefined categories in traditional object detection and can detect corresponding conceptual objects in images based on input text prompts. A typical model is GroundingDINO.

[0079] Example: A real-world case study using "operation logs + fault images" demonstrates the process from overall input to intermediate outputs of each module, and finally to the final output. The confidence scores, gating values, and similarity scores in the table are example values ​​used to illustrate the algorithm flow.

[0080] The core input of this algorithm is a set of electromechanical maintenance image and text pairs, and the core output is the entity boundary, entity type and recognition confidence in the text.

[0081] Table 1: Description of Algorithm Core Inputs, Core Outputs, and Auxiliary Inputs / Optional Outputs

[0082] 1. Enter maintenance logs

[0083] Abnormal vibration occurred during the operation of the permanent magnet synchronous motor in Pump Station No. 3. On-site disassembly and inspection revealed pitting corrosion in the raceway of the drive end bearing. It is recommended to replace the drive end bearing and check the IGBT module of the frequency converter.

[0084] 2. Input a fault image

[0085] The image shows photos taken during on-site disassembly and inspection. The images reveal pitting spots on the motor housing, drive-end bearing, and bearing raceway, along with unrelated background areas such as oil stains and repair tools. Figure 5 This is a diagram of the fault image input.

[0086] 3. Auxiliary Input Resources

[0087] Table 2: Auxiliary Input Resources and Their Functions

[0088] The overall process of the implementation can be summarized as follows: overall input → domain knowledge enhanced representation module → graph-guided cross-modal alignment and fusion module → joint boundary optimization decoding module → final entity output.

[0089] Table 3: Description of Inputs and Intermediate Outputs at Each Node of the Overall Process

[0090] 1. Domain Knowledge Enhancement Representation Module: Multimodal Representation Learning for Domain Knowledge Enhancement

[0091] This module is tasked with processing text and images separately. On the text side, it enhances the representation of specialized entities through a terminology dictionary and gating fusion; on the image side, it locates image regions related to electromechanical maintenance through an open vocabulary detector.

[0092] Table 4: Intermediate results of the domain knowledge enhancement representation module on the text side (examples of term matching and gating values)

[0093] Table 5: Intermediate results of the domain knowledge augmentation representation module on the image side (example of region detection and confidence).

[0094] The output of the domain knowledge augmentation representation module can be abbreviated as: knowledge-enhanced text features Htext ={H(Permanent Magnet Synchronous Motor), H(Abnormal Vibration), H(Drive End Bearing), H(Raceway Pitting), H(Inverter IGBT Module), ...}; Knowledge-Enhanced Text Feature Matrix The image region features Fimg={V0,V1,V2,V3,V4} are represented by a set of nodes on the image side. .

[0095] 2. Graph-guided cross-modal alignment module: Graph-guided cross-modal alignment and fusion

[0096] This module unifies text fragments and image regions into multimodal scene graph nodes and establishes text-internal edges, image-internal edges, and cross-modal edges. Information is then aggregated through a heterogeneous graph attention network, and adaptive gating controls whether visual information participates in the fusion process.

[0097] Table 6: Examples of constructing internal edges of unified multimodal scene graphs and internal edges of images

[0098] Table 7: Examples of Cross-Modal Edge Construction and Graph-Text Similarity in Unified Multimodal Scene Graph

[0099] Table 8: Examples of Adaptive Gated Fusion Results from Graph-Guided Cross-Modal Alignment Module

[0100] The output of the graph-guided cross-modal alignment module can be simplified as: fused text features Z = {Z(Permanent Magnet Synchronous Motor), Z(Abnormal Vibration), Z(Drive End Bearing), Z(Range Pitting), Z(Inverter IGBT Module), ...}. Among them, Z(Drive End Bearing) and Z(Range Pitting) have strong visual enhancement; Z(Abnormal Vibration) and Z(Inverter IGBT Module) basically do not introduce visual information.

[0101] 3. Joint Boundary Optimization Decoding Module: Sequence decoding with joint boundary optimization.

[0102] This module enumerates candidate entity spans, calculates the score for each span belonging to different entity categories, and then combines it with CRF sequence scores to select results with complete boundaries and reasonable label sequences.

[0103] Table 9: Examples of Candidate Span Classification and Decoding Judgment in the Joint Boundary Optimization Decoding Module

[0104] The output of the joint boundary optimization decoding module can be simplified as: optimal label sequence + complete entity span set. For example, the entire "permanent magnet synchronous motor" is labeled as an equipment entity, the entire "drive end bearing" is labeled as a component entity, and the entire "inverter IGBT module" is labeled as a component entity.

[0105] 4. The final core output is a list of structured entities. In the table below, "boundaries" are represented by fragment numbers; in engineering systems, these can be replaced with character-level start and end positions or token-level start and end positions.

[0106] Table 10: Example of the final core output structured entity list

[0107] In this case, the algorithm takes an electromechanical maintenance log and a picture of a bearing failure as input. First, it enhances the representation of technical terms such as "permanent magnet synchronous motor," "drive-end bearing," "raceway pitting," and "inverter IGBT module" using a domain terminology dictionary. Then, it detects the motor housing, bearing, and pitting area in the image, constructs a unified text-image scene graph, and performs fine-grained alignment. For "drive-end bearing" and "raceway pitting" with clearly corresponding areas in the image, the model fuses visual information; for "abnormal vibration" and "inverter IGBT module" without direct evidence in the image, the model reduces the visual fusion ratio to avoid visual noise interference. Finally, through span modeling and CRF joint decoding, it outputs a structured list of entities with complete boundaries.

Claims

1. A domain knowledge-enhanced multimodal named entity recognition algorithm for electromechanical operation and maintenance, characterized in that: Includes the following steps: S1. Domain Knowledge Enhanced Multimodal Representation Learning: A terminology dictionary for the electromechanical domain is constructed. The Word2Vec model is trained on pure text corpora in the electromechanical domain to obtain term embedding vectors. The CLIP text encoder is used to encode the operation and maintenance text to obtain general text features. The term embedding vectors and general text features are adaptively fused through a gated fusion unit to generate a knowledge-enhanced text feature matrix. At the same time, a ResNet-152 backbone network is used to extract high-level convolutional feature maps of the image, and an open vocabulary object detector is combined to extract salient region features related to electromechanical concepts in the image to generate image feature representations. S2. Graph-guided cross-modal alignment and fusion: Using each word in the knowledge-enhanced text features and each detected object in the image features obtained in S1 as nodes, a unified multimodal scene graph containing text internal edges, image internal edges, and cross-modal edges is constructed; a heterogeneous graph attention network is applied to the scene graph for information propagation and aggregation to obtain a visually enhanced text representation. Then, through the adaptive gating module, a gating value is generated for each text word to dynamically adjust the fusion ratio of visual information, resulting in an adaptively fused feature sequence. S3. Sequence Decoding with Joint Boundary Optimization: Enumerate all possible entity candidate spans for the fused feature sequence obtained in S2, and construct a comprehensive feature vector for each candidate span that includes left boundary features, right boundary features, boundary interaction features, internal semantic max pooling features, and internal semantic average pooling features; input the comprehensive feature vector into the feedforward network for span classification to obtain the score of the span belonging to each entity type; at the same time, use the Conditional Random Field framework to obtain the emission score and label transition score of each word, and design a joint scoring function to integrate the span scores into the sequence scoring of the Conditional Random Field; The improved Viterbi algorithm is used to decode the label sequence that results in the highest joint score, and the entity type and boundary information are output.

2. The domain knowledge-enhanced electromechanical maintenance multimodal named entity recognition algorithm according to claim 1, characterized in that: The specific method for constructing the electromechanical terminology dictionary in S1 is as follows: by combining electromechanical equipment manuals, fault case databases and historical maintenance work orders, high-frequency word extraction, phrase mining and expert review are used to construct a core entity terminology dictionary D covering equipment models, component names and fault phenomena.

3. The domain knowledge-enhanced electromechanical maintenance multimodal named entity recognition algorithm according to claim 1, characterized in that: The calculation formula for adaptive fusion of the gated fusion unit in S1 is as follows: ； ； in, for The Middle Features of each word Embed the terms into linearly projected vectors. For the Sigmoid function, and These are learnable parameters. This indicates element-wise multiplication.

4. The domain knowledge-enhanced electromechanical maintenance multimodal named entity recognition algorithm according to claim 1, characterized in that: The open vocabulary target detector in S1 uses GroundingDINO. It takes input prompt words related to electromechanical operation and maintenance, obtains m salient regions in the image related to electromechanical concepts and their bounding boxes, applies RoI-Align to the ResNet-152 feature map to extract region features, and then projects them to a unified dimension through a linear layer.

5. The domain knowledge-enhanced electromechanical maintenance multimodal named entity recognition algorithm according to claim 1, characterized in that: The construction of the unified multimodal scene graph in S2 includes: Internal text edges: Using dependency parsing, if there is a direct dependency relationship between two words, an edge is established; Image internal edges: If the intersection-union ratio of two object bounding boxes is greater than the first threshold or the feature cosine similarity is greater than the second threshold, an edge is established, and the global context node is connected to all other image object nodes; Cross-modal edges: The CLIP projector is enhanced by freezing knowledge to map text nodes and image nodes to a common aligned semantic space. Cosine similarity is calculated, and edges are established when the image node with the highest similarity is retained for each text node and the similarity is greater than a third threshold.

6. The domain knowledge-enhanced electromechanical maintenance multimodal named entity recognition algorithm according to claim 1, characterized in that: The attention coefficient calculation in the heterogeneous graph attention network in S2 distinguishes edge types. For node i, the update process is as follows: ； ； in, It is a node In the figure The set of neighbors in the middle, An embedded representation of the edge type. and These are learnable parameters. This indicates a concatenation operation; after GAT aggregation, the text nodes incorporate the sentence's internal syntactic and semantic information and relevant visual object features, generating a visually enhanced text representation. .

7. The domain knowledge-enhanced electromechanical maintenance multimodal named entity recognition algorithm according to claim 1, characterized in that: The formula for calculating the gate value λi for each text word generated by the adaptive gating module in S2 is as follows: ； in, It is the original knowledge that enhances text features. It is the maximum similarity between the text word and the best-matching image node. The final fusion feature is: ； For visually enhanced features, Original, knowledge-enhanced text features.

8. The domain knowledge-enhanced electromechanical maintenance multimodal named entity recognition algorithm according to claim 1, characterized in that: The comprehensive feature vector constructed for each candidate span (s,e) in S3 is as follows: ； It is a left boundary feature; It is a right boundary feature; It is element-wise multiplication; It is an internal semantic feature (max pooling); It is an internal semantic feature (average pooling).

9. The domain knowledge-enhanced electromechanical maintenance multimodal named entity recognition algorithm according to claim 1, characterized in that: The joint scoring function in S3 is: ； It is the set of start and end positions of all entity spans parsed from the label sequence Y. It belongs to each entity type The span of the score (including "non-entity" type O).

10. The domain knowledge-enhanced electromechanical maintenance multimodal named entity recognition algorithm according to claim 1, characterized in that: The algorithm integrates three stages—domain knowledge enhancement multimodal representation learning, graph-guided cross-modal alignment and fusion, and joint boundary optimization sequence decoding—in an end-to-end manner, forming a progressive architecture of "domain knowledge enhancement - graph-guided alignment and fusion - joint optimization decoding." This architecture is used to extract equipment models, fault types, and maintenance component entities from electromechanical operation and maintenance multimodal data, providing entity support for the automated construction of electromechanical operation and maintenance multimodal knowledge graphs.

Images