Power equipment multi-modal named entity recognition method, system, device and medium

Abstract

The present application belongs to the field of multi-modal named entity recognition, and particularly relates to a power equipment multi-modal named entity recognition method, system, device and medium. The present application adopts a co-occurrence entity method to obtain entities appearing in both text and images, which can ensure text and image matching and reduce image noise. By extracting entity triples and then adopting an entity auxiliary method, semantic information can be supplemented and semantic disambiguation can be reduced. The present application adopts the co-occurrence entity and entity auxiliary method of extracting triples, which helps to improve the multi-modal named entity recognition accuracy, accurately recognize the entities existing in the image or text, and especially performs better in data with higher image-text correlation. Meanwhile, the present application is not limited to the field of power equipment, and can be constructed into a knowledge base of other fields to expand to other fields for multi-modal named entity recognition.

Description

Technical Field

[0001] This invention belongs to the field of multimodal named entity recognition, specifically relating to methods, systems, equipment, and media for multimodal named entity recognition of power equipment. Background Technology

[0002] With the convergence of information technology and human production and life, and the rapid popularization of the internet, global data is experiencing explosive growth and massive aggregation, containing immense value. Named entity recognition (NER) refers to identifying entities in text that have specific meaning or strong referentiality, typically including names of people, places, organizations, dates, and times. It is a fundamental task in the field of natural language processing, playing a crucial role in text search, text recommendation, and knowledge graph construction. However, due to insufficient text information and semantic discrepancies, it is difficult to accurately identify certain entity categories.

[0003] Multimodal named entity recognition (MNER) incorporates image information on top of traditional named entity recognition, supplementing text with semantic information for disambiguation, and has become an important research direction in named entity recognition. However, traditional MNER methods tend to directly concatenate text and image information into the named entity recognition neural network model, resulting in problems such as entity redundancy and high noise, leading to low named entity recognition accuracy.

[0004] Current multimodal named entity recognition methods mainly address two issues: 1) how to extract regions of interest (ROIs) from images for text. Text and images are not always perfectly matched, as seen in images used in social media posts like WeChat Moments and Weibo; 2) how to effectively fuse image and text features to improve model performance. Most multimodal named entity recognition models employ a two-stream approach, first processing a single modality and then using an attention mechanism to facilitate interaction between text and images. However, representations of different modalities originate from different encoders, making it difficult to capture the relationship between text tokens and image regions. This leads to problems such as fine-grained image region-text mismatches, redundancy within the single modality itself, and difficulty in establishing relationships between modalities. Furthermore, deep learning-based single-modal processing relies on pre-trained models with open-source code, and the difference in domain knowledge between pre-training and fine-tuning prevents satisfactory results in domain-knowledge-driven tasks. Summary of the Invention

[0005] The purpose of this invention is to overcome the above-mentioned shortcomings and provide a method, system, device and medium for multimodal named entity recognition of power equipment. It uses co-presented entities to indicate the related parts of text and images, avoiding the problem of image region-text mismatch, while suppressing noise introduced by irrelevant images. It adopts a domain knowledge base, which can solve the problem of domain knowledge difference by using entities in the knowledge base, thereby improving the accuracy of multimodal named entity recognition.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] In a first aspect, the present invention provides a method for multimodal named entity recognition of power equipment, comprising:

[0008] Collect text data from power equipment and corresponding image data, and combine the collected text data and image data into a set;

[0009] Natural language processing tools are used to extract text entities from the collection;

[0010] Image entities are extracted from the set using an image entity retrieval tool, and the image entities are input into a self-attention model to obtain image feature vectors;

[0011] Extract entities that appear in both text entities and image entities as co-present entities;

[0012] Extract the triples of the co-existing entities, combine the triples with the text data to reconstruct the word feature vectors needed in the text, and use natural language processing tools to process the word feature vectors to obtain the text feature vectors.

[0013] Text feature vectors and image feature vectors are concatenated to form a semantic vector of image-text fusion. A multi-head attention mechanism is used to capture the correlation between each word in the semantic vector and each region block in the image. The word order is adjusted according to the correlation to obtain the multimodal named entity recognition result.

[0014] A further improvement of the present invention is that the power equipment includes a transformer and an instrument transformer.

[0015] A further improvement of this invention is that the natural language processing tool adopts the Bert-BiLSTM-CRF model.

[0016] A further improvement of this invention lies in the following working method of the Bert-BiLSTM-CRF model:

[0017] The dataset is input into the Bert-BiLSTM-CRF model, which parses the dataset into three embedding features, including text data. The three embedding features are then added together to obtain the parameter E. N , will parameter E NInputting the pre-trained model BERT, we obtain the final hidden state T of the pre-trained model BERT. N The final hidden state T N Inputting the data into the BiLSTM-CRF model yields the text entities contained in the text data set.

[0018] A further improvement of this invention is that the image entity retrieval tool uses the Mask RCNN model.

[0019] A further improvement of this invention lies in the following specific method for extracting the triples of co-existing entities:

[0020] The co-existing entity is matched with entities in the knowledge base, and the triples of the matched entities are used as the triples of the co-existing entity.

[0021] A further improvement of the present invention is that the triple includes a head entity, a relation, and a tail entity.

[0022] A further improvement of this invention lies in combining triples with text data to reconstruct the word feature vectors required in the text, specifically including:

[0023] By combining triples with text data, a pre-trained K-Bert model for sentence input is formed;

[0024] The pre-trained K-Bert model injects relevant triple data into the received sentence, expanding the sentence into a sentence tree containing knowledge;

[0025] The sentence tree is decomposed to form word feature vectors. A further improvement of this invention lies in adjusting the vectors based on the correlation between words to obtain multimodal named entity recognition results, specifically including:

[0026] The correlation between each word in the semantic vector and the relationship between each region block in the image are input into the Bert-BiLSTM-CRF model, and the Bert-BiLSTM-CRF model outputs multimodal named entity recognition results.

[0027] In a second aspect, the present invention provides a multimodal named entity recognition system for power equipment, comprising:

[0028] The data set generation module is used to collect text data from power equipment and corresponding image data, and to combine the collected text data and image data into a set.

[0029] The text entity extraction module is used to extract text entities from a collection using natural language processing tools.

[0030] The image entity extraction module is used to extract image entities from the set using an image entity retrieval tool, input the image entities into the self-attention model, and obtain the image feature vector;

[0031] The co-occurrence entity extraction module is used to extract entities that appear in both text entities and image entities as co-occurrence entities;

[0032] The text feature vector generation module is used to extract triples of co-existing entities, combine the triples with text data to reconstruct the word feature vectors required in the text, and use natural language processing tools to process the word feature vectors to obtain the text feature vectors.

[0033] The word relationship adjustment module is used to concatenate text feature vectors and image feature vectors to form a semantic vector of image-text fusion. It adopts a multi-head attention mechanism to capture the correlation between each word in the semantic vector and each region block in the image, and adjusts the word order according to the correlation to complete multimodal named entity recognition.

[0034] A further improvement of this invention is that the text entity extraction module adopts the Bert-BiLSTM-CRF module; and the image entity retrieval tool adopts the Mask RCNN module.

[0035] Thirdly, the present invention provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of a method for multimodal named entity recognition of power equipment.

[0036] Fourthly, the present invention provides a storage medium storing a computer program thereon, wherein the computer program, when executed by a processor, implements the steps of a method for recognizing multimodal named entities of power equipment.

[0037] Compared with the prior art, the present invention has the following beneficial effects:

[0038] This invention employs a co-present entity method to acquire entities that appear in both text and images, ensuring text-image matching while reducing image noise. By extracting entity triples and then using an entity-assisted method, semantic information is supplemented while reducing semantic disambiguation. This invention utilizes co-present entities and triple extraction to improve the accuracy of multimodal named entity recognition, accurately identifying entities present in images or text, especially performing better with data where the relationship between text and images is high. Furthermore, this invention is not limited to power equipment; it can be used to build knowledge bases in other domains to extend multimodal named entity recognition. Attached Figure Description

[0039] Figure 1 This is a flowchart of the present invention;

[0040] Figure 2 This is a model framework diagram of Example 1;

[0041] Figure 3 This is a construction diagram of the real entity in Example 1;

[0042] Figure 4 This is a diagram of the Bert-BiLSTM-CRF framework in Example 1;

[0043] Figure 5 This is a system diagram of Example 2;

[0044] Figure 6 The data image is from Example 3;

[0045] Figure 7 This is a system diagram for Example 4. Detailed Implementation

[0046] The invention will now be further described with reference to the accompanying drawings.

[0047] See Figure 1 A method for multimodal named entity recognition of power equipment includes the following steps:

[0048] S1 collects text data from power equipment and corresponding image data, and combines the collected text data and image data into a set.

[0049] S2 uses natural language processing tools to extract text entities from a set.

[0050] S3 uses an image entity retrieval tool to extract image entities from the set, and inputs the image entities into the self-attention model to obtain image feature vectors.

[0051] S4, extract entities that appear in both text entities and image entities as co-present entities.

[0052] S5. Extract the triples of the co-existing entities, combine the triples with the text data to reconstruct the word feature vectors needed in the text, and use natural language processing tools to process the word feature vectors to obtain the text feature vectors.

[0053] S6 concatenates the text feature vector and the image feature vector to form a semantic vector of image-text fusion. A multi-head attention mechanism is used to capture the correlation between each word in the semantic vector and each region block in the image. The word order is adjusted according to the correlation to obtain the multimodal named entity recognition result.

[0054] Example 1:

[0055] See Figure 2A method for multimodal named entity recognition of power equipment includes the following steps:

[0056] S1, collect text data from power equipment and corresponding image data, and combine the collected text data and image data into a set;

[0057] S2 uses the Bert-BiLSTM-CRF model, a natural language processing tool, to extract text entities from the set.

[0058] The Bert-BiLSTM-CRF model works as follows:

[0059] See Figure 4 The set is input into the Bert-BiLSTM-CRF model, which parses the set into three embedding features, including text data: Token Embeddings, Segment Embeddings, and Position Embeddings. The three embedding features are then added together to obtain the parameter E. N N = 1, 2, 3, ..., N, The parameter E N Inputting the pre-trained model BERT, we obtain the final hidden state T of the pre-trained model BERT. N The final hidden state T N The input is fed into the BiLSTM-CRF model to obtain the text entities contained in the text data in the set. Figure 4 In the middle, P N It refers to the output vector of a bidirectional LSTM, Tag N This refers to the final entity label.

[0060] S3, the Mask RCNN model, an image entity retrieval tool, is used to extract image entities from the set. The image entities are then input into the self-attention model to obtain the image feature vector V1.

[0061] The Mask R-CNN model can detect covered entities and distinguish different individuals in an image, and further output the corresponding entity and entity category. The Mask R-CNN model adds a prediction segmentation mask branch to the Faster R-CNN model, which is used for multi-object detection and can determine the location and category of each object in the image.

[0062] S4, see S4 Figure 3 Extract entities that appear in both text entities and image entities as co-present entities, and store entities that appear in only one modality into the entity set TV_e;

[0063] S5, match the co-existing entity with the entity in the knowledge base, and use the triples of the matched entity as the triples of the co-existing entity. Combine the triples with the text data to reconstruct the word feature vectors required in the text. Use natural language processing tools to process the word feature vectors to obtain the text feature vector T2.

[0064] The specific method for constructing word feature vectors is as follows:

[0065] Retrieving the triple containing the target entity from the knowledge base means using an exact search method to find triples where the target entity is the head entity; an exact search means matching the target entity with entities in the knowledge base; the target entity as the head entity means the triple is in the form of a head entity, a relation, and a tail entity;

[0066] The pre-trained K-Bert model injects relevant triple data into the received sentence. For the triple data input to the model, the pre-trained K-Bert model first injects relevant triple data into the knowledge layer and expands it into a sentence tree containing knowledge. Then, this sentence tree is input into the embedding layer and the visibility layer to generate the token embedding and visible matrix, respectively. Finally, the feature vector of each word is obtained, and the feature vector of each word has a dimension of 768.

[0067] S6. The text feature vector T2 and the image feature vector V1 are concatenated to form a semantic vector of image-text fusion. A multi-head attention mechanism is used to capture the correlation between each word in the semantic vector and each region block in the image. The word order is adjusted according to the correlation to obtain the multimodal named entity recognition result.

[0068] Specifically, the correlation between each word in the semantic vector and the relationship between each region block in the image are input into the Bert-BiLSTM-CRF model, and the Bert-BiLSTM-CRF model outputs multimodal named entity recognition results.

[0069] Preferably, the power equipment includes transformers and instrument transformers.

[0070] Example 2:

[0071] See Figure 5 A multimodal named entity recognition system for power equipment, including:

[0072] The data set generation module is used to collect text data from power equipment and corresponding image data, and to combine the collected text data and image data into a set.

[0073] The text entity extraction module, using the Bert-BiLSTM-CRF module, is used to extract text entities from the set using natural language processing tools;

[0074] The image entity extraction module uses the Mask RCNN module to extract image entities from the set using an image entity retrieval tool. The image entities are then input into the attention model to obtain image feature vectors.

[0075] The co-occurrence entity extraction module is used to identify entities that appear in both text entities and image entities as co-occurrence entities.

[0076] The text feature vector generation module is used to extract triples of co-existing entities, combine the triples with text data to reconstruct the word feature vectors required in the text, and use natural language processing tools to process the word feature vectors to obtain the text feature vectors.

[0077] The word relationship adjustment module is used to concatenate text feature vectors and image feature vectors to form a semantic vector of image-text fusion. It adopts a multi-head attention mechanism to capture the correlation between each word in the semantic vector and each region block in the image, and adjusts the word order according to the correlation to complete multimodal named entity recognition.

[0078] Example 3:

[0079] The text data is: Zhang San and Li Si are installing a transformer while wearing professional clothing.

[0080] Image data corresponding to text data, such as Figure 6 As shown.

[0081] First, the Bert-BiLSTM-CRF model is used to extract text entities from the text data: Zhang San (person), Li Si (person), transformer (equipment); then, the Mask RCNN model is used to identify image entities contained in the image: two people (person), transformer (equipment), two poles (facilities).

[0082] Next, the co-existing entities are selected as: people and transformers; the entity set TV_e is: poles and towers;

[0083] By searching the knowledge base, we found transformer-related triples: <transformer, belongs to, power grid equipment>. Combining these triples with the text data, we reconstructed the word feature vectors needed in the text.

[0084]

[0085] The word feature vectors are input into the pre-trained model BERT to obtain the text feature vector T2, and the Mask RCNN is used to output the image feature vector V1.

[0086] Text feature vector T2 and image feature vector V1 are concatenated to form a semantic vector for image-text fusion. The semantic vector of image-text fusion is then input into the BERT model multi-head attention mechanism to fully fuse the image and text. The fused data is then input into the CRF and adjusted according to the relationship between words to obtain the multimodal named entity recognition result.

[0087] Example 4:

[0088] Please see Figure 7 As shown, the present invention also provides an electronic device 100 for implementing a multimodal named entity recognition method for power equipment; the electronic device 100 includes a memory 101, at least one processor 102, a computer program 103 stored in the memory 101 and executable on the at least one processor 102, and at least one communication bus 104.

[0089] The memory 101 can be used to store the computer program 103. The processor 102 implements the steps of the power equipment multimodal named entity recognition method described in Embodiment 1 by running or executing the computer program stored in the memory 101 and calling the data stored in the memory 101. The memory 101 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device 100 (such as audio data), etc. In addition, the memory 101 may include non-volatile memory, such as hard disk, memory, plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, at least one disk storage device, flash memory device, or other non-volatile solid-state storage device.

[0090] The at least one processor 102 may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The processor 102 may be a microprocessor or any conventional processor. The processor 102 is the control center of the electronic device 100, connecting various parts of the electronic device 100 via various interfaces and lines.

[0091] The memory 101 in the electronic device 100 stores multiple instructions to implement a multimodal named entity recognition method for power equipment, and the processor 102 can execute the multiple instructions to achieve the following:

[0092] Collect text data from power equipment and corresponding image data, and combine the collected text data and image data into a set;

[0093] Natural language processing tools are used to extract text entities from the collection;

[0094] Image entities are extracted from the set using an image entity retrieval tool, and the image entities are input into a self-attention model to obtain image feature vectors;

[0095] Extract entities that appear in both text entities and image entities as co-present entities;

[0096] Extract the triples of the co-existing entities, combine the triples with the text data to reconstruct the word feature vectors needed in the text, and use natural language processing tools to process the word feature vectors to obtain the text feature vectors.

[0097] Text feature vectors and image feature vectors are concatenated to form a semantic vector of image-text fusion. A multi-head attention mechanism is used to capture the correlation between each word in the semantic vector and each region block in the image. The word order is adjusted according to the correlation to obtain the multimodal named entity recognition result.

[0098] Example 5:

[0099] If the modules / units integrated in the electronic device 100 are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, and a read-only memory (ROM).

[0100] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0101] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0102] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0103] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0104] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A method for multimodal named entity recognition of power equipment, characterized in that, include: Collect text data from power equipment and corresponding image data, and combine the collected text data and image data into a set; Natural language processing tools are used to extract text entities from the collection; An image entity retrieval tool is used to extract image entities from the set, and the image entities are input into an attention model to obtain image feature vectors; the image entity retrieval tool uses the Mask RCNN model. Extract entities that appear in both text entities and image entities as co-present entities; The triples of co-existing entities are extracted, and these triples are combined with text data to reconstruct the word feature vectors needed in the text. Natural language processing (NLP) tools are then used to process these word feature vectors to obtain the text feature vectors. The NLP tool used is the Bert-BiLSTM-CRF model. The specific steps involved in reconstructing the word feature vectors by combining the triples with the text data include: By combining triples with text data, a pre-trained K-Bert model for sentence input is formed; The pre-trained K-Bert model injects relevant triple data into the received sentence, expanding the sentence into a sentence tree containing knowledge; The sentence tree is decomposed into word feature vectors; Text feature vectors and image feature vectors are concatenated to form a semantic vector of image-text fusion. A multi-head attention mechanism is used to capture the correlation between each word in the semantic vector and each region block in the image. The word order is adjusted according to the correlation to obtain the multimodal named entity recognition result.

2. The method for multimodal named entity recognition of power equipment according to claim 1, characterized in that, Electrical equipment includes transformers and instrument transformers.

3. The method for multimodal named entity recognition of power equipment according to claim 1, characterized in that, The Bert-BiLSTM-CRF model works as follows: The dataset is input into the Bert-BiLSTM-CRF model, which parses the dataset into three embedding features, including text data. The three embedding features are then added together to obtain the parameter E. N , will parameter E N Inputting the pre-trained model BERT, we obtain the final hidden state T of the pre-trained model BERT. N The final hidden state T N The input is fed into the BiLSTM-CRF model to obtain the text entities contained in the text data in the set; among them, the three embedding features are Token Embeddings, Segment Embeddings and PositionEmbeddings.

4. The method for multimodal named entity recognition of power equipment according to claim 1, characterized in that, The specific method for extracting triples from co-existing entities is as follows: The co-existing entity is matched with entities in the knowledge base, and the triples of the matched entities are used as the triples of the co-existing entity.

5. The method for multimodal named entity recognition of power equipment according to claim 1 or 4, characterized in that, A triple consists of a head entity, a relation, and a tail entity.

6. The method for multimodal named entity recognition of power equipment according to claim 1, characterized in that, The word order is adjusted based on relevance to obtain multimodal named entity recognition results, specifically including: The correlation between each word in the semantic vector and the relationship between each region block in the image are input into the Bert-BiLSTM-CRF model, and the Bert-BiLSTM-CRF model outputs multimodal named entity recognition results.

7. A multimodal named entity recognition system for power equipment, characterized in that, include: The data set generation module is used to collect text data from power equipment and corresponding image data, and to combine the collected text data and image data into a set. The text entity extraction module is used to extract text entities from a collection using natural language processing tools. The image entity extraction module is used to extract image entities from the set using an image entity retrieval tool, and input the image entities into the attention model to obtain image feature vectors; wherein, the image entity retrieval tool adopts the Mask RCNN model; The co-occurrence entity extraction module is used to extract entities that appear in both text entities and image entities as co-occurrence entities; The text feature vector generation module extracts triples from co-occurring entities, combines these triples with text data to reconstruct the word feature vectors needed in the text, and processes these word feature vectors using natural language processing tools to obtain the text feature vectors. The natural language processing tool employs the Bert-BiLSTM-CRF model. The process of combining triples with text data to reconstruct the word feature vectors specifically includes: By combining triples with text data, a pre-trained K-Bert model for sentence input is formed; The pre-trained K-Bert model injects relevant triple data into the received sentence, expanding the sentence into a sentence tree containing knowledge; The sentence tree is decomposed into word feature vectors; The word relationship adjustment module is used to concatenate text feature vectors and image feature vectors to form a semantic vector of image-text fusion. It adopts a multi-head attention mechanism to capture the correlation between each word in the semantic vector and each region block in the image, and adjusts the word order according to the correlation to complete multimodal named entity recognition.

8. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method for multimodal named entity recognition of power equipment as described in any one of claims 1 to 6.

9. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the multimodal named entity recognition method for power equipment according to any one of claims 1 to 6.

Images