Multi-modal data-oriented power equipment operation and maintenance knowledge extraction method and related device

Abstract

The invention belongs to the technical field of power equipment operation inspection, and discloses a multi-modal data-oriented power equipment operation inspection knowledge extraction method and a related device. The power equipment operation and maintenance knowledge extraction method comprises the steps of obtaining power equipment operation and maintenance multi-modal data to be subjected to knowledge extraction; inputting the power equipment operation and maintenance multi-modal data into a power equipment multi-modal extraction network model for knowledge extraction to obtain power equipment operation and maintenance knowledge; wherein the power equipment operation inspection multi-modal data comprises text record information and image record information of a specified entity and a relation type in a power equipment operation inspection record; the power equipment multi-modal extraction network model comprises a pre-training language model, a visual self-attention model, a multi-modal feature fusion module and an entity relationship extraction network. According to the technical scheme, mutual complementation of the image information and the text information is realized, and the accuracy of operation and inspection knowledge extraction of the power equipment can be improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of power equipment operation and maintenance, relates to natural language processing technology, and particularly to a method for extracting power equipment operation and maintenance knowledge oriented to multimodal data and related devices. Background Art

[0002] Promoting the deep integration of digital technology and the electric power energy sector and enabling the digital and intelligent transformation and upgrading of traditional industries is a strategic choice to seize new opportunities in the new round of scientific and technological revolution and industrial transformation; therefore, it is of great significance to use artificial intelligence technology to empower internal control means, optimize human resources, improve the intelligence level of power equipment operation and maintenance, and inspection processes, and break through key technologies such as automatic extraction of equipment operation and inspection knowledge. Further explanation, entity relationship extraction technology is one of the key technologies for artificial intelligence to enable knowledge precipitation in the power field. In view of the multimodal characteristics of power equipment operation and inspection records, by improving the multimodal entity relationship extraction model, the accuracy of professional knowledge extraction in the power field can be improved, and then the rapid and accurate positioning and tracing of power equipment faults or defects can be achieved.

[0003] At present, the existing entity relationship extraction methods for multimodal data are mainly cascade (Pipeline) architecture, which first performs multimodal entity extraction to extract entities in the data, and then performs multimodal relationship extraction to form "entity-relationship-entity" triples. The overall idea is to extract visual features in the image to supplement the missing information in the text; the above existing cascade architecture has the defect that the error in the previous stage may propagate backward, and ignores the potential correlation information between entities and relationships. In addition, in addition to the above cascade architecture, the existing technology also has a joint (Joint) architecture. The joint architecture extracts entities and relationships simultaneously during a training process, and can mine potential semantic feature associations. However, due to the complexity of multimodal data, there is currently a lack of research on the joint architecture.

[0004] It should be emphasized that in the process of power equipment operation and maintenance, image information and text information complement each other to form complete power equipment operation and maintenance knowledge. The existing multimodal entity extraction or relationship extraction models have poor domain transferability. Directly applying them to the process of power equipment operation and maintenance knowledge extraction will result in poor accuracy of power equipment operation and maintenance knowledge extraction.

[0005] To sum up, how to integrate multimodal data features, improve the accuracy of entity relationship extraction, and construct more accurate power equipment operation and maintenance knowledge has become a technical problem that needs to be solved urgently in the field of power equipment operation and maintenance. Summary of the invention

[0006] The purpose of the present invention is to provide a method and related device for extracting power equipment operation and maintenance knowledge from multimodal data to solve one or more of the above-mentioned technical problems. The power equipment operation and maintenance knowledge extraction scheme provided by the present invention can accurately extract and transform the multimodal data of power equipment operation and maintenance into power equipment operation and maintenance knowledge, realize the mutual supplementation of image information and text information in the process of power equipment operation and maintenance knowledge extraction, and improve the accuracy of power equipment operation and maintenance knowledge extraction.

[0007] In order to achieve the above object, the present invention adopts the following technical solutions:

[0008] In a first aspect, the present invention provides a method for extracting power equipment operation and maintenance knowledge based on multimodal data, comprising the following steps:

[0009] Acquire multimodal data of power equipment operation and inspection to be extracted; wherein the multimodal data of power equipment operation and inspection includes text record information and image record information of specified entities and relationship types in the power equipment operation and inspection records;

[0010] Input the acquired multimodal data of power equipment operation and inspection into the multimodal extraction network model of power equipment for knowledge extraction to obtain power equipment operation and inspection knowledge;

[0011] Wherein, the multi-modal extraction network model of power equipment includes:

[0012] A pre-trained language model is used to input text record information in the multimodal data of power equipment operation and inspection and output a text feature vector;

[0013] The visual self-attention model is used to input the image record information in the multimodal data of power equipment operation and inspection and output the image feature vector;

[0014] A multimodal feature fusion module, used for inputting the text feature vector and the image feature vector, performing feature fusion, and outputting the fused text feature vector;

[0015] The entity relationship extraction network is used to input the fused text feature vector, extract entities and relationships, and output the power equipment operation and maintenance knowledge in the form of a knowledge quadruple of "power equipment subject in text record information - relationship between power equipment subject and fault object in text record information - fault object in text record information - image record information".

[0016] A further improvement of the method of the present invention is that after obtaining the power equipment operation and maintenance knowledge, the method further includes the following steps:

[0017] Based on the acquired knowledge of power equipment operation and maintenance, the power equipment fault or defect is located and traced to obtain the location result and traceability result of the power equipment fault or defect.

[0018] A further improvement of the method of the present invention is that the pre-trained language model and the visual self-attention model in the power equipment multimodal extraction network model are both obtained by training using a power equipment word source library in an incremental training manner;

[0019] The power equipment word source library includes public domain data, power equipment knowledge and troubleshooting knowledge.

[0020] A further improvement of the method of the present invention is that the multimodal feature fusion module in the multimodal extraction network model of the power equipment is constructed based on the cross-attention mechanism; in the process of feature fusion performed by the multimodal feature fusion module, the image feature vector and the text feature vector are taken as input and the cross-attention weight is calculated, the image feature vector is used as a control condition and the degree of association between the image area of ​​the power equipment and the text sequence is calculated in the form of the cross-attention weight, and finally the fused text feature vector is obtained through normalization processing.

[0021] A further improvement of the method of the present invention is that the entity relationship extraction network in the multimodal extraction network model of the power equipment includes a plurality of double-layer subject extraction and classification networks, each of which corresponds to an operation and maintenance relationship, and each of which includes two neural network layers based on a two-focal curve function; wherein one neural network layer is used to input a text feature vector and predict the start position and end position of the power equipment subject, and another neural network layer is used to input a subject vector composed of the start position and end position of the power equipment subject and the fused text feature vector and extract the relationship and the fault object position to obtain the start position and end position of the fault object; in the extracted knowledge triple of "power equipment subject in text record information-relationship between power equipment subject and fault object in text record information-fault object in text record information", the image record information corresponding to the text record information is supplemented, and finally the power equipment operation and maintenance knowledge is output in the form of a knowledge quadruple of "power equipment subject in text record information-relationship between power equipment subject and fault object in text record information-fault object in text record information-image record information".

[0022] In a second aspect, the present invention provides a system for extracting knowledge from multimodal data of power equipment operation and maintenance, comprising:

[0023] A data acquisition module, used to acquire multimodal data of power equipment operation and inspection to be extracted; wherein the multimodal data of power equipment operation and inspection includes text record information and image record information of specified entities and relationship types in the power equipment operation and inspection records;

[0024] The knowledge extraction module is used to input the acquired multimodal data of power equipment operation and inspection into the multimodal extraction network model of power equipment for knowledge extraction to obtain power equipment operation and inspection knowledge;

[0025] Wherein, the multi-modal extraction network model of power equipment includes:

[0026] A pre-trained language model is used to input text record information in the multimodal data of power equipment operation and inspection, and output a text feature vector;

[0027] The visual self-attention model is used to input the image record information in the multimodal data of power equipment operation and inspection and output the image feature vector;

[0028] A multimodal feature fusion module, used for inputting the text feature vector and the image feature vector, performing feature fusion, and outputting the fused text feature vector;

[0029] The entity relationship extraction network is used to input the fused text feature vector, extract entities and relationships, and output the power equipment operation and maintenance knowledge in the form of a knowledge quadruple of "power equipment subject in text record information - relationship between power equipment subject and fault object in text record information - fault object in text record information - image record information".

[0030] A further improvement of the system of the present invention is that it also includes:

[0031] The post-processing module is used to locate and trace the faults or defects of the power equipment based on the acquired power equipment operation and inspection knowledge after obtaining the power equipment operation and inspection knowledge, and obtain the positioning results and tracing results of the faults or defects of the power equipment.

[0032] A further improvement of the system of the present invention is that the pre-trained language model and the visual self-attention model in the multi-modal extraction network model of the power equipment are both trained by using the power equipment word source library and an incremental training method;

[0033] The power equipment word source library includes public domain data, power equipment knowledge and troubleshooting knowledge.

[0034] A further improvement of the system of the present invention is that the multimodal feature fusion module in the multimodal extraction network model of the power equipment is constructed based on the cross-attention mechanism; in the process of feature fusion performed by the multimodal feature fusion module, the image feature vector and the text feature vector are taken as input and the cross-attention weight is calculated, the image feature vector is used as a control condition and the degree of association between the image area of ​​the power equipment and the text sequence is calculated in the form of the cross-attention weight, and finally the fused text feature vector is obtained through normalization processing.

[0035] A further improvement of the system of the present invention is that the entity relationship extraction network in the multimodal extraction network model of the power equipment includes a plurality of double-layer subject extraction and classification networks, each of which corresponds to an operation and maintenance relationship, and each of which includes two neural network layers based on a two-focal curve function; wherein one neural network layer is used to input a text feature vector and predict the start position and end position of the power equipment subject, and another neural network layer is used to input a subject vector composed of the start position and end position of the power equipment subject and the fused text feature vector and extract the relationship and the fault object position to obtain the start position and end position of the fault object; in the extracted knowledge triple of "power equipment subject in text record information-relationship between power equipment subject and fault object in text record information-fault object in text record information", the image record information corresponding to the text record information is supplemented, and finally the power equipment operation and maintenance knowledge is output in the form of a knowledge quadruple of "power equipment subject in text record information-relationship between power equipment subject and fault object in text record information-fault object in text record information-image record information".

[0036] According to a third aspect of the present invention, there is provided an electronic device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method for extracting knowledge from operation and maintenance of electric power equipment for multimodal data as described in any one of the first aspect of the present invention is implemented.

[0037] According to a fourth aspect of the present invention, a non-transitory computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the method for extracting power equipment operation and maintenance knowledge for multimodal data as described in any one of the first aspect of the present invention is implemented.

[0038] Compared with the prior art, the present invention has the following beneficial effects:

[0039] Aiming at the multimodal data generated during the operation and maintenance of electric power equipment, and aiming at the technical problems to be solved in the existing multimodal data feature fusion and knowledge extraction, the present invention specifically discloses a method for extracting knowledge of electric power equipment operation and maintenance for multimodal data, which utilizes the text record information and image record information in the operation and maintenance records of electric power equipment, and extracts the knowledge of the electric power equipment subject, operation and maintenance relationship and fault object in the operation and maintenance records of electric power equipment through the multimodal extraction network model of electric power equipment, so as to realize the mutual complementation of image information and text information, improve the accuracy of the extraction of knowledge of electric power equipment operation and maintenance, realize the accurate conversion of unstructured multimodal data into knowledge, and help reduce the manpower cost of operation and maintenance of electric power equipment. In summary, the technical solution of the present invention utilizes image information to improve the accuracy of the neural network model in extracting fault objects in text records, and combines the extracted knowledge of the electric power equipment subject, operation and maintenance relationship, and fault object with the image record to form an operation and maintenance knowledge quadruple, and this technical means increases the readability and practicality of the operation and maintenance knowledge. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below; obviously, the drawings described below are some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0041] Figure 1 It is a flow chart of a method for extracting power equipment operation and maintenance knowledge based on multimodal data in an embodiment of the present invention.

[0042] Figure 2 It is a schematic diagram of a multi-modal extraction network model of power equipment in an embodiment of the present invention.

[0043] Figure 3 It is a schematic diagram of the training and application process of a method for extracting power equipment operation and maintenance knowledge based on multimodal data in an embodiment of the present invention.

[0044] Figure 4 It is a schematic diagram of a multimodal feature fusion module in an embodiment of the present invention.

[0045] Figure 5 Schematic diagram of a double-layer subject extraction classification network in an embodiment of the present invention.

[0046] Figure 6 It is a schematic diagram of a power equipment operation and maintenance knowledge extraction system for multimodal data in an embodiment of the present invention. DETAILED DESCRIPTION

[0047] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings in the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0048] See also Figure 1 In an embodiment of the present invention, a method for extracting power equipment operation and inspection knowledge based on multimodal data is provided, comprising the following steps:

[0049] Step 1, obtaining the multimodal data of power equipment operation and inspection to be extracted; wherein the multimodal data of power equipment operation and inspection includes text record information and image record information of specified entities and relationship types in the power equipment operation and inspection records (further explanatory, the text record information and the image record information may be unstructured data in the process of power equipment operation and inspection knowledge extraction); in a further preferred technical solution, the initial power equipment operation and inspection multimodal data obtained may be first subjected to data cleaning (such as cleaning repeated text data, irrelevant text data, blurred image data, etc.), noise removal and other pre-processing operations to obtain the multimodal data of power equipment operation and inspection to be extracted, in order to improve the accuracy of the knowledge extraction results;

[0050] Step 2: input the acquired multimodal data of power equipment operation and inspection into the multimodal extraction network model of power equipment to extract knowledge and obtain power equipment operation and inspection knowledge;

[0051] A further preferred technical solution also includes:

[0052] Step 3: Based on the power equipment operation and inspection knowledge obtained in step 2, locate and trace the power equipment fault or defect, and obtain the location result and traceability result of the power equipment fault or defect;

[0053] See also Figure 2 In the technical solution of the above embodiment, the multi-modal extraction network model of the power equipment includes:

[0054] A pre-trained language model is used to input text record information in the multimodal data of power equipment operation and inspection and output a text feature vector;

[0055] The visual self-attention model is used to input the image record information in the multimodal data of power equipment operation and inspection and output the image feature vector;

[0056] A multimodal feature fusion module, used for inputting the text feature vector and the image feature vector, performing feature fusion, and outputting the fused text feature vector;

[0057] The entity relationship extraction network is used to input the fused text feature vector, extract entities and relationships, and output the power equipment operation and maintenance knowledge in the form of a knowledge quadruple of "power equipment subject in text record information - relationship between power equipment subject and fault object in text record information - fault object in text record information - image record information".

[0058] The embodiment of the present invention specifically provides a method for extracting power equipment operation and maintenance knowledge for multimodal data, which is used to perform data preprocessing operations on unstructured text data and image data recorded during the operation, maintenance, care, and inspection of power equipment. It can effectively solve the problem of inaccurate entity relationship extraction caused by the unpredictability of faults during the operation and maintenance of power equipment, narrow the heterogeneity gap of multimodal data in the power equipment operation and maintenance records, and improve the accuracy of power equipment operation and maintenance knowledge extraction.

[0059] In one embodiment of the present invention, power equipment knowledge, troubleshooting knowledge, etc. can be added to the public domain data to construct a power equipment word source library, and the visual self-attention model (i.e., pre-trained image training model) and pre-trained language model used in the knowledge extraction process are incrementally trained. Then, the image record information in the operation and inspection record is input into the pre-trained image training model to extract the image feature vector of the equipment fault, and the feature vector is fused with the text feature vector through a multimodal feature fusion module. Finally, the entity relationship extraction network of the power equipment operation and inspection extracts the entities and relationships in the power equipment operation and inspection record to form the operation and inspection knowledge. The technical solution provided by the embodiment of the present invention can effectively solve the problems of missing multimodal data fusion and inaccurate extraction of fault or defect objects in the operation and inspection records of power equipment, and enhances the readability and model robustness of the operation and inspection knowledge through multimodal information fusion, and improves the accuracy of multimodal entity relationship extraction.

[0060] See also Figures 3 to 5 In a specific embodiment of the present invention, a method for extracting power equipment operation and maintenance knowledge based on multimodal data is provided, and the specific steps are as follows:

[0061] Step 1), the power equipment operation and inspection records contain text descriptions such as descriptions of fault or defect processes, basic equipment conditions, and image records of equipment disassembly inspections. Data preprocessing operations are performed on these text descriptions and image records, including formatting text content, removing stop words, extracting vocabulary in the field of power equipment operation and inspection, fuzzy image processing, image scaling, and image size normalization. After preprocessing, the text descriptions and corresponding image records are annotated to construct a training data set in the field of power equipment operation and inspection;

[0062] Step 2), using the public domain data as the basic data, combined with the training data set in the field of power equipment operation and inspection constructed in step 1), using the text records therein to incrementally train the encoder BERT (Bidirectional Encoder Representations from Transformers) of the bidirectional deep self-attention transform network in a character splitting manner, and obtaining a pre-trained language model BERT; using the power equipment image records combined with the image records of the public power equipment images on the network to train the visual self-attention model ViT (Vision Transformer), and obtaining a pre-trained image model (i.e., the pre-trained visual self-attention model ViT);

[0063] Step 3), using the text description and image record in the training data set in the field of power equipment operation and inspection in step 1) as input, respectively through the pre-trained language model BERT and the pre-trained visual self-attention model ViT obtained in step 2), to obtain a text feature vector and an image feature vector;

[0064] Step 4), inputting the text feature vector and the image feature vector into a multimodal feature fusion module for feature fusion to obtain a fused feature vector; in a specific exemplary technical solution, the multimodal feature fusion module is constructed based on a cross attention mechanism;

[0065] Step 5), according to the result of step 4), the fused feature vector is input into the entity relationship extraction network of power equipment operation and maintenance, the power equipment subject, operation and maintenance relationship and defect (or fault) object are extracted, and finally the four-element combination of power equipment operation and maintenance knowledge is formed in combination with the operation and maintenance image records.

[0066] In a specific embodiment of the present invention, in step 1), structured data (such as table information) and unstructured data, i.e., image records of defects (faults) and operation and inspection text records, are extracted from the operation and inspection records of power equipment; the unstructured data is preprocessed, such as formatting text content, removing stop words, extracting vocabulary in the field of power equipment operation and inspection, fuzzy image processing, image scaling, and image size normalization, etc.; and finally a multimodal data set D = {d1, i1; ...; d k ,i k;…;d n ;i n}, d k Represents the preprocessed text inspection record, i k represents the preprocessed power equipment image, d k and i k One-to-one correspondence constitutes a sample combination; on the basis of the multimodal set D, the domain professional vocabulary set P is obtained through manual summary and consultation with experts, and the text operation record d k The training data set S = {s1, i1; s2, i2; ...; s n ,i n ; M}, where M is the label corresponding to each sentence, and the annotation content is composed of a triple {subject (Subject, s), relation (Relation, r), object (Object, o)}.

[0067] In a specific embodiment of the present invention, in step 2), based on the domain professional vocabulary set P and the multimodal data set D obtained in step 1), combined with the public data in the power field, a pre-training resource library D'={P; D; L} for power equipment operation and inspection is constructed, where L is the public data; the text data in the pre-training resource library is segmented according to Chinese characters to form a pre-training text D' c , input it into the encoder of the bidirectional deep self-attention transformer network for pre-training, and obtain the pre-trained language model BERT suitable for power equipment operation and inspection e ; Then, the image data in the pre-trained resource library is input into the training visual self-attention model Vision Transformer to obtain the image pre-training model Vit suitable for power equipment e .

[0068] In a specific embodiment of the present invention, in step 3), the accuracy of knowledge extraction of power equipment operation and inspection records is improved based on the information in the power equipment image. First, the training sample pair (s n ,i n ) is input into the pre-trained model to obtain the image feature vector H i and text feature vector H s , as shown in formula (1) and formula (2).

[0069] H i =Vit e (i n ) (1)

[0070] H s =BERT e (s n) (2)

[0071] See also Figure 4 In a specific embodiment of the present invention, in step 4), the image feature vector and the text feature vector are used as input to calculate the cross attention weight, and the image feature vector is used as a control condition to calculate the degree of association between different power equipment image regions and text sequences in the form of weights, that is, for each position of the power equipment image, the similarity score between it and all character positions of the inspection text record is calculated, and then normalized to obtain the text feature vector E after multimodal fusion. cross , as shown in formula (3).

[0072]

[0073] Among them, Q is the image feature vector H i , K and V are the feature vectors of the inspection record text H s , SoftMax is a normalization function, and its calculation method is shown in formula (4).

[0074]

[0075] Among them, z is the calculation process vector.

[0076] See also Figure 5 In a specific embodiment of the present invention, the entity relationship extraction network includes a plurality of double-layer subject extraction classification networks, and the number of the double-layer subject extraction classification networks is the same as the number of relationship types; wherein each double-layer subject extraction classification network includes two layers of neural network layers based on the S-type function (the S-type function is a two-focus curve function, Sigmoid);

[0077] The obtained text feature vector represents H s Input into the two-layer subject extraction classification network, and predict the starting position P of the power equipment subject through the prediction layer head and end position P tail , as shown in formula (5) and formula (6):

[0078] P head =σ(K head H s +b head ) (5)

[0079] P tail =σ(K tail H s +b tail ) (6)

[0080] Among them, K head 、b head , K tail and btail is a parameter in a fully connected neural network;

[0081] The main body vector S1 can be obtained by the starting position and the ending position of the main body of the power equipment, and it is combined with the multimodal fusion feature vector E obtained in step 4) cross The relationship and object position are extracted by fusion (the corresponding position addition fusion method can be used). Specifically, for each power equipment operation and maintenance relationship, a double-layer subject extraction classification network is used to extract the start and end positions of the object. Then, according to the subject, object position and relationship, the power extracted operation and maintenance knowledge triple {power equipment; operation and maintenance relationship; fault object} can be obtained. On this basis, the image in the training sample in step 1) is added as a supplement to form the power equipment operation and maintenance knowledge quadruple {power equipment subject; operation and maintenance relationship; fault object; operation and maintenance image}.

[0082] In the embodiment of the present invention, facing the multimodal data generated during the operation and inspection of power equipment, and aiming at the problems of multimodal data feature fusion and knowledge extraction, a method for extracting entity relationships facing multimodal data is provided, specifically, the equipment subject, operation and inspection relationship and fault object knowledge in the operation and inspection record of power equipment are extracted. The entity relationship extraction method facing multimodal data realizes the mutual complementation of image record information and text record information, improves the accuracy of power equipment operation and inspection knowledge extraction, realizes the accurate conversion of unstructured multimodal data into knowledge, and reduces the human cost of power equipment operation and inspection. In view of the characteristics of multimodal data of power equipment operation and inspection, the embodiment of the present invention proposes a method for extracting power equipment operation and inspection knowledge facing multimodal data based on an artificial intelligence deep neural network model, constructs a pre-trained resource library based on professional vocabulary in the field of power equipment operation and inspection record extraction, and obtains pre-trained language models and visual self-attention models in the field of power equipment operation and inspection through incremental training; through a multimodal feature fusion module, the accuracy of the neural network model in extracting fault objects in text records is improved by using image information, and the extracted power equipment subject, operation and inspection relationship, and fault object knowledge are combined with image records to form an operation and inspection knowledge quadruple, which increases the readability and practicality of the operation and inspection knowledge. The technical solution of the embodiment of the present invention can automatically extract information of designated entities and relationship types from operation and inspection records of different electric power equipment.

[0083] The following are device embodiments of the present invention, which can be used to implement the method embodiments of the present invention. For details not disclosed in the device embodiments, please refer to the method embodiments of the present invention.

[0084] See also Figure 6 In an embodiment of the present invention, a system for extracting knowledge from multimodal data of power equipment operation and inspection is provided, comprising:

[0085] A data acquisition module, used to acquire multimodal data of power equipment operation and inspection to be extracted; wherein the multimodal data of power equipment operation and inspection includes text record information and image record information of specified entities and relationship types in the power equipment operation and inspection records;

[0086] The knowledge extraction module is used to input the acquired multimodal data of power equipment operation and inspection into the multimodal extraction network model of power equipment for knowledge extraction to obtain power equipment operation and inspection knowledge;

[0087] Wherein, the multi-modal extraction network model of power equipment includes:

[0088] A pre-trained language model is used to input text record information in the multimodal data of power equipment operation and inspection and output a text feature vector;

[0089] The visual self-attention model is used to input the image record information in the multimodal data of power equipment operation and inspection and output the image feature vector;

[0090] A multimodal feature fusion module, used for inputting the text feature vector and the image feature vector, performing feature fusion, and outputting the fused text feature vector;

[0091] The entity relationship extraction network is used to input the fused text feature vector, extract entities and relationships, and output the power equipment operation and maintenance knowledge in the form of a knowledge quadruple of "power equipment subject in text record information - relationship between power equipment subject and fault object in text record information - fault object in text record information - image record information".

[0092] In one embodiment of the present invention, a computer device is provided, the computer device comprising a processor and a memory, the memory is used to store a computer program, the computer program includes program instructions, and the processor is used to execute the program instructions stored in the computer storage medium. The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. It is the computing core and control core of the terminal, which is suitable for implementing one or more instructions, and is specifically suitable for loading and executing one or more instructions in the computer storage medium to implement the corresponding method flow or corresponding function; the processor described in the embodiment of the present invention can be used to perform the operation of the power equipment operation and inspection knowledge extraction method for multimodal data.

[0093] In one embodiment of the present invention, a storage medium is provided, specifically a computer-readable storage medium (Memory), which is a memory device in a computer device for storing programs and data. It is understandable that the computer-readable storage medium here can include both built-in storage media in the computer device and, of course, extended storage media supported by the computer device. The computer-readable storage medium provides a storage space, which stores the operating system of the terminal. In addition, one or more instructions suitable for being loaded and executed by the processor are also stored in the storage space, and these instructions can be one or more computer programs (including program codes). It should be noted that the computer-readable storage medium here can be a high-speed RAM memory or a non-volatile memory, such as at least one disk memory. The processor can load and execute one or more instructions stored in the computer-readable storage medium to implement the corresponding steps of the method for extracting knowledge of power equipment operation and inspection for multimodal data in the above embodiment.

[0094] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0095] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0096] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0097] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0098] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A method for extracting power equipment operation and maintenance knowledge from multimodal data, characterized in that: The following steps are involved: Acquire multimodal data of power equipment operation and inspection to be extracted; wherein the multimodal data of power equipment operation and inspection includes text record information and image record information of specified entities and relationship types in the power equipment operation and inspection records; Input the acquired multimodal data of power equipment operation and inspection into the multimodal extraction network model of power equipment for knowledge extraction to obtain power equipment operation and inspection knowledge; Wherein, the multi-modal extraction network model of power equipment includes: A pre-trained language model is used to input text record information in the multimodal data of power equipment operation and inspection and output a text feature vector; The visual self-attention model is used to input the image record information in the multimodal data of power equipment operation and inspection and output the image feature vector; A multimodal feature fusion module, used for inputting the text feature vector and the image feature vector, performing feature fusion, and outputting the fused text feature vector; The entity relationship extraction network is used to input the fused text feature vector, extract entities and relationships, and output power equipment operation and inspection knowledge in the form of a knowledge quadruple of "power equipment subject in text record information - relationship between power equipment subject and fault object in text record information - fault object in text record information - image record information".

2. According to the method for extracting knowledge of power equipment operation and maintenance based on multimodal data according to claim 1, it is characterized in that: After acquiring the knowledge of power equipment operation and maintenance, the following steps are also included: Based on the acquired knowledge of power equipment operation and maintenance, the power equipment fault or defect is located and traced to obtain the location result and traceability result of the power equipment fault or defect.

3. The method for extracting power equipment operation and maintenance knowledge based on multimodal data according to claim 1 is characterized in that: The pre-trained language model and visual self-attention model in the electric power equipment multimodal extraction network model are both obtained by training using an electric power equipment word source library in an incremental training manner; The power equipment word source library includes public domain data, power equipment knowledge and troubleshooting knowledge.

4. The method for extracting power equipment operation and maintenance knowledge from multimodal data according to claim 1 is characterized in that: The multimodal feature fusion module in the multimodal extraction network model of the power equipment is constructed based on the cross-attention mechanism; in the process of feature fusion performed by the multimodal feature fusion module, the image feature vector and the text feature vector are taken as input and the cross-attention weight is calculated, the image feature vector is taken as the control condition and the correlation degree between the image area of ​​the power equipment and the text sequence is calculated in the form of the cross-attention weight, and finally the fused text feature vector is obtained through normalization processing.

5. The method for extracting power equipment operation and maintenance knowledge from multimodal data according to claim 1 is characterized in that: The entity relationship extraction network in the multimodal extraction network model of the power equipment includes a plurality of double-layer subject extraction and classification networks, each of which corresponds to an operation and inspection relationship, and each of which includes two layers of neural network layers based on a two-focal curve function; wherein one neural network layer is used to input a text feature vector and predict the start position and end position of the power equipment subject, and another neural network layer is used to input a subject vector composed of the start position and end position of the power equipment subject and the fused text feature vector and extract the relationship and the fault object position to obtain the start position and end position of the fault object; in the extracted knowledge triple of "power equipment subject in text record information-relationship between power equipment subject and fault object in text record information-fault object in text record information", the image record information corresponding to the text record information is supplemented, and finally the power equipment operation and inspection knowledge is output in the form of a knowledge quadruple of "power equipment subject in text record information-relationship between power equipment subject and fault object in text record information-fault object in text record information-image record information".

6. A power equipment operation and maintenance knowledge extraction system for multimodal data, characterized in that: include: A data acquisition module, used to acquire multimodal data of power equipment operation and inspection to be extracted; wherein the multimodal data of power equipment operation and inspection includes text record information and image record information of specified entities and relationship types in the power equipment operation and inspection records; The knowledge extraction module is used to input the acquired multimodal data of power equipment operation and inspection into the multimodal extraction network model of power equipment for knowledge extraction to obtain power equipment operation and inspection knowledge; Wherein, the multi-modal extraction network model of power equipment includes: A pre-trained language model is used to input text record information in the multimodal data of power equipment operation and inspection and output a text feature vector; The visual self-attention model is used to input the image record information in the multimodal data of power equipment operation and inspection and output the image feature vector; A multimodal feature fusion module, used for inputting the text feature vector and the image feature vector, performing feature fusion, and outputting the fused text feature vector; The entity relationship extraction network is used to input the fused text feature vector, extract entities and relationships, and output power equipment operation and inspection knowledge in the form of a knowledge quadruple of "power equipment subject in text record information - relationship between power equipment subject and fault object in text record information - fault object in text record information - image record information".

7. The power equipment operation and maintenance knowledge extraction system for multimodal data according to claim 6 is characterized in that: Also includes: The post-processing module is used to locate and trace the faults or defects of the power equipment based on the acquired power equipment operation and inspection knowledge after obtaining the power equipment operation and inspection knowledge, and obtain the positioning results and tracing results of the faults or defects of the power equipment.

8. The power equipment operation and maintenance knowledge extraction system for multimodal data according to claim 6 is characterized in that: The pre-trained language model and visual self-attention model in the electric power equipment multimodal extraction network model are both obtained by training using an electric power equipment word source library in an incremental training manner; The power equipment word source library includes public domain data, power equipment knowledge and troubleshooting knowledge.

9. The power equipment operation and maintenance knowledge extraction system for multimodal data according to claim 6 is characterized in that: The multimodal feature fusion module in the multimodal extraction network model of the power equipment is constructed based on the cross-attention mechanism; in the process of feature fusion performed by the multimodal feature fusion module, the image feature vector and the text feature vector are taken as input and the cross-attention weight is calculated, the image feature vector is taken as the control condition and the correlation degree between the image area of ​​the power equipment and the text sequence is calculated in the form of the cross-attention weight, and finally the fused text feature vector is obtained through normalization processing.

10. The power equipment operation and maintenance knowledge extraction system for multimodal data according to claim 6, characterized in that: The entity relationship extraction network in the multimodal extraction network model of the power equipment includes a plurality of double-layer subject extraction and classification networks, each of which corresponds to an operation and inspection relationship, and each of which includes two layers of neural network layers based on a two-focal curve function; wherein one neural network layer is used to input a text feature vector and predict the start position and end position of the power equipment subject, and another neural network layer is used to input a subject vector composed of the start position and end position of the power equipment subject and the fused text feature vector and extract the relationship and the fault object position to obtain the start position and end position of the fault object; in the extracted knowledge triple of "power equipment subject in text record information-relationship between power equipment subject and fault object in text record information-fault object in text record information", the image record information corresponding to the text record information is supplemented, and finally the power equipment operation and inspection knowledge is output in the form of a knowledge quadruple of "power equipment subject in text record information-relationship between power equipment subject and fault object in text record information-fault object in text record information-image record information".

11. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the method for extracting power equipment operation and maintenance knowledge for multimodal data according to any one of claims 1 to 5 is implemented.

12. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for extracting knowledge from operation and maintenance of electric power equipment oriented to multimodal data according to any one of claims 1 to 5 is implemented.

Images