A power multi-modal entity relation extraction method and device based on transfer learning

Abstract

This invention relates to a method for extracting multimodal entity relationships in the power industry based on transfer learning, comprising the following steps: collecting and preprocessing power industry text and image data; constructing a cross-modal relationship feature extraction model to obtain high-level features r of multimodal data in the power industry. t High-level features of open-domain multimodal corpora s Constructing a power multimodal entity relationship extraction model based on transfer learning: This involves extracting high-level features r... s High-level features r corresponding to the power sector t Adaptation enables higher-level features r t With high-level features r s The high-level feature r with the smallest difference value is selected when the difference value is less than a preset threshold. t With high-level features r s The result is input to the next layer; the classification relationship output layer outputs the relationship recognition result. The beneficial effects of this invention are: it utilizes rich corpora from other fields to extract entity relationships from multimodal power data, thus alleviating the problem of limited labeled corpus resources in the power field.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of electric power data processing, and particularly relates to a power multi-modal entity relation extraction method and device based on transfer learning. BACKGROUND

[0002] With the rapid development of information technology, a large amount of power data is generated every day, and the modalities of data are diversified, including text, image, etc. Multi-modal data grows explosively, and the text and image in the power field contain rich unstructured power entities and relations. However, the existing technology is difficult to effectively obtain valuable information from them. Therefore, how to improve the power multi-modal entity relation extraction technology, extract effective information from a large amount of multi-modal data, and depict the association between multi-modal information is crucial for the construction of power multi-modal knowledge graph and business intelligent application in the later period.

[0003] The first prior art solution: the invention patent with the publication number CN116467679A discloses a method for discovering multi-modal relations in the field of semiconductors. The solution is as follows: acquiring image data and text data; generating a scene graph according to the image data; generating a dependency syntax tree according to the text data; using an encoder to encode the text data to obtain a text encoding vector corresponding to the entity in the text data; generating a visual semantic vector corresponding to the entity node in the scene graph; fusing the scene graph and the dependency syntax tree according to the similarity between entities to obtain a first association graph; splicing the visual semantic vector and the text encoding vector according to the similarity to obtain a first vector; splicing the first vector and a second vector corresponding to the entity in the first association graph according to the similarity to obtain a third vector; acquiring a fourth vector corresponding to a known entity, and inputting the third vector and the fourth vector after splicing into a relation prediction model to obtain the relation between the entities corresponding to the third vector and the fourth vector. The first solution is for multi-modal relation discovery in the field of semiconductors, but there are problems such as less labeled corpus resources in the field, high labor cost of labeling, and difficulty in applying deep learning methods that rely on large-scale corpus, resulting in poor learning and recognition effect.

[0004] The prior art scheme two: the invention patent with the publication number CN116595187A discloses a method of education field multi-modal relation extraction based on deep learning and device, which comprises the following steps: acquiring multi-modal data of the education field to be recognized; inputting the multi-modal data into a constructed multi-modal relation extraction model; wherein the multi-modal relation extraction model comprises a multi-modal relation class extraction model and a multi-modal entity recognition model; obtaining the recognition result of the multi-modal data of the education field according to the multi-modal data and the multi-modal relation extraction model. The present application can sort out the relationship between teaching courses in education, optimize the teaching scheme, and provide help for the construction of the knowledge graph in the education field. The scheme two is aimed at multi-modal relation discovery in the education field, but also has the problems of less annotated corpus resources in the field and high labor cost of annotation, which makes it difficult to apply deep learning and other methods that rely on large-scale corpus, and makes the learning and recognition effect poor.

[0005] Due to the lack of annotated corpus resources in the power field, it is difficult to use deep learning and other methods that rely on large-scale corpus, therefore, how to use the rich corpus of other fields to enhance the learning effect of power multi-modal data has become a problem to be solved. SUMMARY

[0006] To solve the above problems, the purpose of the present application is to provide a power multi-modal entity relation extraction method based on transfer learning, which uses rich corpus of other fields to extract entity relations in power multi-modal data, to alleviate the problem of lack of annotated corpus resources in the power field, and to support subsequent power multi-modal knowledge graph construction and business application.

[0007] To achieve the above purpose, the present application adopts the following technical scheme:

[0008] Technical scheme one

[0009] A power multi-modal entity relation extraction method based on transfer learning, comprising the following steps: collecting and preprocessing power text and image data; constructing a cross-modal relation feature extraction model: inputting the multi-modal data of the power field into the cross-modal relation feature extraction model, extracting image features and text features, and fusing them to obtain high-level features r of the power field multi-modal data t ; inputting the multi-modal corpus of the open field into the cross-modal relation feature extraction model to obtain high-level features r of the open field multi-modal corpus s ; constructing a power multi-modal entity relation extraction model based on transfer learning: adapting the high-level features r obtained from open field learning to the corresponding high-level features r of the power field s , so that the high-level features r t adapt to the high-level features r t ; and constructing a power multi-modal entity relation extraction model based on transfer learning: adapting the high-level features r obtained from open field learning to the corresponding high-level features r of the power field sThe high-level feature r with the smallest difference value among them is selected when the difference value is less than a preset threshold. t With high-level features r s The results are input to the next layer; the classification relationship output layer processes the high-level features of the input through the softmax function and outputs the relationship recognition results.

[0010] More preferably, the cross-modal relationship feature extraction model includes an image entity relationship detection model, a text description branch network, and feature fusion; the image is input into the image entity relationship detection model, which outputs a visual embedding of the relationship; the text is input into the text description branch network, which first obtains word vectors, and then converts the word vectors into text description features; the feature fusion combines the text feature description and the visual embedding of the relationship to obtain high-level features; multimodal data from the power sector is input into the cross-modal relationship feature extraction model, which outputs high-level features r. t The open-domain multimodal corpus is input into the cross-modal relation feature extraction model, which outputs high-level features r. s .

[0011] More preferably, the image entity relationship detection model includes a backbone network, a Faster R-CNN network, a RoI pooling layer, and a fully connected layer; the image is input into the backbone network to obtain the global features of the image, then the Faster R-CNN layer is used to obtain the subject, relationship, and object, and finally the RoI pooling layer is used to obtain the corresponding feature embedding Z. S Z P Z O Finally, the Intersection over Union (IoU) is calculated. When IoU > 0, the bounding boxes of the subject and object intersect, indicating a relationship between them. This yields the corresponding image features and bounding boxes. After obtaining the feature embeddings and boundary regions of the subject, relationship, and object, two fully connected layers are connected after the subject image feature branch, relationship branch, and object image feature branch, respectively. This process is then followed by the fully connected layer W. 1,2 Then output the three middle hidden features. and Hidden features for the main image feature branches Then through the fully connected layer Obtain the final visual embedding x S Hidden features for object image feature branches Then through the fully connected layer Obtain the final visual embedding x O For relational branches, first, the three hidden features are... and Concatenate them and input them into the fully connected layer. To obtain higher-level hidden relationship features Then embed the main visual element into x.S and object visual embedding x O with relation hidden feature concatenate and input to two fully connected layers get relation visual embedding x P .

[0012] More preferably, the text description branch network is based on the Google Word2Vec model, and each input text entity can obtain a multi-dimensional word vector through the Word2Vec model, and then the word vector is normalized.

[0013] More preferably, the feature fusion is to obtain a text description feature by using a Bi-LSTM model based on the word vector obtained by the text description branch network, and then fuse the text description feature with the feature output by the image entity relationship detection model to obtain a high-level feature; the text-image interaction loss function is set as follows: wherein, denotes the variance of the relation type, and the greater the variance, the smaller the proportion of the loss.

[0014] More preferably, the power multi-modal entity relationship extraction model based on transfer learning uses a maximum mean difference method to compare the distribution difference between r s and r t ; the formula definition of MMD measurement is: wherein R s and R t respectively represent the high-level feature sets of the corresponding fields learned by the open field data set and the power field data set, table is the i-th power field high-level feature r t , and n represents the number of high-level features r t in the power field data set R t table is the i-th open field high-level feature r s , m represents the number of high-level features r s in the power field data set R s , and H represents that this measurement method is performed by a feature mapping function mapping the high-level feature into a reproducing kernel Hilbert space, and the new feature space after the function mapping reflects the accuracy of the MMD measurement.

[0015] Technical solution two

[0016] A power multi-modal entity relationship extraction device based on transfer learning, comprising: a preprocessing module: collecting power text and image data and preprocessing;

[0017] ​The cross-modal relation feature extraction model: inputting the multi-modal data in the power field into the cross-modal relation feature extraction model, extracting image features and text features, and fusing them to obtain high-level features r of the multi-modal data in the power field t , inputting the multi-modal corpus in the open field into the cross-modal relation feature extraction model to obtain high-level features r of the multi-modal corpus in the open field s

[0018] The power multi-modal entity relation extraction model: inputting the high-level features r obtained from the open field learning into the power multi-modal entity relation extraction model s corresponding to the power field t adaptation, minimizing the difference value between the high-level features r t and the high-level features r s with a difference value less than a preset threshold t and the high-level features r s as a result input the next layer

[0019] The classification relation output layer processes the input high-level features through a softmax function and outputs a relation recognition result.

[0020] More preferably, the cross-modal relation feature extraction model includes an image entity relation detection model, a text description branch network, and a feature fusion; inputting an image into the image entity relation detection model outputs a relation visual embedding, inputting a text into the text description branch network first obtains a word vector, and then converts the word vector into a text description feature, the feature fusion fuses the text feature description and the relation visual embedding to obtain a high-level feature; inputting the multi-modal data in the power field into the cross-modal relation feature extraction model outputs the high-level feature r t , inputting the multi-modal corpus in the open field into the cross-modal relation feature extraction model outputs the high-level feature r s

[0021] More preferably, the image entity relation detection model includes a backbone network, a Faster R-CNN network, a RoI pooling layer, and a full connection layer; inputting an image into the backbone network obtains global features of the image, then obtaining a subject, a relation, and an object through the Faster R-CNN layer, and then obtaining corresponding feature embeddings Z S , Z P , Z O ​​, and finally calculate the intersection over union IoU. When IoU>0, the bounding boxes of the subject and the object have intersection, and there is a relationship between the subject and the object. The picture features and the bounding boxes corresponding to the relationship are obtained. After obtaining the feature embedding and the bounding area of the subject, the relationship and the object, two fully connected layers are connected after the subject image feature branch, the relationship branch and the object image feature branch respectively, and three intermediate hidden features are output after the fully connected layer W 1,2 and For the hidden feature of the subject image feature branch , the final visual embedding x S is obtained after the fully connected layer . O For the hidden feature of the object image feature branch , the final visual embedding x O is obtained after the fully connected layer . S For the relationship branch, first, the three hidden features and are spliced and input into the fully connected layer to obtain a higher level of relationship hidden feature . Then, the subject visual embedding x S and the object visual embedding x O are spliced with the relationship hidden feature , and input into two fully connected layers to obtain the relationship visual embedding x P .

[0022] The text description branch network is based on the Word2Vec model of Google. Each input text entity can obtain a multi-dimensional word vector through the Word2Vec model, and then the word vector is normalized.

[0023] The feature fusion is to obtain the text description feature by using the Bi-LSTM model based on the word vector obtained by the text description branch network, and then fuse the text description feature with the feature output by the image entity relationship detection model to obtain high-level features. The text-image interaction loss function is set as follows: wherein, is the variance of the relationship type, and the greater the variance, the smaller the proportion of the loss.

[0024] More preferably, the power multi-modal entity relationship extraction model based on transfer learning uses the method of maximum mean difference MMD to compare the distribution difference between r s and r t . The formula definition of MMD measurement is: wherein R s and R trespectively represent the high-level feature sets of the corresponding fields learned by the open field dataset and the power field dataset respectively, Table is the high-level feature r of the i-th power field t , n represents the number of high-level features r in the power field dataset R t t Table is the high-level feature r of the i-th open field s , m represents the number of high-level features r in the power field dataset R s s , H represents that this metric method is carried out by mapping the high-level features into the reproducing kernel Hilbert space by the feature mapping function The new feature space after the function mapping reflects the accuracy of the MMD metric.

[0025] The present application has the following beneficial effects:

[0026] 1. The power multi-modal entity relation extraction method and device based on transfer learning, considering that the entity relation extracted by the text and image single-modal information has the disadvantages of high ambiguity and weak expression, a cross-modal relation feature extraction model is further constructed on the basis of the text and image relation detection single-modal model. The constructed model introduces a text description branch, dynamically fuses text coding and image feature extraction, guides the text and image information to maximize the commonality of each other by designing a text-image interaction loss function, and effectively improves the robustness of the relation detection model in a complex scene.

[0027] 2. The power multi-modal entity relation extraction method and device based on transfer learning of the present application, which migrates the open field knowledge into the power field, constructs a power multi-modal entity relation extraction model based on transfer learning based on the high-level feature layer, minimizes the maximum mean difference MMD between the high-level feature layers of the two fields, adapts the feature spaces of the open field and the power field, and simultaneously learns the features in the two fields, thereby effectively improving the relation extraction effect of the power field.

[0028] 3. The power multi-modal entity relation extraction method and device based on transfer learning of the present application, which migrates the knowledge of other open fields and the power field into the same feature space, uses the rich corpus of other open fields to enhance the learning effect of the power field multi-modal data, so as to alleviate the problem of insufficient labeled corpus. DETAILED DESCRIPTION

[0029] Figure 1 is a flowchart of the power multi-modal entity relation extraction method of the present application;

[0030] Figure 2 ​​​​A cross-modal relationship feature extraction model of the present application;

[0031] Figure 3 A power multi-modal entity relationship extraction device based on transfer learning of the present application. DETAILED DESCRIPTION

[0032] The present application will be further described in detail below in combination with the accompanying drawings and specific embodiments:

[0033] Embodiment one

[0034] Please refer to Figure 1 and Figure 3 A power multi-modal entity relationship extraction method based on transfer learning, comprising the following steps:

[0035] Step 1, collect and preprocess power text and image data; collect massive text and image data related to power equipment, with the number of text and image reaching more than 2 million and more than 300,000 respectively. Using the semi-manual labeling method, first select 500 pieces of text and image corresponding data, use the jieba word segmentation tool to complete the power equipment preprocessing word segmentation and stop word removal, and complete the manual labeling of entities and relationships; use the labelImg image target labeling tool to complete the automatic labeling of power equipment image targets and manual correction.

[0036] Step 2, build a cross-modal relationship feature extraction model: input the multi-modal data in the power field into the cross-modal relationship feature extraction model, extract image features and text features, and fuse them to obtain high-level features r of the power field multi-modal data t Input the multi-modal corpus in the open field into the cross-modal relationship feature extraction model to obtain high-level features r of the open field multi-modal corpus s The open field multi-modal corpus uses the open source version of the "ShuSheng·WanJuan1.0" multi-modal corpus, which contains three parts of text data set, image-text data set and video data set, with a total data amount of more than 2TB, and is a corpus built based on the large model data alliance.

[0037] Please refer to Figure 2 The cross-modal relationship feature extraction model mainly consists of an image entity relationship detection model, a text description branch network and feature fusion.

[0038] 1) Image Entity Relationship Detection Model: The image is input into the image entity relationship detection model, which outputs a visual embedding of the relationship. The image entity relationship detection model includes a backbone network, a Faster R-CNN network, a RoI pooling layer, and fully connected layers. The image is input into the backbone network to obtain the global features of the image. For example, here, Conv1_1 to Conv5_3 of the VGG16 network are used, that is, from the first sub-layer of the first convolutional layer to the third sub-layer of the fifth convolutional layer, thus obtaining the global features of the image. Then, the bounding boxes R of the subject, relationship, and object are obtained through the Faster R-CNN layer. S R p R O Then, the corresponding feature embedding Z is obtained through the RoI pooling layer. S Z P Z O Finally, the Intersection over Union (IoU) is calculated. When IoU > 0, the bounding boxes of the subject and object intersect, indicating a relationship between the subject and object. This yields the corresponding image features and bounding boxes for the relationship. After obtaining the feature embeddings and boundary regions of the subject, relationship, and object, two fully connected layers are connected after the subject image feature branch, relationship branch, and object image feature branch, respectively. This is followed by a fully connected layer W. 1,2 Then output the three intermediate hidden features. and Hidden features for the main image feature branches Then through a fully connected layer Obtain the final visual embedding x S Hidden features for object image feature branches Then through the fully connected layer Obtain the final visual embedding x O For the relational branches, a two-level feature fusion was applied. First, the three hidden features were fused together. and Concatenate them and input them into the fully connected layer. To obtain higher-level hidden relationship features Then embed the main visual element into x. S and object visual embedding x O Hidden features of relationships Concatenate them and input them into two fully connected layers. Obtain relational visual embeddings x P Finally, the visual is embedded in x. S x O and x P The result is sent to the loss function for image ternary relation calculation, which is calculated using the following formula:

[0039]

[0040] wherein L represents a loss value, N is the number of positive RoIs, K is the number of negative samples for each positive RoI, and the m parameter is set to 0.2, represents the i-th RoI for the positive, and the j-th negative sample corresponding thereto.

[0041] 2) Text description branch network: based on the Google Word2Vec model, each input text entity can obtain a multi-dimensional word vector through the Word2Vec model, for example, a 300-dimensional word vector is obtained, and then the word vector is normalized. For words that do not exist in the word library, the network will filter the short sentences in which they are located.

[0042] 3) Feature fusion: the feature fusion fuses the text feature description and the relationship visual embedding to obtain high-level features. The multi-modal data in the power field is input into the cross-modal relationship feature extraction model, and the high-level features r are output after feature fusion. t The multi-modal corpus in the open field is input into the cross-modal relationship feature extraction model, and the high-level features r are output after feature fusion. s Specifically: the word vector obtained based on the text description branch network is used to obtain the text description feature by using the Bi-LSTM model, and then the high-level feature is obtained by fusing the text description feature with the feature output by the image entity relationship detection model; the text-image interaction loss function is set as follows: wherein, denotes the variance of the relationship type, L text_image represents a loss value. When the variance is larger, the proportion of the loss is smaller.

[0043] Step 3, constructing a power multi-modal entity relationship extraction model based on transfer learning: the high-level features r s learned from the open field are input into the power field, and the high-level features r t are obtained. t Step 4, making the difference value between the high-level features r s and the high-level features r t minimum, and the high-level features r s with a difference value less than a preset threshold are input into the next layer as a result. The method of making the difference between the two feature distributions minimum is to find a transformation, which makes the distance between the two field features after transformation minimum. The maximum mean difference MMD method is used to compare the distribution difference between r s and r t . The formula definition of MMD measurement is as follows: wherein R s and R tThese represent the sets of high-level features for their respective domains, obtained through training on the open-domain dataset and the power domain dataset presented in this paper. The table represents the high-level features r of the i-th power sector. t , where n represents the power sector dataset R t Medium and high-level features r t Quantity, Table r represents the high-level features of the i-th open domain. s m represents the power sector dataset R s Medium and high-level features r s The number of features, H represents the number of features measured by the feature mapping function. The mapping of high-level features to the reproducing kernel Hilbert space is performed, after a function. The new feature space after mapping reflects the accuracy of the MMD metric. Ideally, features from different domains and distributions show the greatest difference in the new distribution, while features from the same domain show near-zero differences.

[0044] Step 4: The classification relationship output layer processes the input high-level features using a softmax function and outputs the relationship recognition results. Compared to the solution without an open-domain multimodal corpus, this invention improves the F1 score by 8.12%, reaching 83.65%, demonstrating that utilizing rich corpora from other domains can enhance the learning effect in the power sector and alleviate the problem of insufficient corpus. Finally, manual proofreading and correction are performed to better support the construction and application of the power multimodal knowledge graph.

[0045] This invention presents a method for extracting multimodal entity relations in the power industry based on transfer learning. It utilizes rich corpora from other fields to enhance the learning effect of multimodal entity relation extraction in the power industry, alleviates the problem of insufficient labeled corpora in the power industry, and reduces the cost of manual labeling.

[0046] Based on the same inventive concept, this application provides an apparatus corresponding to Embodiment 1, as detailed in Embodiment 2.

[0047] Example 2

[0048] Please see Figure 3 A power industry multimodal entity relation extraction device based on transfer learning includes: a preprocessing module for collecting and preprocessing power industry text and image data; and a cross-modal relation feature extraction model for inputting multimodal data from the power industry into the model, extracting image and text features, and fusing them to obtain high-level features r of the multimodal data from the power industry. t The open-domain multimodal corpus is input into the cross-modal relation feature extraction model to obtain the high-level features r of the open-domain multimodal corpus. s; the power multi-modal entity relation extraction model: the high-level features r s corresponding to the power field t adaptation, so that the difference value between the high-level features r t and r s is minimum, and the high-level features r t with a difference value less than a preset threshold are selected s as the final high-level features r t .

[0049] The cross-modal relation feature extraction model includes an image entity relation detection model, a text description branch network and feature fusion. The image is input into the image entity relation detection model to output relation visual embedding, the text is input into the text description branch network to obtain word vector first, and then the word vector is converted into text description feature. The feature fusion fuses the text feature description and the relation visual embedding to obtain high-level features. The multi-modal data of the power field is input into the cross-modal relation feature extraction model to output high-level features r t , and the multi-modal corpus of the open field is input into the cross-modal relation feature extraction model to output high-level features r s .

[0050] The image entity relation detection model includes a backbone network, a FasterR-CNN network, a RoI pooling layer and a full connection layer. The image is input into the backbone network to obtain the global feature of the image, then the subject, the relation and the object are obtained through the FasterR-CNN layer, and then the corresponding feature embedding Z S , Z P and Z O are obtained through the RoI pooling layer. Finally, the intersection over union IoU is calculated. When IoU>0, the boundary boxes of the subject and the object have intersection, and only then the subject and the object have relation. The picture feature and the boundary box corresponding to the relation are obtained. After obtaining the feature embedding and the boundary region of the subject, the relation and the object, two full connection layers are connected respectively in the subject image feature branch, the relation branch and the object image feature branch. After the full connection layer W 1,2 , three intermediate hidden features and are output. For the hidden feature of the subject image feature branch , the final visual embedding x S is obtained through the full connection layer . For the hidden feature of the object image feature branch , the final visual embedding x O is obtained through the full connection layer .; for the relation branch, first concatenate the three hidden features and and input them into a fully connected layer to obtain higher-level relation hidden features Then concatenate the subject visual embedding x S and the object visual embedding x O with the relation hidden features and input them into two fully connected layers to obtain the relation visual embedding x P Finally, the visual embeddings x S , x O and x P are sent to the loss calculation, and the calculation of the image triple relation loss function is as follows:

[0051]

[0052] where L represents the loss value, N is the number of positive RoIs, K is the number of negative samples for each positive RoI, and the m parameter is set to 0.2, represents the i-th positive RoI and its corresponding j-th negative sample.

[0053] The text description branch network is based on Google's Word2Vec model, and each input text entity can obtain a multi-dimensional word vector through the Word2Vec model, and then the word vector is normalized.

[0054] The feature fusion is to obtain the text description feature by using the Bi-LSTM model based on the word vector obtained by the text description branch network, and then fuse the text description feature with the feature output by the image entity relation detection model to obtain high-level features; the text-image interaction loss function is set as follows: where, indicates the variance of the relation type, and the greater the variance, the smaller the loss proportion.

[0055] The power multi-modal entity relation extraction model based on transfer learning uses the maximum mean difference (MMD) method to compare the distribution difference between r s and r t The formula definition of MMD measurement is: where R s and R t represent the corresponding field high-level feature sets learned by the text open field data set and the power field data set respectively, is the i-th power field high-level feature r t , and n represents the power field data set R tThe number of middle-level features r t The number of middle-level features r The number of middle-level features r s , m represents the electric power field data set R s The number of middle-level features r s The number of middle-level features r The number of middle-level features r The number of middle-level features r

[0056] Since the device introduced in the second embodiment of the present application is the device used for implementing the method of the first embodiment of the present application, the specific structure and the deformation of the device can be understood by those skilled in the art based on the method introduced in the first embodiment of the present application, and thus will not be described here again. Any device used for the method of the first embodiment of the present application belongs to the scope of the present application.

[0057] The above merely describes the specific embodiments of the present application, and does not limit the patent scope of the present application. Any equivalent structural transformation based on the content of the specification and the drawings, or direct or indirect application in other related technical fields, is also included in the patent protection scope of the present application.

Claims

1. A power multi-modal entity relation extraction method based on transfer learning, characterized in that, Comprising the following steps: Collecting and preprocessing power text and image data; The cross-modal relationship feature extraction model is constructed: the multi-modal data in the power field is input into the cross-modal relationship feature extraction model, image features and text features are extracted, and are fused to obtain high-level features r of the multi-modal data in the power field t The multi-modal corpus in the open field is input into the cross-modal relationship feature extraction model to obtain high-level features r of the multi-modal corpus in the open field s Specifically, the cross-modal relationship feature extraction model includes an image entity relationship detection model, a text description branch network, and feature fusion; the image is input into the image entity relationship detection model to output relationship visual embedding, the text is input into the text description branch network to first obtain a word vector, and then the word vector is converted into a text description feature; the feature fusion fuses the text feature description and the relationship visual embedding to obtain high-level features; Inputting the multi-modal data in the power field into the cross-modal relationship feature extraction model outputs high-level features r t Inputting the multi-modal corpus in the open field into the cross-modal relationship feature extraction model outputs high-level features r s ; wherein the feature fusion is to obtain the word vector based on the text description branch network, obtain the text description feature by using the Bi-LSTM model, and then fuse the text description feature with the feature output by the image entity relationship detection model to obtain the high-level feature; the text-image interaction loss function is set as follows: wherein, the variance of the relationship type, and the greater the variance, the smaller the proportion of the loss. Building a power multi-modal entity relation extraction model based on transfer learning: the advanced features r s corresponding to the power field t Adaptation, so that the difference value between the advanced features r t is minimal s The advanced features r t with a difference value less than a preset threshold are selected s As a result, input the next layer; specifically, the power multi-modal entity relation extraction model based on transfer learning uses the method of maximum mean difference MMD to compare the distribution difference between r s and r t The formula definition of MMD measurement is: Where R s and R t represent the corresponding field advanced feature sets learned by the open field dataset and the power field dataset respectively, Table is the i-th advanced feature r t of the power field, n represents the number of advanced features r t in the power field dataset R t , Table is the i-th advanced feature r s of the open field, m represents the number of advanced features r s in the power field dataset R s , and H represents that this measurement method is carried out by mapping the advanced features to the reproducing kernel Hilbert space by the feature mapping function The new feature space after mapping by the function reflects the accuracy of MMD measurement; The classification relationship output layer processes the input high-level features through a softmax function and outputs a relationship recognition result.

2. The power multi-modal entity relation extraction method based on transfer learning according to claim 1, characterized in that: The image entity relationship detection model comprises a backbone network, a Faster R-CNN network, an RoI pooling layer and a full connection layer; an image is input into the backbone network to obtain global features of the image, then a subject, a relationship and an object are obtained through the Faster R-CNN layer, and corresponding feature embeddings Z S , Z P , Z O are obtained through the RoI pooling layer, finally, an intersection over union IoU is calculated, when the IoU>0, the boundary boxes of the subject and the object have intersection, and only then the subject and the object have relationship, picture features and boundary boxes corresponding to the relationship are obtained, after the features of the subject, the relationship and the object are obtained and the boundary regions are obtained, two full connection layers are connected after the subject image feature branch, the relationship branch and the object image feature branch, hidden features 1,2 are output after the full connection layer W and , for the hidden features of the subject image feature branch , the final visual embedding x S is obtained through the full connection layer , for the hidden features of the object image feature branch O ; for the relationship branch, first, the three hidden features and are spliced, and are input into the full connection layer to obtain higher-level relationship hidden features , then the subject visual embedding x S and the object visual embedding x O are spliced with the relationship hidden features , and are input into two full connection layers to obtain the relationship visual embedding x P .

3. The power multi-modal entity relation extraction method based on transfer learning according to claim 1, characterized in that: The text description branch network is based on a Word2Vec model of Google, each input text entity is converted into a multi-dimensional word vector through the Word2Vec model, and then the word vector is normalized.

4. An electric power multi-modal entity relation extraction device based on transfer learning, characterized by: Comprise: A preprocessing module for collecting and preprocessing power text and image data; The cross-modal relationship feature extraction model: inputting the multi-modal data in the power field into the cross-modal relationship feature extraction model, extracting image features and text features, and fusing to obtain high-level features r of the multi-modal data in the power field t , inputting the multi-modal corpus in the open field into the cross-modal relationship feature extraction model to obtain high-level features r of the multi-modal corpus in the open field s ; specifically, the cross-modal relationship feature extraction model includes an image entity relationship detection model, a text description branch network, and feature fusion; inputting the image into the image entity relationship detection model to output relationship visual embedding, inputting the text into the text description branch network to first obtain word vector, and then converting the word vector into text description feature, and the feature fusion fuses the text feature description and the relationship visual embedding to obtain high-level features; Inputting the multi-modal data in the power field into the cross-modal relationship feature extraction model outputs high-level features r t Inputting the multi-modal corpus in the open field into the cross-modal relationship feature extraction model outputs high-level features r s ; wherein the feature fusion is to obtain the word vector based on the text description branch network, obtain the text description feature by using the Bi-LSTM model, and then fuse the text description feature with the feature output by the image entity relationship detection model to obtain the high-level feature; the text-image interaction loss function is set as follows: wherein, the variance of the relationship type, and the greater the variance, the smaller the proportion of the loss. Electricity Multimodal Entity Relationship Extraction Model: High-level features r obtained from open-domain learning s High-level features r corresponding to the power sector t Adaptation enables higher-level features r t With high-level features r s The high-level feature r with the smallest difference value among them is selected when the difference value is less than a preset threshold. t With high-level features r s The result is input to the next layer; specifically, the power multimodal entity relation extraction model based on transfer learning uses the maximum mean difference (MMD) method to compare r. s and r t The distributional differences between them; the formula for the MMD metric is defined as: Where R s and R t These represent the sets of high-level features for their respective domains, obtained through training on the open-domain dataset and the power domain dataset presented in this paper. The table represents the high-level features r of the i-th power sector. t , where n represents the power sector dataset R t Medium and high-level features r t Quantity, Table r represents the high-level features of the i-th open domain. s m represents the power sector dataset R s Medium and high-level features r s The number of features, H represents the number of features measured by the feature mapping function. The mapping of high-level features to the reproducing kernel Hilbert space is performed, after a function. The new feature space after mapping reflects the accuracy of the MMD metric; A classification relationship output layer for processing input high-level features through a softmax function and outputting a relationship recognition result.

5. The power multi-modal entity relation extraction device based on transfer learning according to claim 4, characterized in that: The image entity relationship detection model comprises a backbone network, a FasterR-CNN network, an RoIpooling layer and a full connection layer; an image is input into the backbone network to obtain global features of the image, then a subject, a relationship and an object are obtained through the FasterR-CNN layer, and corresponding feature embeddings Z S P O are obtained through the RoIpooling layer, finally, an intersection-over-union IoU is calculated, when IoU>0, the boundary boxes of the subject and the object have intersection, and only then the subject and the object have relationship, picture features and boundary boxes corresponding to the relationship are obtained, after obtaining the feature embeddings and the boundary boxes of the subject, the relationship and the object, two full connection layers are connected after a subject image feature branch, a relationship branch and an object image feature branch respectively, and three intermediate hidden features 1,2 are output after the full connection layers For the hidden feature of the subject image feature branch, a final visual embedding x is obtained through a full connection layer S ; for the hidden feature of the object image feature branch, a final visual embedding x is obtained through a full connection layer O ; for the relationship branch, three hidden features and are spliced first, and are input into a full connection layer to obtain a higher-level relationship hidden feature , then the subject visual embedding x S and the object visual embedding x O are spliced with the relationship hidden feature , and are input into two full connection layers to obtain a relationship visual embedding x P ;​​​ The text description branch network is based on a Word2Vec model of Google, each input text entity is converted into a multi-dimensional word vector through the Word2Vec model, and then the word vector is normalized.

Images