Power scene graph generation method and device based on multi-modal data

Abstract

The invention provides an electric power scene graph generation method and device based on multi-modal data, and the method comprises the steps: receiving the multi-modal data which comprises an electric power image and an electric power text description; inputting the power image into a pre-trained power scene graph generation model to generate a power scene graph corresponding to the power image; the electric power scene graph is used for representing objects in the multi-modal data and visual relations thereof; inputting the power scene graph into a pre-trained multi-modal graph convolutional network to generate graph semantic features of the power image; inputting the power text description into a pre-trained text extraction model to generate text semantic features; and calculating the similarity between the graph semantic feature and the text semantic feature, and updating the electric power scene graph based on the similarity to obtain the updated electric power scene graph, so that when the multi-modal data is processed, the information of the graph modal and the text modal can be effectively integrated, and effective semantic association is established between the vision and the text.

Description

Technical Field

[0001] The present invention relates to the technical field of multimodal data processing, and in particular to a method and device for generating a power scene graph based on multimodal data. Background Art

[0002] Relation extraction is a basic task in the field of natural language processing and information extraction. Its goal is to identify and classify the semantic relationships between entity pairs in a given text. This technology has a wide range of applications in many fields, such as building knowledge graphs, question-answering systems, and text analysis. Relation extraction enables machines to understand the complex connections between entities in a text, thereby providing a deeper level of semantic understanding.

[0003] With the increasing amount of multimedia data, it has become a challenge to effectively extract semantic relationships from multimodal data such as images, videos, and text when generating power scene graphs. Most existing methods focus on a single modality, such as pure text or pure images, which results in the inability to fully utilize the correlation and complementarity between the two modal data in the relationship extraction in multimodal scenarios. In the process of generating power scene graphs from multimodal data, how to effectively integrate information from different modalities, especially to establish effective semantic associations between vision and language, is an urgent problem to be solved. Summary of the invention

[0004] The present invention provides a method and device for generating an electric power scene graph based on multimodal data, so as to solve the problem that most of the existing technologies focus on a single mode, such as pure text or pure image, resulting in the inability to fully utilize the correlation and complementarity between the two modal data in the relationship extraction in the multimodal scene. The present invention realizes the effective integration of information of different modalities when processing multimodal data, and establishes effective semantic associations between vision and text.

[0005] The present invention provides a method for generating a power scene graph based on multimodal data, comprising: receiving multimodal data, the multimodal data comprising an electric power image and an electric power text description; Inputting the power image into a pre-trained power scene graph generation model to generate a power scene graph corresponding to the power image; the power scene graph is used to characterize objects and their visual relationships in multimodal data; Inputting the power scene graph into a pre-trained multimodal graph convolutional network to generate graph semantic features of the power image; Inputting the electric power text description into a pre-trained text extraction model to generate text semantic features; The similarity between the graph semantic feature and the text semantic feature is calculated, and the power scene graph is updated based on the similarity to obtain an updated power scene graph.

[0006] According to the method for generating a power scene graph based on multimodal data provided by the present invention, the power scene graph generation model includes: an object recognition layer, a feature extraction layer, a visual relationship recognition layer and a power scene graph output layer; Inputting the power image into a pre-trained power scene graph generation model to generate a power scene graph corresponding to the power image specifically includes: Inputting the power image into the object recognition layer to obtain a plurality of objects in the power image and a relative position of each of the objects in the power image; Inputting each detected object into the feature extraction layer to extract visual features of the object; Inputting the visual features of each of the objects and the relative positions of the objects in the power image into the visual relationship recognition layer to detect the visual relationship between the objects; The objects and the visual relationships between the objects are input into the power scene graph output layer, and the power scene graph is output. The power scene graph includes multiple nodes and edges connecting the nodes, each node contains the object, and the edges represent the visual relationships between different objects.

[0007] According to the method for generating a power scene graph based on multimodal data provided by the present invention, the multimodal graph convolutional network includes a pre-trained visual feature extractor, a multimodal fusion layer and a graph convolutional network, and the graph convolutional network includes multiple graph convolutional layers; The power scene graph is input into a pre-trained multimodal graph convolutional network to generate graph semantic features of the power image, specifically including: Inputting the power scene graph into the visual feature extractor to obtain a visual feature vector of the object and a visual feature vector of the visual relationship; Inputting the visual feature vector of the object and the visual feature vector of the visual relationship into a multimodal fusion layer to obtain a fused feature representation of the object and a fused feature representation of the visual relationship respectively; The fused feature representation of the object and the fused feature representation of the visual relationship are input into a plurality of graph convolutional layers of the graph convolutional network for processing to obtain graph semantic features of the object and graph semantic features of the visual relationship.

[0008] According to the method for generating a power scene graph based on multimodal data provided by the present invention, the visual feature vector of the object and the visual feature vector of the visual relationship are input into the multimodal fusion layer to obtain the fused feature representation of the object and the fused feature representation of the visual relationship, respectively, which specifically includes: Inputting the visual feature vector of the object and the visual feature vector of the visual relationship into a multimodal fusion layer; fusing the visual feature vector of any one of the objects and the visual feature vector of the visual relationship associated with the object to obtain a fused feature representation of the object; The visual feature vectors of any two objects and the visual feature vector of the visual relationship connecting the two objects are fused to obtain a fused feature representation of the visual relationship.

[0009] According to the method for generating a power scene graph based on multimodal data provided by the present invention, the text extraction model includes a plurality of text encoding layers connected in sequence; The power text description is input into the pre-trained text extraction model to generate text semantic features, including: The electric power text description is input into a text extraction model, and is processed in sequence through a plurality of text encoding layers to generate the text semantic features.

[0010] According to the method for generating a power scene graph based on multimodal data provided by the present invention, the power scene graph includes a plurality of nodes and edges connecting the nodes, each node includes the object, and the edges represent the visual relationship between different objects; Calculating the similarity between the graph semantic features and the text semantic features, and updating the power scene graph based on the similarity to obtain an updated power scene graph, specifically including: respectively calculating the similarity between the graph semantic feature of the object and the text semantic feature and the similarity between the graph semantic feature of the visual relationship and the text semantic feature; If the similarity between the graph semantic feature of the object and the text semantic feature exceeds a first threshold, retaining the node corresponding to the object; otherwise, deleting the node and its connected edges; If the similarity between the graph semantic feature and the text semantic feature of the visual relationship exceeds a second threshold, the edge corresponding to the visual relationship is retained; otherwise, the edge corresponding to the visual relationship is deleted.

[0011] The present invention also provides a device for generating a power scene graph based on multimodal data, comprising: A data receiving module, used to receive multimodal data, wherein the multimodal data includes a power image and a power text description; A power scene graph generation module, used to input the power image into a pre-trained power scene graph generation model to generate a power scene graph corresponding to the power image; the power scene graph is used to characterize objects and their visual relationships in multimodal data; A graph semantic feature generation module, used for inputting the power scene graph into a pre-trained multimodal graph convolutional network to generate graph semantic features of the power image; A text semantic feature generation module, used for inputting the electric power text description into a pre-trained text extraction model to generate text semantic features; The power scene graph updating module is used to calculate the similarity between the graph semantic features and the text semantic features, and update the power scene graph based on the similarity to obtain an updated power scene graph.

[0012] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, a method for generating a power scene graph based on multimodal data as described above is implemented.

[0013] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the method for generating a power scene graph based on multimodal data as described in any one of the above is implemented.

[0014] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the method for generating a power scene graph based on multimodal data as described in any one of the above is implemented.

[0015] The present invention provides a method and device for generating a power scene graph based on multimodal data, which generates a power scene graph corresponding to the power image by inputting a power image into a pre-trained power scene graph generation model, and then generates graph semantic features of the power image by inputting the power scene graph into a pre-trained multimodal graph convolutional network; and generates text semantic features by inputting a power text description into a pre-trained text extraction model; and then calculates the similarity between the graph semantic features and the text semantic features, and updates the power scene graph based on the similarity to obtain an updated power scene graph, thereby achieving effective integration of information of the graph modality and the text modality when processing multimodal data, and establishing effective semantic associations between vision and text. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0017] Figure 1 It is one of the flow charts of the method for generating a power scene graph based on multimodal data provided by the present invention.

[0018] Figure 2 It is a structural schematic diagram of the multimodal semantic relationship extraction model provided by the present invention.

[0019] Figure 3 It is a structural schematic diagram of the power scene graph generation model provided by the present invention.

[0020] Figure 4 This is the second flow chart of the method for generating a power scene graph based on multimodal data provided by the present invention.

[0021] Figure 5 It is a schematic diagram of the structure of the multimodal graph convolutional network provided by the present invention.

[0022] Figure 6 This is the third flow chart of the method for generating a power scene graph based on multimodal data provided by the present invention.

[0023] Figure 7 It is a structural schematic diagram of a power scene graph generating device based on multimodal data provided by the present invention.

[0024] Figure 8 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0025] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0026] First, the terms involved in the embodiments of the present invention are schematically explained.

[0027] Mask R-CNN: A state-of-the-art computer vision algorithm for object detection and instance segmentation in images.

[0028] Graph Convolutional Network (GCNs): A deep learning model for processing graph-structured data. It extends the concept of convolutional neural network (CNN) to make it applicable to graph-structured data, such as social networks, molecular structures, and traffic networks. GCNs can capture the topological structure of nodes as well as the feature information of nodes, thereby achieving excellent performance in tasks such as node classification, graph classification, and link prediction. In order to encode the multimodal fusion feature graph, a K-layer GCN is adopted, and a new update mechanism is proposed to update the two types of nodes in different ways. Object nodes will generate object-level features, which can be regarded as first-order features of the image. The representation of relationship nodes can be enhanced by their neighboring object nodes. Therefore, relationship nodes are updated by aggregating information from their neighboring nodes, while object nodes are updated from themselves.

[0029] BERT: A pre-trained language model obtained by unsupervised training on a large amount of unlabeled text. It uses multiple encoding layers based on Transformer for bidirectional encoding. By building a labeled language model, BERT can randomly cover or replace any word in a sentence, allowing the model to predict the randomly covered part of the context and obtain a distributed contextual representation of the word.

[0030] Power scene graph: includes multiple nodes and edges connecting the nodes, wherein each node contains an object and the edges represent the visual relationship between different objects.

[0031] Combine the following Figure 1-Figure 6 A method for generating a power scene graph based on multimodal data according to an embodiment of the present invention is described.

[0032] Figure 1 This is one of the flow charts of the method for generating a power scene graph based on multimodal data provided by the present invention, which is used for a multimodal semantic relationship extraction model. Figure 2 ,The multimodal semantic relationship extraction model includes: power scene graph generation model, ,multimodal graph convolutional network and text extraction model.

[0033] like Figure 1 As shown, the method comprises the following steps: 101. Receive multimodal data, where the multimodal data includes an electric power image and an electric power text description.

[0034] There are many ways to obtain multimodal data, such as collecting high-definition images of power equipment, such as substations and transmission lines, from the power inspection system; at the same time, collecting inspection reports or maintenance logs corresponding to these images. These text materials contain descriptions of the image content.

[0035] Furthermore, after receiving the multimodal data, the multimodal data can also be preprocessed, such as image preprocessing, including scaling, cropping, rotation correction, etc., to ensure that the image meets the requirements of the model input; and sentence segmentation, word segmentation, stop word removal and other preprocessing operations on the text description.

[0036] Take the example of multimodal data including power images and power text descriptions. The power image is an image of on-site power equipment including transmission towers, wires, and insulators, and the text description is "Three sets of high-voltage insulators are installed on the top of the transmission tower, and the transmission line passes through from left to right."

[0037] 102. Input the power image into a pre-trained power scene graph generation model to generate a power scene graph corresponding to the power image; the power scene graph is used to characterize objects and their visual relationships in multimodal data.

[0038] In this process, object-level visual features are regarded as the basic representation of power image information, and the visual semantic features of power images are obtained by extracting the representation of objects. To achieve this goal, the input power image is first input into a pre-trained power scene graph generation model to generate a power scene graph of the input power image. This power scene graph consists of multiple nodes and edges connecting these nodes. Each node contains object features as its internal information, while the edges represent the visual relationships between different objects, such as dependence or connection.

[0039] The structure of a sentence (i.e., dependency tree) can provide important information for the relationship extraction model. Therefore, the power scene graph generation model of the embodiment of the present invention is intended to generate a unidirectional graph for the input power image, thereby providing structural information for multimodal power relationship extraction. It should be noted that in the power scene graph, the visual object features play the role of node features.

[0040] Specifically, the power image may be an image of power equipment, which may include power equipment such as transformers, electric poles, insulators, etc. The input power image is preprocessed, including resizing, normalization, etc., to meet the input requirements of the pre-trained model.

[0041] Use a pre-trained power scene graph generation model, such as Mask R-CNN, to identify individual objects in the image. For example, the model might identify the following objects: transformer, utility pole, insulator, and ground wire.

[0042] Visual features are then extracted for each detected object, which may include shape, size, texture, etc.

[0043] The power scene graph generation model predicts the visual relationship between every two objects. For example: the transformer is "next to the pole", the insulator is "hanging on the pole", and the ground wire is "connected to the bottom of the transformer".

[0044] Finally, based on the detected objects and predicted relationships, an electric scene graph is generated. The scene graph may look like this: [Transformer] --(nearby)-->[Electricity pole]; [Electricity pole] --(hanging on)-->[Insulator]; [Transformer] --(Connected to)-->[Ground wire].

[0045] Output: A power scene graph containing nodes (objects) and edges (visual relationships between objects).

[0046] The nodes include: Node 1: Transformer (contains the visual features of the transformer); Node 2: Telephone pole (contains the visual features of the telephone pole); Node 3: Insulator (contains the visual features of the insulator); Node 4: ground line (contains the visual features of the ground line); Edges include: Edge 1: located next to (the relationship between the transformer and the electric pole); Side 2: hanging on (the relationship between the pole and the insulator); Side 3: Connected to (relationship between transformer and ground wire).

[0047] In this way, the power scene graph generation model is able to extract rich visual information from power images and identify complex relationships between objects, providing a basis for subsequent semantic analysis and application.

[0048] 103. Input the power scene graph into a pre-trained multimodal graph convolutional network to generate graph semantic features of the power image.

[0049] Among them, the multimodal graph convolutional network can be a GCNs network. In order to encode the multimodal fusion feature map, a K-layer GCN is adopted, and a new update mechanism is proposed to update two types of nodes in different ways. Object nodes will generate object-level features, which can be regarded as first-order features of the image. The representation of relationship nodes can be enhanced by their neighboring object nodes. Therefore, relationship nodes are updated by aggregating information from their neighboring nodes, while object nodes are updated from themselves.

[0050] The multimodal graph convolutional network includes a pre-trained visual feature extractor, a multimodal fusion layer and a graph convolutional network, and the graph convolutional network includes multiple graph convolutional layers.

[0051] The features of nodes and edges in the power scene graph are input into a pre-trained visual feature extractor (such as CNN) to obtain the visual features of nodes and edges. In this embodiment, nodes include objects, and edges represent visual relationships between different objects.

[0052] Process the visual features of nodes and edges and fuse them to generate richer representations. Use fully connected layers and activation functions (such as ReLU) to integrate features.

[0053] The fused features are input into the graph convolutional network (GCN). Through multiple layers of GCN, each layer updates the feature representation of nodes and edges to capture local structural information.

[0054] After being processed by the graph convolutional network, each node and edge will have a new graph semantic feature representation. These feature representations can capture the contextual information of nodes (objects) and edges (relationships). Finally, the graph semantic features of the power image are output, including the graph semantic features of objects and the graph semantic features of visual relationships.

[0055] Through step 103, it is possible to generate graph semantic feature representations of various objects in the power image and the relationships between them, and these representations can be used for subsequent similarity calculations and updates of the power scene graph.

[0056] 104. Input the electric power text description into a pre-trained text extraction model to generate text semantic features.

[0057] The text extraction model can be a BERT model. The high-dimensional vector of each word is generated by the pre-trained BERT model. By concatenating the high-dimensional vectors of each word, the vector of the entire text description can be obtained. The encoding and high-dimensional vector form an embedding vector for each word. BERT is a language model obtained by unsupervised training on a large amount of unlabeled text. The encoder part based on Transformer performs bidirectional encoding. By building a marked language model, BERT can randomly cover or replace any word in a sentence, allowing the model to predict the randomly covered part of the context and obtain a distributed contextual representation of the word.

[0058] For example, the power text description fed into the text extraction model: "The maintenance team inspected the transformer located in the third block and recorded that it was operating at 85°C, with no obvious damage found." First, tokenize the text: "The maintenance / team / inspected / the / transformer / located / in / the / third / block / and / recorded / that / it / was / operating / at / 85°C / with / no / obvious / damage / found."

[0059] Then the processed text is input into the BERT model to obtain the embedding representation of each word. The context-dependent embedding of the text is generated through the encoding layer of the BERT model. The output of the [CLS] tag is used as the semantic feature vector of the text. For example: [0.25, -0.45, 0.13, ...] (illustrative vector). This vector can be used for subsequent similarity calculation with the graph semantic features of the power scene graph.

[0060] 105. Calculate the similarity between the graph semantic feature and the text semantic feature, and update the power scene graph based on the similarity to obtain an updated power scene graph.

[0061] In the embodiment of the present invention, a similarity function is designed to optimize the semantic representation of the power scene graph, thereby updating the power scene graph. .

[0062] In order to measure the similarity of two semantic features (image semantic features and text semantic features) under different modalities, the following similarity function is defined:

[0063] in, is the fully connected layer, and They represent the semantic feature representations of the object nodes and edges of the Kth layer respectively, N represents the total number of nodes in the power scene graph, and cos represents the cosine similarity.

[0064] The present invention uses a triplet loss function to optimize the multimodal semantic relationship extraction model. The loss function constrains the similarity score of the matched power image-power text pair to be greater than the similarity score of the unmatched pair. The specific formula is as follows:

[0065] Among them, power image and power sentence description is matched, that is, the power sentence describes It is a description of the power image . Power Image and power sentence description is not matched, that is, the power sentence describes Not describing the image of electricity , randomly sampled from other power image descriptions.

[0066] represents the similarity score of the matching pair, i.e., the similarity between the power image f and the corresponding description text t; represents the similarity score of the unmatched pair, i.e., the similarity between the power image f and a randomly sampled, unmatched description text t′.

[0067] The training ends by optimizing the above loss function until the entire multimodal semantic relationship extraction model converges.

[0068] During the training process, the similarity function and the triplet loss function work together on the entire system to optimize the parameters of all components. The purpose is to enable the entire multimodal semantic relationship extraction model to accurately generate accurate power scene graphs from power images, extract accurate text semantic features from text descriptions, and calculate and use the similarity between graph semantic features and text semantic features to update the power scene graph.

[0069] In specific use, for example, graph semantic feature vector (from power scene graph): h g =[0.1, 0.3, 0.4], text semantic feature vector (from the BERT model of power text description): h t =[0.2,0.2,0.6]. g and h t Perform L2 normalization, then calculate the cosine similarity, and set the threshold to 0.9. If the similarity is greater than 0.9, the two are considered similar enough and there is no need to update the scene graph; if the similarity is less than 0.9, check whether the objects and relationships in the scene graph are consistent with the text description, and update the power scene graph: for example, delete some nodes and / or edges in the power scene graph.

[0070] The method for generating a power scene graph based on multimodal data provided in an embodiment of the present invention generates a power scene graph corresponding to the power image by inputting a power image into a pre-trained power scene graph generation model, and then inputting the power scene graph into a pre-trained multimodal graph convolutional network to generate graph semantic features of the power image; inputting the power text description into a pre-trained text extraction model to generate text semantic features; then calculating the similarity between the graph semantic features and the text semantic features, and updating the power scene graph based on the similarity to obtain an updated power scene graph, thereby achieving effective integration of information of the graph modality and the text modality when processing multimodal data, and establishing effective semantic associations between vision and text.

[0071] Specifically, see Figure 3,The power scene graph generation model includes: object recognition layer, feature extraction layer, visual relationship recognition layer and power scene graph output layer.

[0072] See also Figure 4 , step 102 specifically includes: 401. Input the power image into the object recognition layer to obtain multiple objects in the power image and the relative position of each object in the power image.

[0073] After the object recognition layer recognizes the power image, the position and bounding box of each object are determined, and finally a list of power equipment objects is obtained, including transformers, poles, insulators, etc.

[0074] For example, if the input power image is a high-definition picture of a substation, 1 transformer, 3 poles and 5 insulators can be identified.

[0075] 402. Input each detected object into the feature extraction layer to extract visual features of the object.

[0076] The feature extraction layer extracts visual features for each object. These features may include shape descriptors, color histograms, texture features, etc. These features are used for subsequent node representations, and finally output the visual feature vector of each object. For example, the feature vector of the transformer is [0.5, 0.2, 0.3].

[0077] 403. Input the visual features of each of the objects and the relative position of the object in the power image into the visual relationship recognition layer to detect the visual relationship between the objects.

[0078] The spatial relationships between objects are determined through the visual relationship recognition layer. The recognized relationships may include "is above", "is connected to", "is adjacent to", etc. The output is a list of visual relationships between objects, each with a type label.

[0079] Continuing with the previous example, the relationship between the transformer and pole 1 is determined to be "located next to", and the relationship between pole 1 and the insulator is determined to be "supported".

[0080] 404. Input the objects and the visual relationships between the objects into the power scene graph output layer, and output the power scene graph.

[0081] The power scene graph includes a plurality of nodes and edges connecting the nodes, each node includes the object, and the edges represent visual relationships between different objects.

[0082] In the process of generating power scene graphs, the power scene graph generation model Mask R-CNN is used to obtain the input power image objects and the visual relationships between them. Since each relationship is unidirectional, similar to a dependency tree, each object is also associated with other objects from the power image. Therefore, the power scene graph representation of the input power image can be obtained. . It contains various relationship pairs between objects detected in the power image and can be expressed as follows:

[0083]

[0084] in, Indicates the first The node corresponding to the object; Indicates the number of objects detected; Indicates The object and The edge of an object.

[0085] Through steps 401 to 404, a structured power scene graph can be generated from the original power image, and the scene graph can be used for more in-depth semantic analysis and information extraction later.

[0086] Alternatively, see Figure 5 The multimodal graph convolutional network includes a pre-trained visual feature extractor, a multimodal fusion layer and a graph convolutional network, wherein the graph convolutional network includes multiple graph convolutional layers.

[0087] See also Figure 6 , step 103 specifically includes: 601. Input the power scene graph into the visual feature extractor to obtain a visual feature vector of the object and a visual feature vector of the visual relationship.

[0088] In this embodiment, a pre-trained visual feature extractor is used to encode the power image region into a feature vector, which can be a pre-trained CNN network or an object detector (such as Mask R-CNN). In the visual power scene graph, each node is encoded as a dimension of The visual feature vector of the object node , whose visual feature vector is extracted from its corresponding power image area; for relationship nodes , whose visual feature vector is from and The joint image region is extracted.

[0089] 602. Input the visual feature vector of the object and the visual feature vector of the visual relationship into a multimodal fusion layer to obtain a fused feature representation of the object and a fused feature representation of the visual relationship, respectively.

[0090] After obtaining the visual feature vector of each node and the visual feature vector of the visual relationship, they need to be fused into a unified representation. Therefore, a multimodal fusion feature map is generated. Specifically, the visual feature vector of each node and the visual feature vector of the visual relationship are concatenated, and then converted into a fused feature representation through a fusion layer.

[0091] Specifically, step 602 includes: inputting the visual feature vector of the object and the visual feature vector of the visual relationship into the multimodal fusion layer; fusing the visual feature vector of any object and the visual feature vector of the visual relationship associated with the object to obtain a fused feature representation of the object; fusing the visual feature vectors of any two objects and the visual feature vector of the visual relationship connecting the two objects to obtain a fused feature representation of the visual relationship.

[0092] The specific formula is as follows:

[0093]

[0094] in, is the fusion feature representation of the object, is the fusion feature representation of visual relations, is the trainable parameter of the fusion layer, Representation and Object Nodes The feature vector of the associated relationship node.

[0095] v on Represents the object node o n This is the visual feature vector extracted from the object detected in the power image, which may include visual information such as color, shape, texture, etc.

[0096] v oi Represents the object node o i The visual feature vector, v oj Represents the object node o j The visual feature vector of Represents the feature vector of the edge in the power scene graph, which connects the object node o i and object node o j .

[0097] 603. Input the fused feature representation of the object and the fused feature representation of the visual relationship into multiple graph convolutional layers of the graph convolutional network for processing to obtain the graph semantic features of the object and the graph semantic features of the visual relationship.

[0098] In this embodiment, a K-layer GCN is used to encode the multimodal fusion feature map, and a new update mechanism is proposed to update the two types of nodes in different ways. Object nodes will generate object-level features, which can be regarded as first-order features of the image. The representation of relationship nodes can be enhanced by their neighboring object nodes. Therefore, relationship nodes are updated by aggregating information from their neighboring nodes, while object nodes are updated from themselves. Specifically, given a multimodal fusion feature map , the k-th layer of GCN is calculated as follows

[0099] in, It is a fully connected layer, and the initial state is the fusion feature. .

[0100] Finally, the output of the K-layer GCN is an encoded visual feature map, which contains two types of vertices: and .in, Represents the object node o in the Kth layer of the graph convolutional network n The characteristic representation of Represents the edge e in the Kth layer of the graph convolutional network i,j The feature representation of .

[0101] Taking the electric power scene graph whose scene graph content includes transformers, electric poles and insulators as an example, the relationship between the transformer and the electric pole is "located next to", and the relationship between the electric pole and the insulator is "support".

[0102] In step 601, a visual feature extractor CNN is used to extract visual features from the region of each object (transformer, pole, insulator). For example, the visual feature vector of the transformer is [0.1, 0.3, 0.7].

[0103] In step 602, the visual features of the transformer are fused with its category label to generate a fused feature vector. For example, the feature vector of the transformer after fusion is [0.1, 0.35, 0.7].

[0104] In step 603, the fused feature vector is input into the graph convolutional network GCN and processed by two layers of graph convolution. The first layer updates the graph semantic features of the transformer to [0.2, 0.4, 0.5], and the second layer further updates it to [0.25, 0.45, 0.55].

[0105] Through these steps, high-level graph semantic features of each object and relationship can be generated from the power scene graph. These features capture rich contextual information and provide a basis for subsequent similarity calculation and scene graph update.

[0106] Specifically, step 104 includes: inputting the electric power text description into a text extraction model, processing it through a plurality of text encoding layers in sequence, and generating the text semantic features.

[0107] For example, the electric power text description is: "Three sets of high-voltage insulators are installed on the top of the transmission tower, and the transmission line passes through from left to right." The text is segmented as: "Three sets of high-voltage insulators are installed on the top of the transmission tower, and the transmission line passes through from left to right."

[0108] Input the processed text into the BERT model to obtain the embedding representation of each word. Generate the context-sensitive embedding of the text through the encoding layer of the BERT model. Use the output of the [CLS] tag as the semantic feature vector of the text, for example: [0.25, -0.45, 0.13, ...]. This vector can be used for subsequent similarity calculation with the graph semantic features of the power scene graph.

[0109] Specifically, step 105 includes: respectively calculating the similarity between the graph semantic feature of the object and the text semantic feature and the similarity between the graph semantic feature of the visual relationship and the text semantic feature; if the similarity between the graph semantic feature of the object and the text semantic feature exceeds a first threshold, retaining the node corresponding to the object; otherwise, deleting the node and the edges connected to it; if the similarity between the graph semantic feature of the visual relationship and the text semantic feature exceeds a second threshold, retaining the edge corresponding to the visual relationship; otherwise, deleting the edge corresponding to the visual relationship.

[0110] In a specific example, after the training is completed, the power image f is input to the entire multimodal semantic relationship extraction model to obtain the corresponding power scene graph Its semantic features ; Input the matching power sentence description t to the entire multimodal semantic relationship extraction model to obtain the text semantic features .

[0111] Specifically, by calculating the power scenario graph The semantic features of each node and edge in the text are The similarity of the power scene graph Update, the update rules are as follows: if , then the power scene diagram Reserved Node Otherwise, delete the node and its connected edges . is the first threshold value set; if , then the power scene diagram Keep Edges On the contrary, deleting the edge . The second threshold is set.

[0112] Through step 105, irrelevant or low-relevance nodes and edges can be deleted, making the power scene graph more concise and accurate; by accurately calculating and comparing similarities, the consistency between the scene graph and the text description is ensured, thereby improving the accuracy of information extraction.

[0113] In addition, it also achieves effective fusion of information from image and text modalities, enhancing the model's ability to understand and process multimodal data.

[0114] Through the embodiments of the present invention, it is possible to ensure that the power scenario diagram accurately reflects the content of the text description, improve the quality and availability of data, and lay a solid foundation for subsequent analysis and application.

[0115] The following describes an electric power scene graph generation device based on multimodal data provided by an embodiment of the present invention. The electric power scene graph generation device based on multimodal data described below and the electric power scene graph generation method based on multimodal data described above can refer to each other.

[0116] The embodiment of the present invention provides a device for generating a power scene graph based on multimodal data, see Figure 7 ,include: The data receiving module 701 is used to receive multimodal data, wherein the multimodal data includes a power image and a power text description; The power scene graph generation module 702 is used to input the power image into a pre-trained power scene graph generation model to generate a power scene graph corresponding to the power image; the power scene graph is used to characterize objects and their visual relationships in multimodal data; A graph semantic feature generation module 703, used for inputting the power scene graph into a pre-trained multimodal graph convolutional network to generate graph semantic features of the power image; A text semantic feature generation module 704 is used to input the power text description into a pre-trained text extraction model to generate text semantic features; The power scene graph updating module 705 is used to calculate the similarity between the graph semantic features and the text semantic features, and update the power scene graph based on the similarity to obtain an updated power scene graph.

[0117] Optionally, the power scene graph generation model includes: an object recognition layer, a feature extraction layer, a visual relationship recognition layer, and a power scene graph output layer; The power scene graph generation module 702 is specifically used for: Inputting the power image into the object recognition layer to obtain a plurality of objects in the power image and a relative position of each of the objects in the power image; Inputting each detected object into the feature extraction layer to extract visual features of the object; Inputting the visual features of each of the objects and the relative positions of the objects in the power image into the visual relationship recognition layer to detect the visual relationship between the objects; The objects and the visual relationships between the objects are input into the power scene graph output layer, and the power scene graph is output. The power scene graph includes multiple nodes and edges connecting the nodes, each node contains the object, and the edges represent the visual relationships between different objects.

[0118] Optionally, the multimodal graph convolutional network includes a pre-trained visual feature extractor, a multimodal fusion layer and a graph convolutional network, and the graph convolutional network includes a plurality of graph convolutional layers; The graph semantic feature generation module 703 is specifically used for: Inputting the power scene graph into the visual feature extractor to obtain a visual feature vector of the object and a visual feature vector of the visual relationship; Inputting the visual feature vector of the object and the visual feature vector of the visual relationship into a multimodal fusion layer to obtain a fused feature representation of the object and a fused feature representation of the visual relationship respectively; The fused feature representation of the object and the fused feature representation of the visual relationship are input into a plurality of graph convolutional layers of the graph convolutional network for processing to obtain graph semantic features of the object and graph semantic features of the visual relationship.

[0119] Optionally, the graph semantic feature generation module 703 is specifically used to: input the visual feature vector of the object and the visual feature vector of the visual relationship into the multimodal fusion layer; fuse the visual feature vector of any object and the visual feature vector of the visual relationship associated with the object to obtain a fused feature representation of the object; fuse the visual feature vectors of any two objects and the visual feature vector of the visual relationship connecting the two objects to obtain a fused feature representation of the visual relationship.

[0120] Optionally, the text extraction model includes a plurality of text encoding layers connected in sequence; The text semantic feature generation module 704 is specifically used to: input the power text description into the text extraction model, process it through the multiple text encoding layers in sequence, and generate the text semantic features.

[0121] Optionally, the power scene graph includes a plurality of nodes and edges connecting the nodes, each node includes the object, and the edges represent visual relationships between different objects; The power scene graph updating module 705 is specifically used for: respectively calculating the similarity between the graph semantic feature of the object and the text semantic feature and the similarity between the graph semantic feature of the visual relationship and the text semantic feature; If the similarity between the graph semantic feature of the object and the text semantic feature exceeds a first threshold, retaining the node corresponding to the object; otherwise, deleting the node and its connected edges; If the similarity between the graph semantic feature and the text semantic feature of the visual relationship exceeds a second threshold, the edge corresponding to the visual relationship is retained; otherwise, the edge corresponding to the visual relationship is deleted.

[0122] The electric power scene graph generation device based on multimodal data provided by an embodiment of the present invention generates an electric power scene graph corresponding to the electric power image by inputting an electric power image into a pre-trained electric power scene graph generation model, and generates graph semantic features of the electric power image by inputting the electric power scene graph into a pre-trained multimodal graph convolutional network; the electric power text description is input into a pre-trained text extraction model to generate text semantic features; then the similarity between the graph semantic features and the text semantic features is calculated, and the electric power scene graph is updated based on the similarity to obtain an updated electric power scene graph, thereby achieving effective integration of information of the graph modality and the text modality when processing multimodal data, and establishing effective semantic associations between vision and text.

[0123] Figure 8 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 8As shown, the electronic device may include: a processor 810, a communication interface 820, a memory 830 and a communication bus 840, wherein the processor 810, the communication interface 820 and the memory 830 communicate with each other through the communication bus 840. The processor 810 may call the logic instructions in the memory 830 to execute a method for generating a power scene graph based on multimodal data, the method comprising: receiving multimodal data, the multimodal data comprising a power image and a power text description; inputting the power image into a pre-trained power scene graph generation model to generate a power scene graph corresponding to the power image; the power scene graph is used to characterize objects and their visual relationships in the multimodal data; inputting the power scene graph into a pre-trained multimodal graph convolutional network to generate graph semantic features of the power image; inputting the power text description into a pre-trained text extraction model to generate text semantic features; calculating the similarity between the graph semantic features and the text semantic features, and updating the power scene graph based on the similarity to obtain an updated power scene graph.

[0124] In addition, the logic instructions in the above-mentioned memory 830 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0125] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the power scene graph generation method based on multimodal data provided by the above methods, the method including: receiving multimodal data, the multimodal data including a power image and a power text description; inputting the power image into a pre-trained power scene graph generation model to generate a power scene graph corresponding to the power image; the power scene graph is used to characterize objects and their visual relationships in the multimodal data; inputting the power scene graph into a pre-trained multimodal graph convolutional network to generate graph semantic features of the power image; inputting the power text description into a pre-trained text extraction model to generate text semantic features; calculating the similarity between the graph semantic features and the text semantic features, and updating the power scene graph based on the similarity to obtain an updated power scene graph.

[0126] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the power scene graph generation method based on multimodal data provided by the above-mentioned methods, the method comprising: receiving multimodal data, the multimodal data comprising a power image and a power text description; inputting the power image into a pre-trained power scene graph generation model to generate a power scene graph corresponding to the power image; the power scene graph is used to characterize objects and their visual relationships in the multimodal data; inputting the power scene graph into a pre-trained multimodal graph convolutional network to generate graph semantic features of the power image; inputting the power text description into a pre-trained text extraction model to generate text semantic features; calculating the similarity between the graph semantic features and the text semantic features, and updating the power scene graph based on the similarity to obtain an updated power scene graph.

[0127] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0128] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0129] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for generating a power scene graph based on multimodal data, characterized in that: include: receiving multimodal data, the multimodal data comprising an electric power image and an electric power text description; Inputting the power image into a pre-trained power scene graph generation model to generate a power scene graph corresponding to the power image; the power scene graph is used to characterize objects and their visual relationships in multimodal data; Inputting the power scene graph into a pre-trained multimodal graph convolutional network to generate graph semantic features of the power image; Inputting the electric power text description into a pre-trained text extraction model to generate text semantic features; The similarity between the graph semantic feature and the text semantic feature is calculated, and the power scene graph is updated based on the similarity to obtain an updated power scene graph.

2. The method for generating a power scene graph based on multimodal data according to claim 1, characterized in that: The power scene graph generation model includes: an object recognition layer, a feature extraction layer, a visual relationship recognition layer and a power scene graph output layer; Inputting the power image into a pre-trained power scene graph generation model to generate a power scene graph corresponding to the power image specifically includes: Inputting the power image into the object recognition layer to obtain a plurality of objects in the power image and a relative position of each of the objects in the power image; Inputting each detected object into the feature extraction layer to extract visual features of the object; Inputting the visual features of each of the objects and the relative positions of the objects in the power image into the visual relationship recognition layer to detect the visual relationship between the objects; The objects and the visual relationships between the objects are input into the power scene graph output layer, and the power scene graph is output. The power scene graph includes multiple nodes and edges connecting the nodes, each node contains the object, and the edges represent the visual relationships between different objects.

3. The method for generating a power scene graph based on multimodal data according to claim 1, characterized in that: The multimodal graph convolutional network includes a pre-trained visual feature extractor, a multimodal fusion layer and a graph convolutional network, and the graph convolutional network includes multiple graph convolutional layers; The power scene graph is input into a pre-trained multimodal graph convolutional network to generate graph semantic features of the power image, specifically including: Inputting the power scene graph into the visual feature extractor to obtain a visual feature vector of the object and a visual feature vector of the visual relationship; Inputting the visual feature vector of the object and the visual feature vector of the visual relationship into a multimodal fusion layer to obtain a fused feature representation of the object and a fused feature representation of the visual relationship respectively; The fused feature representation of the object and the fused feature representation of the visual relationship are input into a plurality of graph convolutional layers of the graph convolutional network for processing to obtain graph semantic features of the object and graph semantic features of the visual relationship.

4. The method for generating a power scene graph based on multimodal data according to claim 3, characterized in that: Inputting the visual feature vector of the object and the visual feature vector of the visual relationship into a multimodal fusion layer to obtain a fused feature representation of the object and a fused feature representation of the visual relationship, respectively, specifically comprising: Inputting the visual feature vector of the object and the visual feature vector of the visual relationship into a multimodal fusion layer; fusing the visual feature vector of any one of the objects and the visual feature vector of the visual relationship associated with the object to obtain a fused feature representation of the object; The visual feature vectors of any two objects and the visual feature vector of the visual relationship connecting the two objects are fused to obtain a fused feature representation of the visual relationship.

5. The method for generating a power scene graph based on multimodal data according to claim 1, characterized in that: The text extraction model includes a plurality of text encoding layers connected in sequence; The power text description is input into the pre-trained text extraction model to generate text semantic features, including: The electric power text description is input into a text extraction model, and is processed in sequence through a plurality of text encoding layers to generate the text semantic features.

6. The method for generating a power scene graph based on multimodal data according to claim 3, characterized in that: The power scene graph includes a plurality of nodes and edges connecting the nodes, each node includes the object, and the edges represent visual relationships between different objects; Calculating the similarity between the graph semantic features and the text semantic features, and updating the power scene graph based on the similarity to obtain an updated power scene graph, specifically including: Calculating the similarity between the graph semantic feature of the object and the text semantic feature and the similarity between the graph semantic feature of the visual relationship and the text semantic feature respectively; If the similarity between the graph semantic feature of the object and the text semantic feature exceeds a first threshold, retaining the node corresponding to the object; otherwise, deleting the node and its connected edges; If the similarity between the graph semantic feature and the text semantic feature of the visual relationship exceeds a second threshold, the edge corresponding to the visual relationship is retained; otherwise, the edge corresponding to the visual relationship is deleted.

7. A device for generating a power scene graph based on multimodal data, characterized in that: include: A data receiving module, used to receive multimodal data, wherein the multimodal data includes a power image and a power text description; A power scene graph generation module, used to input the power image into a pre-trained power scene graph generation model to generate a power scene graph corresponding to the power image; the power scene graph is used to characterize objects and their visual relationships in multimodal data; A graph semantic feature generation module, used for inputting the power scene graph into a pre-trained multimodal graph convolutional network to generate graph semantic features of the power image; A text semantic feature generation module, used for inputting the electric power text description into a pre-trained text extraction model to generate text semantic features; The power scene graph updating module is used to calculate the similarity between the graph semantic features and the text semantic features, and update the power scene graph based on the similarity to obtain an updated power scene graph.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the method for generating a power scene graph based on multimodal data as described in any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for generating a power scene graph based on multimodal data as claimed in any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method for generating a power scene graph based on multimodal data as claimed in any one of claims 1 to 6 is implemented.

Images