A power scene graph generation method and device based on multi-modal data

Abstract

The application provides a power scene graph generation method and device based on multi-modal data, wherein the method comprises: receiving multi-modal data, wherein the multi-modal data comprises a power image and a power text description; inputting the power image into a pre-trained power scene graph generation model to generate a power scene graph corresponding to the power image; the power scene graph is used to represent objects and their visual relationships in the multi-modal data; inputting the power scene graph into a pre-trained multi-modal graph convolution network to generate a graph semantic feature of the power image; inputting the power text description into a pre-trained text extraction model to generate a text semantic feature; calculating the similarity of the graph semantic feature and the text semantic feature, and updating the power scene graph based on the similarity to obtain an updated power scene graph, so that the information of the graph mode and the text mode can be effectively integrated when processing the multi-modal data, and effective semantic association between the vision and the text can be established.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of multi-modal data processing, and particularly relates to a power scene graph generation method and device based on multi-modal data. BACKGROUND

[0002] Relation extraction is a fundamental task in natural language processing and information extraction, aiming to identify and classify semantic relationships between entity pairs in given texts. This technology has wide applications in building knowledge graphs, question-answering systems, text analysis, and other fields. Relation extraction enables machines to understand the complex connections between entities in texts, providing deeper semantic understanding.

[0003] With the increasing amount of multimedia data, how to effectively extract semantic relationships from multi-modal data such as images, videos, and texts when generating power scene graphs has become a challenge. Existing methods mostly focus on a single modality, such as pure text or pure image, resulting in the inability to fully utilize the relevance and complementarity between two modal data in multi-modal scene relation extraction. In the process of generating power scene graphs through multi-modal data, how to effectively integrate information from different modalities, especially establishing effective semantic associations between vision and language, is a problem that needs to be solved urgently. SUMMARY

[0004] The present application provides a power scene graph generation method and device based on multi-modal data, to solve the defect that most existing technologies focus on a single modality, such as pure text or pure image, resulting in the inability to fully utilize the relevance and complementarity between two modal data in multi-modal scene relation extraction, effectively integrating information from different modalities when processing multi-modal data, and establishing effective semantic associations between vision and text.

[0005] The present application provides a power scene graph generation method based on multi-modal data, comprising:

[0006] receiving multi-modal data, the multi-modal data comprising power images and power text descriptions;

[0007] inputting the power images into a pre-trained power scene graph generation model to generate power scene graphs corresponding to the power images; the power scene graphs are used to represent objects and their visual relationships in multi-modal data;

[0008] inputting the power scene graphs into a pre-trained multi-modal graph convolution network to generate graph semantic features of the power images;

[0009] inputting the power text descriptions into a pre-trained text extraction model to generate text semantic features;

[0010] Calculate the similarity between the graph semantic features and the text semantic features, and update the power scene graph based on the similarity to obtain the updated power scene graph.

[0011] According to the power scene graph generation method based on multimodal data provided by the present invention, the power scene graph generation model includes: an object recognition layer, a feature extraction layer, a visual relationship recognition layer, and a power scene graph output layer;

[0012] The power image is input into a pre-trained power scene image generation model to generate a power scene image corresponding to the power image, specifically including:

[0013] The power image is input into the object recognition layer to obtain multiple objects in the power image and the relative position of each object in the power image;

[0014] Each detected object is input into the feature extraction layer to extract the visual features of the object;

[0015] The visual features of each object and its relative position in the power image are input to the visual relationship recognition layer to detect the visual relationship between the objects.

[0016] The objects and the visual relationships between them are input into the power scene graph output layer, and the power scene graph is output. The power scene graph includes multiple nodes and edges connecting the nodes. Each node contains the object, and the edges represent the visual relationships between different objects.

[0017] According to the power scene graph generation method based on multimodal data provided by the present invention, the multimodal graph convolutional network includes a pre-trained visual feature extractor, a multimodal fusion layer and a graph convolutional network, wherein the graph convolutional network includes multiple graph convolutional layers;

[0018] The power scene image is input into a pre-trained multimodal graph convolutional network to generate graph semantic features of the power image, specifically including:

[0019] The power scene image is input into the visual feature extractor to obtain the visual feature vector of the object and the visual feature vector of the visual relationship;

[0020] The visual feature vector of the object and the visual feature vector of the visual relationship are input into the multimodal fusion layer to obtain the fused feature representation of the object and the fused feature representation of the visual relationship, respectively.

[0021] The fused feature representation of the object and the fused feature representation of the visual relationship are input into multiple graph convolutional layers of the graph convolutional network for processing to obtain the graph semantic features of the object and the graph semantic features of the visual relationship.

[0022] According to the power scene map generation method based on multimodal data provided by the present invention, the visual feature vector of the object and the visual feature vector of the visual relationship are input into a multimodal fusion layer to obtain the fused feature representation of the object and the fused feature representation of the visual relationship, respectively. Specifically, this includes:

[0023] The visual feature vectors of the object and the visual feature vectors of the visual relationship are input into the multimodal fusion layer;

[0024] The fused feature representation of the object is obtained by fusing the visual feature vector of any of the objects and the visual feature vector of the visual relationship associated with the object.

[0025] The visual feature vectors of any two objects and the visual feature vectors connecting the visual relationship between the two objects are fused to obtain the fused feature representation of the visual relationship.

[0026] According to the power scene map generation method based on multimodal data provided by the present invention, the text extraction model includes multiple text encoding layers connected in sequence;

[0027] The electrical text description is input into a pre-trained text extraction model to generate text semantic features, specifically including:

[0028] The power text description is input into the text extraction model and processed sequentially through multiple text encoding layers to generate the text semantic features.

[0029] According to the power scene graph generation method based on multimodal data provided by the present invention, the power scene graph includes multiple nodes and edges connecting the nodes, each node contains the object, and the edges represent the visual relationship between different objects;

[0030] Calculate the similarity between the semantic features of the graph and the semantic features of the text, and update the power scene graph based on the similarity to obtain the updated power scene graph, specifically including:

[0031] Calculate the similarity between the graph semantic features and the text semantic features of the object, and the similarity between the graph semantic features and the text semantic features of the visual relationship, respectively.

[0032] If the similarity between the graph semantic features of the object and the text semantic features exceeds a first threshold, then the node corresponding to the object is retained; otherwise, the node and its connected edges are deleted.

[0033] If the similarity between the graph semantic features of the visual relationship and the text semantic features exceeds a second threshold, then the edge corresponding to the visual relationship is retained; otherwise, the edge corresponding to the visual relationship is deleted.

[0034] The present invention also provides a power scene map generation device based on multimodal data, comprising:

[0035] A data receiving module is used to receive multimodal data, including power images and power text descriptions;

[0036] The power scene graph generation module is used to input the power image into a pre-trained power scene graph generation model to generate a power scene graph corresponding to the power image; the power scene graph is used to represent objects and their visual relationships in multimodal data;

[0037] The graph semantic feature generation module is used to input the power scene graph into a pre-trained multimodal graph convolutional network to generate graph semantic features of the power image;

[0038] The text semantic feature generation module is used to input the power text description into a pre-trained text extraction model to generate text semantic features;

[0039] The power scene graph update module is used to calculate the similarity between the graph semantic features and the text semantic features, and update the power scene graph based on the similarity to obtain the updated power scene graph.

[0040] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the power scene map generation method based on multimodal data as described above.

[0041] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the power scene diagram generation method based on multimodal data as described above.

[0042] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the power scene map generation method based on multimodal data as described above.

[0043] The present invention provides a method and apparatus for generating power scene maps based on multimodal data. This method generates a power scene map corresponding to the power image by inputting a power image into a pre-trained power scene map generation model. The power scene map is then input into a pre-trained multimodal graph convolutional network to generate graph semantic features of the power image. Power text descriptions are input into a pre-trained text extraction model to generate text semantic features. The similarity between the graph semantic features and the text semantic features is then calculated, and the power scene map is updated based on this similarity to obtain an updated power scene map. This allows for the effective integration of graph and text modal information when processing multimodal data, establishing an effective semantic association between visual and textual data. Attached Figure Description

[0044] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0045] Figure 1 This is one of the flowcharts illustrating the power scene diagram generation method based on multimodal data provided by the present invention.

[0046] Figure 2 This is a schematic diagram of the structure of the multimodal semantic relation extraction model provided by the present invention.

[0047] Figure 3 This is a schematic diagram of the structure of the power scene diagram generation model provided by the present invention.

[0048] Figure 4 This is the second flowchart of the power scene diagram generation method based on multimodal data provided by the present invention.

[0049] Figure 5 This is a schematic diagram of the structure of the multimodal graph convolutional network provided by the present invention.

[0050] Figure 6 This is the third flowchart of the power scene diagram generation method based on multimodal data provided by the present invention.

[0051] Figure 7 This is a schematic diagram of the power scene map generation device based on multimodal data provided by the present invention.

[0052] Figure 8 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0053] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0054] First, the terminology used in the embodiments of the present invention will be explained illustratively.

[0055] Mask R-CNN: An advanced computer vision algorithm for object detection and instance segmentation in images.

[0056] Graph Convolutional Networks (GCNs) are deep learning models for processing graph-structured data. They extend the concept of Convolutional Neural Networks (CNNs) to apply to graph-structured data such as social networks, molecular structures, and transportation networks. GCNs can capture the topological structure and feature information of nodes, thus achieving excellent performance in tasks such as node classification, graph classification, and link prediction. To encode multimodal fusion feature maps, a K-layer GCN is employed, and a novel update mechanism is proposed to update two types of nodes in different ways. Object nodes generate object-level features, which can be viewed as first-order features of the image. The representation of relation nodes can be enhanced by their neighboring object nodes. Therefore, relation nodes are updated by aggregating information from their neighboring nodes, while object nodes are updated from themselves.

[0057] BERT: A pre-trained language model obtained through unsupervised training on a large amount of unlabeled text. It uses bidirectional encoding based on multiple encoding layers of Transformer. By building a marked-up language model, BERT can randomly cover or replace any word in a sentence, enabling the model to predict the randomly covered parts of the context and obtain distributed contextual representations of words.

[0058] Power scene diagram: includes multiple nodes and edges connecting the nodes, where each node contains objects and the edges represent the visual relationships between different objects.

[0059] The following is combined with Figures 1-6 This invention describes a method for generating power scene diagrams based on multimodal data, according to an embodiment of the present invention.

[0060] Figure 1 This is one of the flowcharts illustrating the power scene graph generation method based on multimodal data provided by this invention. This method is used in a multimodal semantic relationship extraction model. See also... Figure 2The multimodal semantic relation extraction model includes: power scene graph generation model, multimodal graph convolutional network and text extraction model.

[0061] like Figure 1 As shown, the method includes the following steps:

[0062] 101. Receive multimodal data, the multimodal data including power images and power text descriptions.

[0063] There are multiple ways to acquire multimodal data, such as collecting high-definition images of power equipment, such as substations and transmission lines, from power inspection systems; at the same time, collecting inspection reports or maintenance logs corresponding to these images, which contain descriptions of the image content.

[0064] Furthermore, after receiving multimodal data, the multimodal data can be preprocessed, such as image preprocessing including scaling, cropping, and rotation correction, to ensure that the image meets the requirements of the model input; and text descriptions can be preprocessed by sentence segmentation, word segmentation, and removal of stop words.

[0065] Taking multimodal data including power images and power text descriptions as an example, the power images are on-site power equipment images containing transmission towers, wires and insulators, and the text description is "Three sets of high-voltage insulators are installed on the top of the transmission tower, and the transmission line runs from left to right."

[0066] 102. Input the power image into a pre-trained power scene graph generation model to generate a power scene graph corresponding to the power image; the power scene graph is used to represent objects and their visual relationships in multimodal data.

[0067] In this process, object-level visual features are considered the fundamental representation of power image information, and visual semantic features of the power image are obtained by extracting object representations. To achieve this, the input power image is first fed into a pre-trained power scene graph generation model to generate a power scene graph of the input power image. This power scene graph consists of multiple nodes and edges connecting these nodes. Each node contains object features as its internal information, while edges represent the visual relationships between different objects, such as dependence or connection.

[0068] The structure of a sentence (i.e., the dependency tree) can provide important information for relation extraction models. Therefore, the power scene graph generation model in this embodiment of the invention aims to generate a one-way graph for an input power image, thereby providing structural information for multimodal power relation extraction. It is important to note that in the power scene graph, visual object features play the role of node features.

[0069] Specifically, the power image can be an image of power equipment, which may include transformers, utility poles, insulators, and other power devices. The input power image undergoes preprocessing, including resizing and normalization, to adapt it to the input requirements of the pre-trained model.

[0070] Use a pre-trained power scene graph generation model (such as Mask R-CNN) to identify individual objects in the image. For example, the model might identify objects such as transformers, utility poles, insulators, and grounding wires.

[0071] Then, visual features are extracted for each detected object, which may include shape, size, texture, etc.

[0072] The model predicts the visual relationships between any two objects using a power scene graph generation method. For example: a transformer is "located next to" a "power pole", an insulator is "suspended on" the "power pole", and a grounding wire is "connected to the bottom of the transformer".

[0073] Finally, based on the detected objects and the predicted relationships, a power scene map is generated. The scene map might look like this:

[0074] [Transformer] --(located next to)-->[Utility pole];

[0075] [Utility pole] --(suspended on)-->[insulator];

[0076] [Transformer] --(connected to)-->[Grounding wire].

[0077] Output: A power scene graph containing nodes (objects) and edges (visual relationships between objects).

[0078] The nodes include:

[0079] Node 1: Transformer (including visual features of the transformer);

[0080] Node 2: Utility pole (including visual features of the utility pole);

[0081] Node 3: Insulator (including visual features of the insulator);

[0082] Node 4: Grounding wire (including visual features of the grounding wire);

[0083] The edges include:

[0084] Side 1: Located next to (the relationship between the transformer and the utility pole);

[0085] Side 2: Suspended in (the relationship between the utility pole and the insulator);

[0086] Side 3: Connected to (the relationship between the transformer and the grounding wire).

[0087] In this way, the power scene graph generation model can extract rich visual information from power images and identify complex relationships between objects, providing a foundation for subsequent semantic analysis and applications.

[0088] 103. Input the power scene graph into a pre-trained multimodal graph convolutional network to generate graph semantic features of the power image.

[0089] The multimodal graph convolutional network can be a GCN (Graph Convolutional Network). To encode the multimodal fused feature map, a K-layer GCN is employed, and a novel update mechanism is proposed to update two types of nodes in different ways. Object nodes generate object-level features, which can be considered as first-order features of the image. The representation of relation nodes can be enhanced by their neighboring object nodes; therefore, relation nodes are updated by aggregating information from their neighbors, while object nodes are updated from themselves.

[0090] The multimodal graph convolutional network includes a pre-trained visual feature extractor, a multimodal fusion layer, and a graph convolutional network, wherein the graph convolutional network includes multiple graph convolutional layers.

[0091] The features of nodes and edges in the power scene graph are input into a pre-trained visual feature extractor (such as a CNN) to obtain the visual features of nodes and edges. In this embodiment, nodes contain objects, and edges represent the visual relationships between different objects.

[0092] The visual features of nodes and edges are processed and fused to generate richer representations. Fully connected layers and activation functions such as ReLU are used to integrate the features.

[0093] The fused features are then fed into a Graph Convolutional Network (GCN). Through multiple layers of GCN, each layer updates the feature representations of nodes and edges to capture local structural information.

[0094] After processing by the graph convolutional network, each node and edge will have a new graph semantic feature representation. These feature representations are able to capture the contextual information of nodes (objects) and edges (relationships). Finally, the graph semantic features of the power image are output, including graph semantic features of objects and graph semantic features of visual relationships.

[0095] Step 103 generates graph semantic feature representations of each object in the power image and the relationships between them. These representations can be used for subsequent similarity calculations and updates to the power scene graph.

[0096] 104. Input the power text description into a pre-trained text extraction model to generate text semantic features.

[0097] The text extraction model can be the BERT model. A high-dimensional vector for each word is generated using a pre-trained BERT model. By concatenating the high-dimensional vectors of each word, the vector describing the entire text can be obtained. Encoding and high-dimensional vectors form the embedding vector for each word. BERT is a language model derived through unsupervised training on a large amount of unlabeled text. The Transformer-based encoder part performs bidirectional encoding. By building a labeled language model, BERT can randomly overwrite or replace any word in a sentence, enabling the model to predict the randomly overwritten parts of the context and obtain distributed contextual representations of words.

[0098] For example, consider the following power-related text description from the input text extraction model: "The maintenance team inspected the transformer located in the third block and recorded its operating temperature as 85°C, with no obvious damage found." First, the text is segmented into the following words: "Maintenance / team / inspected / the / transformer / located / in / the / third / block / and / recorded / its / operating / temperature / as / 85°C / , / without / finding / obvious / damage / ."

[0099] The processed text is then input into the BERT model to obtain the embedding representation of each word. The BERT model's encoding layer generates context-dependent embeddings for the text. The output, labeled with [CLS], is used as the semantic feature vector of the text. For example: [0.25, -0.45, 0.13, ...] (illustrative vector). This vector can be used for subsequent similarity calculations with the graph semantic features of the power scene graph.

[0100] 105. Calculate the similarity between the graph semantic features and the text semantic features, and update the power scene graph based on the similarity to obtain the updated power scene graph.

[0101] In this embodiment of the invention, a similarity function is designed to optimize the semantic representation of the power scene graph, thereby updating the power scene graph. .

[0102] To measure the similarity between two semantic features (image semantic features and text semantic features) in different modalities, the following similarity function is defined:

[0103]

[0104] in, It is a fully connected layer. and denoted as the semantic feature representations of object nodes and edges in the Kth layer, respectively; N represents the total number of nodes in the power scene graph; and cos represents the cosine similarity.

[0105] This invention uses a triplet loss function to optimize a multimodal semantic relation extraction model. This loss function constrains the similarity score of matched power image-power text pairs to be greater than the similarity score of unmatched pairs. The specific formula is as follows:

[0106]

[0107] Among them, power images And electricity sentence description It is a match, that is, a description of the power sentence. It describes an electrical image. Power images And electricity sentence description It's a mismatch, i.e., the description of the electricity sentence. Not describing an electrical image It was randomly sampled from other power image descriptions.

[0108] This represents the similarity score of the matching pair, i.e., the similarity between the power image f and the corresponding descriptive text t; This represents the similarity score of a mismatched pair, i.e., the similarity between an electrical image f and a randomly sampled, mismatched descriptive text t′.

[0109] Training ends when the loss function is optimized until the entire multimodal semantic relation extraction model converges.

[0110] During training, the similarity function and the triplet loss function work together on the entire system to optimize the parameters of all components. The goal is to enable the entire multimodal semantic relation extraction model to accurately generate accurate power scene maps from power images, extract accurate textual semantic features from text descriptions, and calculate and utilize the similarity between graph semantic features and textual semantic features to update the power scene maps.

[0111] In practical applications, for example, graph semantic feature vectors (from a power scenario graph): h g =[0.1,0.3,0.4], Text semantic feature vector (from the BERT model of electricity text description): h t =[0.2,0.2,0.6]. h g and h t L2 normalization is performed, then cosine similarity is calculated, and a threshold of 0.9 is set. If the similarity is greater than 0.9, the two are considered sufficiently similar, and no scene graph update is needed; if the similarity is less than 0.9, the objects and relationships in the scene graph are checked to see if they match the text description, and the power scene graph is updated: for example, by deleting certain nodes and / or edges in the power scene graph.

[0112] The power scene graph generation method based on multimodal data provided in this invention generates a power scene graph corresponding to the power image by inputting a power image into a pre-trained power scene graph generation model. The power scene graph is then input into a pre-trained multimodal graph convolutional network to generate graph semantic features of the power image. Power text descriptions are input into a pre-trained text extraction model to generate text semantic features. The similarity between the graph semantic features and the text semantic features is then calculated, and the power scene graph is updated based on the similarity to obtain an updated power scene graph. This method effectively integrates graph modality and text modality information when processing multimodal data, establishing an effective semantic association between visual and textual data.

[0113] Specifically, see Figure 3 The power scene map generation model includes: an object recognition layer, a feature extraction layer, a visual relationship recognition layer, and a power scene map output layer.

[0114] See Figure 4 Step 102 specifically includes:

[0115] 401. Input the power image into the object recognition layer to obtain multiple objects in the power image and the relative position of each object in the power image.

[0116] After the object recognition layer identifies the power image, it determines the position and bounding box of each object, and finally obtains a list of power equipment objects, including transformers, utility poles, insulators, etc.

[0117] For example, if the input power image is a high-resolution picture of a substation, it can identify 1 transformer, 3 utility poles, and 5 insulators.

[0118] 402. Input each detected object into the feature extraction layer to extract the visual features of the object.

[0119] Visual features are extracted for each object through a feature extraction layer. These features may include shape descriptors, color histograms, texture features, etc. These features are used for subsequent node representations, ultimately outputting a visual feature vector for each object. For example, the feature vector of a transformer is [0.5, 0.2, 0.3].

[0120] 403. Input the visual features of each object and the relative position of the object in the power image into the visual relationship recognition layer to detect the visual relationship between the objects.

[0121] Spatial relationships between objects are determined using a visual relationship recognition layer. Recognized relationships may include "above," "connected to," and "adjacent." The output is a list of visual relationships between objects, with each relationship having a type label.

[0122] Continuing with the previous example, the relationship between the transformer and pole 1 is determined to be "located next to", and the relationship between pole 1 and the insulator is "supported".

[0123] 404. Input the objects and the visual relationships between them into the power scene diagram output layer, and output the power scene diagram.

[0124] The power scene graph includes multiple nodes and edges connecting the nodes. Each node contains the object, and the edges represent the visual relationships between different objects.

[0125] In the process of generating power scene maps, the Mask R-CNN power scene map generation model is used to obtain data from the input power image. Each object represents a visual representation of the power scene from the input power image. Since each relationship is unidirectional, similar to a dependency tree, each object is also associated with other objects from the power image. Therefore, a graphical representation of the power scene from the input power image can be obtained. . It includes various relationship pairs between objects detected in the power image, and can be represented as follows:

[0126]

[0127]

[0128] in, Indicates the first detected power in the image The node corresponding to each object; Indicates the number of objects detected; Indicates the first The object and the first Edges of an object.

[0129] Through steps 401-404, a structured power scene map can be generated from the original power image, which can then be used for more in-depth semantic analysis and information extraction.

[0130] Optionally, see Figure 5 The multimodal graph convolutional network includes a pre-trained visual feature extractor, a multimodal fusion layer, and a graph convolutional network, wherein the graph convolutional network includes multiple graph convolutional layers.

[0131] See Figure 6 Step 103 specifically includes:

[0132] 601. Input the power scene image into the visual feature extractor to obtain the visual feature vector of the object and the visual feature vector of the visual relationship.

[0133] In this embodiment, a pre-trained visual feature extractor is used to encode power image regions into feature vectors. This can be a pre-trained CNN network or an object detector (e.g., Mask R-CNN). In the visual power scene graph, each node is encoded into a feature vector with a dimension of... The visual feature vector for object nodes. Its visual feature vector It is extracted from its corresponding power image region; for relation nodes Its visual feature vector From and Extracted from the combined image region.

[0134] 602. Input the visual feature vector of the object and the visual feature vector of the visual relationship into the multimodal fusion layer to obtain the fused feature representation of the object and the fused feature representation of the visual relationship, respectively.

[0135] After obtaining the visual feature vectors of each node and the visual feature vectors of the visual relationships, they need to be fused into a unified representation. Therefore, a multimodal fusion feature map is generated. Specifically, the visual feature vectors of each node and the visual feature vectors of the visual relationships are concatenated, and then transformed into a fused feature representation through a fusion layer.

[0136] Specifically, step 602 includes: inputting the visual feature vector of the object and the visual feature vector of the visual relationship into the multimodal fusion layer; fusing the visual feature vector of any object and the visual feature vector of the visual relationship associated with the object to obtain the fused feature representation of the object; and fusing the visual feature vectors of any two objects and the visual feature vector of the visual relationship connecting the two objects to obtain the fused feature representation of the visual relationship.

[0137] The specific formula is as follows:

[0138]

[0139]

[0140] in, For the fusion feature representation of objects, This represents the fusion features of visual relationships. These are the trainable parameters of the fusion layer. Representation and object nodes Feature vectors of related relational nodes.

[0141] v on Represents object node o nThe visual feature vector. This is the visual feature extracted from the objects detected in the power image, which may include visual information such as color, shape, and texture.

[0142] v oi Represents object node o i The visual feature vector, v oj Represents object node o j Visual feature vectors; This represents the feature vector of an edge in the power scene graph, where the edge connects to object node o. i and object node o j .

[0143] 603. The fused feature representation of the object and the fused feature representation of the visual relationship are input into multiple graph convolutional layers of the graph convolutional network for processing to obtain the graph semantic features of the object and the graph semantic features of the visual relationship.

[0144] In this embodiment, a K-layer GCN is used to encode the multimodal fusion feature map, and a novel update mechanism is proposed to update two types of nodes in different ways. Object nodes generate object-level features, which can be regarded as first-order features of the image. The representation of relation nodes can be enhanced by their neighboring object nodes; therefore, relation nodes are updated by aggregating information from their neighboring nodes, while object nodes are updated from themselves. Specifically, given a multimodal fusion feature map... The k-th layer of GCN is calculated as follows:

[0145]

[0146] in, It is a fully connected layer, and its initial state is a fused feature layer. .

[0147] Finally, the output of the K-layer GCN is an encoded visual feature map containing two types of vertices: and .in, This represents the object node o in the Kth layer of the graph convolutional network. n Feature representation, Represents the edge e in the Kth layer of the graph convolutional network. i,j The characteristics are represented.

[0148] Taking a power scene diagram containing transformers, utility poles, and insulators as an example, the relationship between the transformer and the utility pole is "located next to", and the relationship between the utility pole and the insulator is "supporting".

[0149] In step 601, visual features are extracted for the region of each object (transformer, utility pole, insulator) using a CNN visual feature extractor. For example, the visual feature vector of the transformer is [0.1, 0.3, 0.7].

[0150] In step 602, the visual features of the transformer are fused with its category label to generate a fused feature vector. For example, the fused feature vector of the transformer is [0.1, 0.35, 0.7].

[0151] In step 603, the fused feature vector is input into the graph convolutional network GCN and processed through two layers of graph convolution. The first layer updates the graph semantic features of the transformer to [0.2, 0.4, 0.5], and the second layer further updates them to [0.25, 0.45, 0.55].

[0152] These steps generate high-level graph semantic features for each object and relationship from the power scene graph. These features capture rich contextual information, providing a foundation for subsequent similarity calculations and scene graph updates.

[0153] Specifically, step 104 includes: inputting the power text description into a text extraction model, processing it sequentially through multiple text encoding layers, and generating the text semantic features.

[0154] For example, the power-related text description is: "Three sets of high-voltage insulators are installed on the top of the transmission tower, and the transmission line runs through it from left to right." The text is segmented into the following words: "Transmission tower / top / installed / three sets / high-voltage / insulators / , / transmission / line / from / left / to / right / through".

[0155] The processed text is input into the BERT model to obtain the embedding representation of each word. The BERT model's encoding layer generates context-dependent embeddings for the text. The output, labeled with [CLS], is used as the semantic feature vector of the text, for example: [0.25, -0.45, 0.13, ...]. This vector can be used for subsequent similarity calculations with the graph semantic features of the power scene graph.

[0156] Specifically, step 105 includes: calculating the similarity between the graph semantic features of the object and the text semantic features, and the similarity between the graph semantic features of the visual relationship and the text semantic features, respectively; if the similarity between the graph semantic features of the object and the text semantic features exceeds a first threshold, then the node corresponding to the object is retained; otherwise, the node and its connected edges are deleted; if the similarity between the graph semantic features of the visual relationship and the text semantic features exceeds a second threshold, then the edge corresponding to the visual relationship is retained; otherwise, the edge corresponding to the visual relationship is deleted.

[0157] In a specific example, after training, the power image f is input into the entire multimodal semantic relation extraction model to obtain the corresponding power scene map. and its semantic features The input matching electricity sentence description t is fed into the entire multimodal semantic relation extraction model to obtain text semantic features. .

[0158] Specifically, by calculating the power scenario diagram The semantic features of each node and edge are compared with the semantic features of the text. Similarity to power scene map The update will be performed according to the following rules:

[0159] if Then the power scenario diagram Reserved nodes Conversely, delete a node. and its connected edges . The first threshold is set;

[0160] if Then the power scenario diagram Preserve edges Conversely, delete the edge. . The second threshold is set.

[0161] Step 105 removes irrelevant or low-relevance nodes and edges, making the power scene graph more concise and accurate; it also ensures consistency between the scene graph and the text description by accurately calculating and comparing similarity, thus improving the accuracy of information extraction.

[0162] In addition, it effectively integrates information from image and text modalities, enhancing the model's ability to understand and process multimodal data.

[0163] Through the embodiments of the present invention, it is possible to ensure that the power scene diagram accurately reflects the content of the text description, improve the quality and usability of the data, and lay a solid foundation for subsequent analysis and application.

[0164] The following describes the power scene map generation device based on multimodal data provided in the embodiments of the present invention. The power scene map generation device based on multimodal data described below and the power scene map generation method based on multimodal data described above can be referred to in correspondence.

[0165] This invention provides a power scene map generation device based on multimodal data. See [link to relevant documentation]. Figure 7 ,include:

[0166] The data receiving module 701 is used to receive multimodal data, which includes power images and power text descriptions;

[0167] The power scene graph generation module 702 is used to input the power image into a pre-trained power scene graph generation model to generate a power scene graph corresponding to the power image; the power scene graph is used to represent objects and their visual relationships in multimodal data;

[0168] The graph semantic feature generation module 703 is used to input the power scene graph into a pre-trained multimodal graph convolutional network to generate graph semantic features of the power image;

[0169] The text semantic feature generation module 704 is used to input the power text description into a pre-trained text extraction model to generate text semantic features;

[0170] The power scene graph update module 705 is used to calculate the similarity between the graph semantic features and the text semantic features, and update the power scene graph based on the similarity to obtain the updated power scene graph.

[0171] Optionally, the power scene graph generation model includes: an object recognition layer, a feature extraction layer, a visual relationship recognition layer, and a power scene graph output layer;

[0172] The power scene diagram generation module 702 is specifically used for:

[0173] The power image is input into the object recognition layer to obtain multiple objects in the power image and the relative position of each object in the power image;

[0174] Each detected object is input into the feature extraction layer to extract the visual features of the object;

[0175] The visual features of each object and its relative position in the power image are input to the visual relationship recognition layer to detect the visual relationship between the objects.

[0176] The objects and the visual relationships between them are input into the power scene graph output layer, and the power scene graph is output. The power scene graph includes multiple nodes and edges connecting the nodes. Each node contains the object, and the edges represent the visual relationships between different objects.

[0177] Optionally, the multimodal graph convolutional network includes a pre-trained visual feature extractor, a multimodal fusion layer, and a graph convolutional network, wherein the graph convolutional network includes multiple graph convolutional layers;

[0178] The graph semantic feature generation module 703 is specifically used for:

[0179] The power scene image is input into the visual feature extractor to obtain the visual feature vector of the object and the visual feature vector of the visual relationship;

[0180] The visual feature vector of the object and the visual feature vector of the visual relationship are input into the multimodal fusion layer to obtain the fused feature representation of the object and the fused feature representation of the visual relationship, respectively.

[0181] The fused feature representation of the object and the fused feature representation of the visual relationship are input into multiple graph convolutional layers of the graph convolutional network for processing to obtain the graph semantic features of the object and the graph semantic features of the visual relationship.

[0182] Optionally, the graph semantic feature generation module 703 is specifically used to: input the visual feature vector of the object and the visual feature vector of the visual relationship into the multimodal fusion layer; fuse the visual feature vector of any object and the visual feature vector of the visual relationship associated with the object to obtain the fused feature representation of the object; and fuse the visual feature vectors of any two objects and the visual feature vector of the visual relationship connecting the two objects to obtain the fused feature representation of the visual relationship.

[0183] Optionally, the text extraction model includes multiple text encoding layers connected in sequence;

[0184] The text semantic feature generation module 704 is specifically used to: input the power text description into the text extraction model, process it sequentially through multiple text encoding layers, and generate the text semantic features.

[0185] Optionally, the power scene graph includes multiple nodes and edges connecting the nodes, each node contains the object, and the edges represent the visual relationships between different objects;

[0186] The power scene map update module 705 is specifically used for:

[0187] Calculate the similarity between the graph semantic features and the text semantic features of the object, and the similarity between the graph semantic features and the text semantic features of the visual relationship, respectively.

[0188] If the similarity between the graph semantic features of the object and the text semantic features exceeds a first threshold, then the node corresponding to the object is retained; otherwise, the node and its connected edges are deleted.

[0189] If the similarity between the graph semantic features of the visual relationship and the text semantic features exceeds a second threshold, then the edge corresponding to the visual relationship is retained; otherwise, the edge corresponding to the visual relationship is deleted.

[0190] The power scene graph generation device based on multimodal data provided in this invention generates a power scene graph corresponding to the power image by inputting a power image into a pre-trained power scene graph generation model. The power scene graph is then input into a pre-trained multimodal graph convolutional network to generate graph semantic features of the power image. Power text descriptions are input into a pre-trained text extraction model to generate text semantic features. The similarity between the graph semantic features and the text semantic features is then calculated, and the power scene graph is updated based on the similarity to obtain an updated power scene graph. This device effectively integrates graph modality and text modality information when processing multimodal data, establishing an effective semantic association between visual and textual data.

[0191] Figure 8 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 8 As shown, the electronic device may include a processor 810, a communications interface 820, a memory 830, and a communication bus 840, wherein the processor 810, communications interface 820, and memory 830 communicate with each other via the communication bus 840. The processor 810 can call logical instructions in the memory 830 to execute a power scene map generation method based on multimodal data. This method includes: receiving multimodal data, including power images and power text descriptions; inputting the power images into a pre-trained power scene map generation model to generate a power scene map corresponding to the power images; the power scene map representing objects and their visual relationships in the multimodal data; inputting the power scene map into a pre-trained multimodal graph convolutional network to generate graph semantic features of the power images; inputting the power text descriptions into a pre-trained text extraction model to generate text semantic features; calculating the similarity between the graph semantic features and the text semantic features, and updating the power scene map based on the similarity to obtain an updated power scene map.

[0192] Furthermore, the logical instructions in the aforementioned memory 830 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0193] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the power scene map generation method based on multimodal data provided by the above methods. The method includes: receiving multimodal data, the multimodal data including power images and power text descriptions; inputting the power images into a pre-trained power scene map generation model to generate a power scene map corresponding to the power images; the power scene map is used to represent objects and their visual relationships in the multimodal data; inputting the power scene map into a pre-trained multimodal graph convolutional network to generate graph semantic features of the power images; inputting the power text descriptions into a pre-trained text extraction model to generate text semantic features; calculating the similarity between the graph semantic features and the text semantic features, and updating the power scene map based on the similarity to obtain an updated power scene map.

[0194] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the power scene map generation method based on multimodal data provided by the above methods. The method includes: receiving multimodal data, the multimodal data including power images and power text descriptions; inputting the power images into a pre-trained power scene map generation model to generate a power scene map corresponding to the power images; the power scene map being used to represent objects and their visual relationships in the multimodal data; inputting the power scene map into a pre-trained multimodal graph convolutional network to generate graph semantic features of the power images; inputting the power text descriptions into a pre-trained text extraction model to generate text semantic features; calculating the similarity between the graph semantic features and the text semantic features, and updating the power scene map based on the similarity to obtain an updated power scene map.

[0195] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0196] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0197] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for generating power scene diagrams based on multimodal data, characterized in that, include: Receive multimodal data, including power images and power text descriptions; The power images are images of on-site power equipment; The power image is input into a pre-trained power scene image generation model to generate a power scene image corresponding to the power image; The power scene map is used to represent objects and their visual relationships in multimodal data; The power scene graph is input into a pre-trained multimodal graph convolutional network to generate graph semantic features of the power image; The electrical text description is input into a pre-trained text extraction model to generate text semantic features; Calculate the similarity between the graph semantic features and the text semantic features, and update the power scene graph based on the similarity to obtain the updated power scene graph; The power scene graph generation model includes: an object recognition layer, a feature extraction layer, a visual relationship recognition layer, and a power scene graph output layer; The power image is input into a pre-trained power scene image generation model to generate a power scene image corresponding to the power image, specifically including: The power image is input into the object recognition layer to obtain multiple objects in the power image and the relative position of each object in the power image; Each detected object is input into the feature extraction layer to extract the visual features of the object; The visual features of each object and its relative position in the power image are input to the visual relationship recognition layer to detect the visual relationship between the objects. The objects and the visual relationships between them are input into the power scene graph output layer, and the power scene graph is output. The power scene graph includes multiple nodes and edges connecting the nodes. Each node contains the object, and the edges represent the visual relationships between different objects. Calculate the similarity between the semantic features of the graph and the semantic features of the text, and update the power scene graph based on the similarity to obtain the updated power scene graph, specifically including: Calculate the similarity between the graph semantic features and the text semantic features of the object, and the similarity between the graph semantic features and the text semantic features of the visual relationship, respectively. If the similarity between the graph semantic features of the object and the text semantic features exceeds a first threshold, then the node corresponding to the object is retained; otherwise, the node and its connected edges are deleted. If the similarity between the graph semantic features of the visual relationship and the text semantic features exceeds a second threshold, then the edge corresponding to the visual relationship is retained; otherwise, the edge corresponding to the visual relationship is deleted.

2. The method for generating power scene diagrams based on multimodal data according to claim 1, characterized in that, The multimodal graph convolutional network includes a pre-trained visual feature extractor, a multimodal fusion layer, and a graph convolutional network, wherein the graph convolutional network includes multiple graph convolutional layers; The power scene image is input into a pre-trained multimodal graph convolutional network to generate graph semantic features of the power image, specifically including: The power scene image is input into the visual feature extractor to obtain the visual feature vector of the object and the visual feature vector of the visual relationship; The visual feature vector of the object and the visual feature vector of the visual relationship are input into the multimodal fusion layer to obtain the fused feature representation of the object and the fused feature representation of the visual relationship, respectively. The fused feature representation of the object and the fused feature representation of the visual relationship are input into multiple graph convolutional layers of the graph convolutional network for processing to obtain the graph semantic features of the object and the graph semantic features of the visual relationship.

3. The method for generating power scene diagrams based on multimodal data according to claim 2, characterized in that, The visual feature vectors of the object and the visual feature vectors of the visual relationship are input into a multimodal fusion layer to obtain the fused feature representations of the object and the visual relationship, respectively. Specifically, this includes: The visual feature vectors of the object and the visual feature vectors of the visual relationship are input into the multimodal fusion layer; The fused feature representation of the object is obtained by fusing the visual feature vector of any of the objects and the visual feature vector of the visual relationship associated with the object. The visual feature vectors of any two objects and the visual feature vectors connecting the visual relationship between the two objects are fused to obtain the fused feature representation of the visual relationship.

4. The method for generating power scene diagrams based on multimodal data according to claim 1, characterized in that, The text extraction model includes multiple text encoding layers connected in sequence; The electrical text description is input into a pre-trained text extraction model to generate text semantic features, specifically including: The power text description is input into the text extraction model and processed sequentially through multiple text encoding layers to generate the text semantic features.

5. A device for generating power scene diagrams based on multimodal data, characterized in that, include: A data receiving module is used to receive multimodal data, including power images and power text descriptions; The power images are images of on-site power equipment; The power scene graph generation module is used to input the power image into a pre-trained power scene graph generation model to generate a power scene graph corresponding to the power image; the power scene graph is used to represent objects and their visual relationships in multimodal data; The graph semantic feature generation module is used to input the power scene graph into a pre-trained multimodal graph convolutional network to generate graph semantic features of the power image; The text semantic feature generation module is used to input the power text description into a pre-trained text extraction model to generate text semantic features; The power scene graph update module is used to calculate the similarity between the graph semantic features and the text semantic features, and update the power scene graph based on the similarity to obtain the updated power scene graph; The power scene graph generation model includes: an object recognition layer, a feature extraction layer, a visual relationship recognition layer, and a power scene graph output layer; The power scene diagram generation module is specifically used for: The power image is input into the object recognition layer to obtain multiple objects in the power image and the relative position of each object in the power image; Each detected object is input into the feature extraction layer to extract the visual features of the object; The visual features of each object and its relative position in the power image are input to the visual relationship recognition layer to detect the visual relationship between the objects. The objects and the visual relationships between them are input into the power scene graph output layer, and the power scene graph is output. The power scene graph includes multiple nodes and edges connecting the nodes. Each node contains the object, and the edges represent the visual relationships between different objects. The power scene map update module is specifically used for: Calculate the similarity between the graph semantic features and the text semantic features of the object, and the similarity between the graph semantic features and the text semantic features of the visual relationship, respectively. If the similarity between the graph semantic features of the object and the text semantic features exceeds a first threshold, then the node corresponding to the object is retained; otherwise, the node and its connected edges are deleted. If the similarity between the graph semantic features of the visual relationship and the text semantic features exceeds a second threshold, then the edge corresponding to the visual relationship is retained; otherwise, the edge corresponding to the visual relationship is deleted.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the power scene map generation method based on multimodal data as described in any one of claims 1 to 4.

7. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the power scene diagram generation method based on multimodal data as described in any one of claims 1 to 4.

8. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the power scene diagram generation method based on multimodal data as described in any one of claims 1 to 4.

Images