Image knowledge graph automatic generation method based on graph convolutional network

Abstract

The invention discloses an automatic generation method for an image knowledge graph based on a graph convolutional network. The method comprises the following steps: carrying out key feature extraction on an input image by using a transfer learning method containing a cross attention mechanism; and inputting the extracted key features as node features into a graph convolutional network, modeling a relationship and a structure between the node features through a message passing and updating mechanism, and generating an image knowledge graph. The method has the advantages that the transfer learning method is adopted, and the adaptive capacity to different types of images is improved by utilizing the universal features learned on multiple data sets by the pre-training model.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to an automated method for generating image knowledge graphs based on graph convolutional networks. Background Technology

[0002] Image knowledge graphs, as a structured semantic representation method, can model entities, attributes, and relationships in images in a graphical form, and have significant application value in fields such as medical diagnosis, remote sensing analysis, and intelligent monitoring. Taking the medical field as an example, medical image knowledge graphs can integrate multi-dimensional information such as lesion regions, pathological features, and clinical diagnoses, assisting doctors in quickly locating abnormal areas and associating them with pathological knowledge, thereby improving diagnostic accuracy and efficiency. Simultaneously, their structured knowledge representation also provides intuitive learning resources for medical education, helping students understand the imaging characteristics and pathological mechanisms of complex diseases.

[0003] Existing methods for constructing knowledge graphs have significant limitations:

[0004] Insufficient data type adaptability: Traditional knowledge graph construction is mostly geared towards structured data such as text and tables. There is little research on automated construction methods for unstructured image data, especially in professional fields such as medicine and remote sensing, where there is a lack of mature end-to-end solutions from images to knowledge graphs.

[0005] Lack of feature relationship modeling: Existing technologies often convert images into structured data (such as object detection boxes and category labels) through manual annotation or simple rules, but do not delve into the semantic relationships between features at different levels within the image (such as "the spatial relationship between lesion A and lesion B" and "the topological structure of different land cover types"), which limits the semantic integrity and reasoning ability of knowledge graphs.

[0006] Weak cross-domain generalization ability: Traditional methods often design feature extraction rules for specific scenarios (such as medical images of a single organ). When applied to new scenarios or new types of images, it is necessary to manually adjust the model parameters or label the data, making it difficult to achieve automated generation of knowledge graphs for cross-domain images.

[0007] Therefore, how to break through the dependence of existing technologies on structured data, achieve accurate extraction of image features and high-order relationship modeling, and then build an automated image knowledge graph generation method with strong generalization ability has become a technical problem that urgently needs to be solved in this field. Summary of the Invention

[0008] The main technical problem solved by this invention is to provide an automated method for generating image knowledge graphs based on graph convolutional networks, thereby solving one or more of the problems mentioned above in the prior art.

[0009] To solve the above-mentioned technical problems, the present invention adopts the following technical solution: an automated image knowledge graph generation method based on graph convolutional networks, the innovation of which is: including the following steps:

[0010] S1. Extract key features from the input image using a transfer learning method that includes a cross-attention mechanism;

[0011] S2. Input the key features extracted in step S1 into the graph convolutional network as node features, and model the relationship and structure between node features through message passing and update mechanism to generate an image knowledge graph.

[0012] In some implementations, the key feature extraction process in step S2 includes:

[0013] S21. Extract preliminary image features using a pre-trained model:

[0014] F = f pre (I)

[0015] In the formula, F represents the extracted image features, f pre This represents the pre-trained transfer learning model, where I represents the input image;

[0016] S22. Calculate the attention weights and generate a weighted feature map:

[0017] Calculate the query, key, and value matrix: Among them W Q W K and W V The learned weight matrix;

[0018] Calculate the attention weight matrix:

[0019]

[0020] Where d k Let be the dimension of the key vector; the softmax operation is used to normalize the weights.

[0021] Generate weighted feature map: F att =AV;

[0022] S23. Concatenate the original features with the weighted features to generate the final feature representation: F final =concat(F,F att ), where concat(·) represents the concatenation of feature maps.

[0023] In some implementations, in step S22, the attention weight matrix is ​​normalized using the softmax function so that the sum of the weights is 1.

[0024] In some implementations, in step S23, the final feature representation is obtained by comparing the original feature F with the weighted feature F att It is obtained by splicing together.

[0025] In some implementations, the modeling process in step S2 includes message passing, node feature normalization, and node feature updating:

[0026] Message passing: For node v, compute the message from neighbor node Nv: m v =∑u∈N v Wh u , where W is the weight matrix and hu is the feature representation of node u;

[0027] Node feature normalization: Normalizing the message: Where |N v | represents the number of neighboring nodes;

[0028] Node feature update: Update node features using a non-linear function: h ' v =ReLU(Wh v +b+m v ), where b is the bias, ReLU is the activation function, and h ' v This refers to the updated node features.

[0029] In some implementations, during node feature updates, the combination of weighted messages and current node features is processed using the ReLU activation function.

[0030] In some implementations, the calculation formula for the graph convolutional network layer in step S2 is:

[0031]

[0032] In the formula, H (l) Let A be the node feature matrix of the l-th layer, A be the adjacency matrix, D be the degree matrix, and W be the number of nodes. (l) Let be the weight matrix of the l-th layer.

[0033] In some implementations, the diagonal elements of the degree matrix D are the degrees of each row of the adjacency matrix A.

[0034] In some implementations, in step S1, the transfer learning model learns features from different datasets through pre-training to improve its generalization ability to new tasks.

[0035] In some implementations, in step S2, the graph convolutional network captures high-order relationships between image features through multi-layer message passing and node updates.

[0036] The beneficial effects of this invention are:

[0037] Strong generalization ability: By adopting the transfer learning method, the general features learned by the pre-trained model on multiple datasets are utilized to improve the adaptability to different types of images.

[0038] Precise feature selection: Introducing a cross-attention mechanism, dynamically calculating feature weights and focusing on important regions, reducing redundant information interference, and improving the relevance of feature representation;

[0039] High-order relationship capture: Through message passing and multi-level iteration of GNN, it effectively models the complex structural relationships between image features, supports multi-level feature fusion, and generates a knowledge graph with more semantic integrity. Attached Figure Description

[0040] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort, wherein:

[0041] Figure 1 This is an overall framework diagram of the present invention;

[0042] Figure 2 This is a schematic diagram of the transfer learning feature extraction process of the present invention;

[0043] Figure 3 This is a schematic diagram of the attention mechanism weight calculation process of the present invention;

[0044] Figure 4 This is a schematic diagram of the graph convolutional network relationship extraction process of the present invention. Detailed Implementation

[0045] The technical solutions in the embodiments of the present invention will be clearly and completely described below. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0046] This invention includes: an automated method for generating image knowledge graphs based on graph convolutional networks, the overall process of which is as follows: Figure 1 As shown, it includes the following steps:

[0047] S1: Extract key features using transfer learning methods that include cross-attention mechanisms.

[0048] This step combines transfer learning and cross-attention mechanisms to extract semantically significant key features from the input image. The specific process is as follows:

[0049] S21: Pre-trained model extracts preliminary features

[0050] like Figure 2 As shown, a pre-trained transfer learning model (such as ResNet, VGG, etc.) is used to perform preliminary feature extraction on the input image, as shown in the following formula:

[0051] F = f pre (I)

[0052] Where I is the input image, f pre For a pre-trained model (such as a convolutional neural network pre-trained on the ImageNet dataset), FF represents the initial image features extracted.

[0053] S22: Cross-attention mechanism for calculating weighted features

[0054] like Figure 3 As shown, to focus on regions in the image that are more important for knowledge graph generation, a cross-attention mechanism is introduced to weight and optimize the initial features. The specific steps include:

[0055] Generate query, key, and value matrices: using the learned weight matrix W Q W K W V A linear transformation is performed on the initial feature F to obtain the query matrix Q, the key matrix K, and the value matrix V:

[0056]

[0057] Calculate the attention weight matrix: Calculate the correlation weights between features using the scaled dot product attention formula, and then normalize using softmax.

[0058]

[0059] Where, d k The dimension of the key vector is used to alleviate the vanishing gradient problem; the softmax function ensures that the sum of the weights is 1, making the attention distribution more uniform.

[0060] Generate a weighted feature map: Multiply the attention weight matrix A by the value matrix V to obtain the weighted feature map F. att :

[0061] F att =AV

[0062] S23: Feature concatenation generates the final feature representation.

[0063] Compare the original preliminary feature F with the weighted feature map F attChannel-level concatenation is performed, and basic features and attention-enhanced features are merged to obtain the final key feature F. final :

[0064] F final =concat(F,F att )

[0065] Here, concat(·) represents the feature map concatenation operation, ensuring that key features retain both global and local salient information.

[0066] S2: Modeling feature relationships and generating knowledge graphs based on graph convolutional networks.

[0067] like Figure 4 As shown, the key feature F extracted in step S1 final As input to the graph node GNN, the complex relationships between features are modeled through message passing and node update mechanisms. The specific process is as follows:

[0068] Message passing

[0069] For each node v in the graph, aggregate its neighbor nodes N. v The characteristic information is used to generate the message m of node v. v :

[0070] m v =∑u∈N v Wh u

[0071] Where W is the learnable weight matrix, h u Let u be the feature representation of the neighbor node, and let the neighbor information be transmitted through weighted summation.

[0072] Node feature normalization

[0073] To avoid the impact of differences in node degree (number of neighbors) on feature updates, for message m... v Normalization is performed:

[0074]

[0075] Where, |N v | represents the number of neighbors of node v, ensuring that nodes of different degrees receive messages with a balanced contribution.

[0076] Node feature updates combine the current node features with the normalized message, and update the node features using a non-linear activation function:

[0077] h ' v =ReLU(Wh v +b+m v )

[0078] Among them, h v Here, b represents the original features of node v, b is the bias term, and the ReLU function is used to introduce a nonlinear transformation to enhance the model's expressive power. ' v This refers to the updated node features.

[0079] GNN layer iterative optimization

[0080] High-order relation modeling is achieved by stacking multiple layers of GNNs, with each layer outputting a feature matrix H. (l+1) Calculated by the following formula:

[0081]

[0082] Among them, H (l) Let A be the node feature matrix of layer l, A be the graph adjacency matrix, D be the degree matrix (the diagonal elements are the degree of each row of A), and W be the node feature matrix of layer l. (l) Let be the weight matrix of the l-th layer. Through multiple iterations, the GNN gradually captures the higher-order dependencies between features, ultimately outputting a structured representation of the image knowledge graph.

[0083] The working principle of this technical solution is as follows: This invention achieves automated generation of image knowledge graphs through a two-stage process of "feature extraction - relation modeling".

[0084] Feature extraction stage: Preliminary features of the image are extracted using a pre-trained transfer learning model, and key regions are dynamically focused by a cross-attention mechanism. Key features with both global and local information are generated through feature weighting and splicing.

[0085] Relationship modeling stage: Key features are input into the GNN as graph nodes. Neighbor information is aggregated through message passing. Node features are optimized through normalization and nonlinear updates. High-order relationships between features are captured through multi-layer GNN iterations, and finally, a structured image knowledge graph is output.

[0086] The advantages of this technical solution are:

[0087] Strong generalization ability: By adopting the transfer learning method, the general features learned by the pre-trained model on multiple datasets are utilized to improve the adaptability to different types of images.

[0088] Precise feature selection: Introducing a cross-attention mechanism, dynamically calculating feature weights and focusing on important regions, reducing redundant information interference, and improving the relevance of feature representation;

[0089] High-order relationship capture: Through message passing and multi-level iteration of GNN, it effectively models the complex structural relationships between image features, supports multi-level feature fusion, and generates a knowledge graph with more semantic integrity.

[0090] Example 1: Generation of a Knowledge Graph of Lesions from Medical CT Images

[0091] This embodiment uses the method of the present invention to generate a knowledge graph containing lesion areas, pathological features and correlations from lung CT images to assist in clinical diagnosis.

[0092] Step S1: Key Feature Extraction

[0093] Pre-trained model selection: ResNet-50 pre-trained on the ImageNet dataset was used as the transfer learning model, with the input being 512×512 pixel lung CT image slices.

[0094] Preliminary feature extraction: The preliminary feature map F is output through the conv5 layer of ResNet-50, with dimensions of 2048×16×16 (number of channels × height × width).

[0095] Attention mechanism parameters: query / key / value matrix dimension d k =64, weight matrix W Q W K W V Initialized as a random normal distribution, it is updated after training. The attention weight matrix A is normalized using softmax to generate a weighted feature map F. att The final feature F is obtained by concatenating it with the original feature F. final (Dimensions 4096×16×16).

[0096] Step S2: GNN Relation Modeling

[0097] Node definition: F final The system is divided into 256 nodes (16×16 grid) based on spatial location, with each node having a feature dimension of 4096.

[0098] Adjacency matrix construction: The graph structure is constructed using an 8-neighborhood connection method, and the adjacency matrix A is a sparse matrix of 256×256.

[0099] GNN parameter settings: 3-layer GNN, weight matrix W for each layer. (l) The dimensions are 4096→2048→1024→512, and the diagonal elements of the degree matrix D are the number of adjacencies of each node (the degree of boundary nodes is 3-5, and the degree of internal nodes is 8).

[0100] Knowledge graph output: After clustering, the final node features correspond to lesion regions in CT images (such as nodules and ground-glass opacities). The edge weights between nodes represent the spatial positional relationships between regions (such as "adjacent" and "contains"). A knowledge graph containing 12 core lesion entities and 8 spatial relationships is generated.

[0101] Example 2: Generation of Knowledge Graph of Ground Features from Remote Sensing Images

[0102] This embodiment generates a knowledge graph containing land cover types and topological relationships such as buildings, roads, and vegetation from high-resolution remote sensing images for urban planning analysis.

[0103] Step S1: Key Feature Extraction

[0104] Pre-trained model selection: EfficientNet-B4, pre-trained on a remote sensing dataset (such as NWPU-RESISC45), was used as the transfer learning model, with the input being a 1024×1024 pixel urban remote sensing image.

[0105] Preliminary feature extraction: The initial feature map F is output through the MBConv5 layer of EfficientNet-B4, with dimensions of 1792×32×32.

[0106] Attention mechanism parameters: query / key / value matrix dimension d k =128, weight matrix W Q W K W V The feature map F is initialized randomly and optimized during training. att After concatenation with the original feature F, the final feature F is obtained. final (Dimensions 3584×32×32).

[0107] Step S2: GNN Relation Modeling

[0108] Node definition: F final The grid is divided into 1024 nodes using a 32×32 grid, with each node having a feature dimension of 3584.

[0109] Adjacency matrix construction: A 4-neighbor (top, bottom, left, right) connection method is adopted, and the adjacency matrix A is a sparse matrix of 1024×1024.

[0110] GNN parameter settings: 4 layers of GNN, weight matrix W for each layer. (l) The dimensions are 3584→2048→1024→512→256, and the diagonal elements of the degree matrix D are the number of node adjacencies (mostly 4).

[0111] Knowledge graph output: After semantic classification of node features, corresponding to land feature types (buildings, roads, vegetation, water bodies, etc.), edge weights represent the topological relationships between land features (such as "parallel", "intersecting", "encircling"), generating a knowledge graph containing 18 types of land feature entities and 12 types of topological relationships.

[0112] The above embodiments demonstrate the universality of the invention through specific scenarios: in medical CT images, it focuses on the subtle features and spatial relationships of lesion areas; in remote sensing images, it captures the global topological structure of ground objects. Both achieve accurate feature extraction through transfer learning and attention mechanisms, and model high-order relationships through multi-layer iterative GNN to ultimately generate a structured knowledge graph.

[0113] Highly adaptable to various scenarios: It can flexibly adapt to images from different fields such as medicine and remote sensing, and achieve targeted modeling of features and relationships by adjusting the pre-trained model and GNN parameters;

[0114] High structural interpretability: The nodes and edges of the knowledge graph directly correspond to the entities and relationships in the image, and the output results can be intuitively mapped to the original image area, which is convenient for manual verification and application implementation.

[0115] The above description is merely an embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent structural or procedural transformations made based on the content of the present invention specification, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. An automated method for generating image knowledge graphs based on graph convolutional networks, characterized in that: Includes the following steps: S1. Extract key features from the input image using a transfer learning method that includes a cross-attention mechanism; S2. Input the key features extracted in step S1 into the graph convolutional network as node features, and model the relationship and structure between node features through message passing and update mechanism to generate an image knowledge graph.

2. The method for automatically generating image knowledge graphs based on graph convolutional networks according to claim 1, characterized in that: The key feature extraction process in step S2 includes: S21. Extract preliminary image features using a pre-trained model: F=f pre (I) In the formula, F represents the extracted image features, f pre This represents the pre-trained transfer learning model, where I represents the input image; S22. Calculate the attention weights and generate a weighted feature map: Calculate the query, key, and value matrix: Q = FW Q K = FW K V = FW V W Q W K and W V The learned weight matrix; Calculate the attention weight matrix: Where d k Let be the dimension of the key vector; the softmax operation is used to normalize the weights. Generate weighted feature map: F att =AV; S23. Concatenate the original features with the weighted features to generate the final feature representation: F final =concat(F,F att ), where concat(·) represents the concatenation of feature maps.

3. The method for automatically generating image knowledge graphs based on graph convolutional networks according to claim 2, characterized in that: In step S22, the attention weight matrix is ​​normalized using the softmax function so that the sum of the weights is 1.

4. The method for automatically generating image knowledge graphs based on graph convolutional networks according to claim 2, characterized in that: In step S23, the final feature representation is obtained by comparing the original feature F with the weighted feature F. att It is obtained by splicing together.

5. The method for automatically generating image knowledge graphs based on graph convolutional networks according to claim 1, characterized in that: The modeling process in step S2 includes message passing, node feature normalization, and node feature updating: Message passing: For node v, compute the message from neighbor node Nv: m v =∑u∈N v Wh u , where W is the weight matrix and hu is the feature representation of node u; Node feature normalization: Normalizing the message: Where |N v | represents the number of neighboring nodes; Node feature update: Update node features using a non-linear function: h ' v =ReLU(Wh v +b+m v ), where b is the bias, ReLU is the activation function, and h ' v This refers to the updated node features.

6. The method for automatically generating image knowledge graphs based on graph convolutional networks according to claim 5, characterized in that: In node feature updates, the combination of weighted messages and current node features is processed using the ReLU activation function.

7. The method for automatically generating image knowledge graphs based on graph convolutional networks according to claim 1, characterized in that: In step S2, the calculation formula for the graph convolutional network layer is: In the formula, H (l) Let A be the node feature matrix of the l-th layer, A be the adjacency matrix, D be the degree matrix, and W be the number of nodes. (l) Let be the weight matrix of the l-th layer.

8. The method for automatically generating image knowledge graphs based on graph convolutional networks according to claim 7, characterized in that: The diagonal elements of the degree matrix D are the degree values ​​of each row of the adjacency matrix A.

9. The method for automatically generating image knowledge graphs based on graph convolutional networks according to claim 1, characterized in that: In step S1, the transfer learning model learns features from different datasets through pre-training to improve its generalization ability to new tasks.

10. The method for automatically generating image knowledge graphs based on graph convolutional networks according to claim 1, characterized in that: In step S2, the graph convolutional network captures high-order relationships between image features through multi-layer message passing and node updates.

Images