Image description method based on external knowledge graph
By combining external knowledge graphs with image description models, the shortcomings of existing image description methods in recognizing and describing complex image content are addressed, achieving more accurate and logical image descriptions.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- XINJIANG UNIVERSITY
- Filing Date
- 2024-07-12
- Publication Date
- 2026-07-31
AI Technical Summary
Existing image description methods struggle to accurately identify and describe complex and varied image content, especially images with multiple targets and complex backgrounds, resulting in descriptions that lack logic and richness.
By combining external knowledge graphs with image description models, logical and rich descriptions are generated through object detection, keyword prediction, knowledge retrieval, and evaluation.
By introducing external knowledge graphs, the accuracy and logic of image descriptions are improved, resulting in more meaningful text descriptions.
Smart Images
Figure CN118708735B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of image processing technology, and more specifically to an image description method based on an external knowledge graph. Background Technology
[0002] With the rapid development of big data and artificial intelligence technologies, image description tasks have gradually expanded from simple target recognition and description to understanding complex scenes, semantic relationships, and deeper meanings in images. Against this backdrop, the introduction of external knowledge graphs has provided new solutions for image description, making the descriptions more accurate and richer.
[0003] In existing technologies, traditional image description methods are mostly based on low-level visual features and predefined rules or templates. These methods mainly rely on basic visual information of the image, such as color, shape, and texture, to generate text by matching predefined description templates. However, these methods often fall short when dealing with complex and varied image content, resulting in inaccurate descriptions and difficulty in capturing deep semantic information within the image. For example, when faced with an image containing multiple targets and a complex background, traditional methods may only be able to describe some salient features while ignoring the relationships between targets and background information. Due to a lack of in-depth understanding of the relationships between entities in the image, the generated descriptions often lack logic and coherence, making the final text unnatural.
[0004] In recent years, the rise of deep learning technology has brought new progress to the field of image description. Deep learning-based image description methods can automatically learn image features and generate descriptive text that matches the image content. These methods typically use convolutional neural networks (CNNs), recurrent neural networks (RNNs), or long short-term memory networks (LSTMs) to generate descriptive text. However, these methods still face some challenges. For example, when processing images containing multiple targets or complex scenes, deep learning models often struggle to accurately identify and describe all the key information; furthermore, due to the lack of external knowledge guidance, the generated descriptions may lack accuracy and richness.
[0005] Therefore, there is an urgent need for a method that can combine external knowledge graphs with image descriptions to help image description models better understand image content by introducing semantic information from knowledge graphs. Summary of the Invention
[0006] Purpose of the Invention: To overcome the above shortcomings, the purpose of this invention is to provide a method that combines image description with an external knowledge graph. This method uses computer vision technology to extract key information from images, such as target objects, scenes, and actions. At the same time, it combines natural language processing technology to transform the extracted image information into descriptive text. Then, by introducing rich semantic information from the knowledge graph, it can more accurately understand the image content and generate more meaningful and logical descriptions.
[0007] To address the aforementioned technical problems, this invention provides an image description method based on an external knowledge graph, comprising:
[0008] Step S1: Perform object detection on the image to be introduced, including: image preprocessing, image feature extraction, object detection, post-processing optimization, and outputting the detection results;
[0009] Step S2: Use a text generation model as a keyword predictor to predict keywords in the detection results, and then input the predicted keywords into the knowledge retrieval machine to retrieve knowledge information from the external knowledge graph;
[0010] Step S3: Use a knowledge retrieval tool to search the external knowledge graph, obtain the knowledge subgraph associated with the keyword, and then input the knowledge subgraph and the keyword into the subgraph knowledge evaluator for knowledge evaluation;
[0011] Step S4: Use a subgraph knowledge evaluator to convert the knowledge subgraph into a text sequence while preserving the subgraph structure and node information, and then match it with the keywords to obtain the knowledge with the highest correlation to the target detection result;
[0012] Step S5: Use a language model to generate text containing relevant knowledge content.
[0013] In a preferred embodiment of the present invention, step S1 further includes the following steps:
[0014] Step S11: Extract features from the image to be introduced by performing convolution operations through convolutional layers and downsampling through pooling operations to reduce the number of parameters and computational cost;
[0015] Step S12: Repeat the residual block connection operation a preset number of times to extract features at a deeper level. The gradient vanishing problem is solved by skip connections, and the identity mapping is performed using a scaling factor to control the intensity and speed of the residual transformation.
[0016] Step S13: For all detected results, post-processing optimization is performed using the NMS algorithm to remove redundant detection results.
[0017] In a preferred embodiment of the present invention, step S11 further includes the following steps:
[0018] Step S111: Extract features from the images to be introduced using convolutional kernels of size 3×3 and number of channels 32, 64 and 128 respectively;
[0019] Step S112: The extracted features are downsampled through a max pooling layer with a stride of 2 to obtain feature maps with a size of 11×11 and 32, 64 and 128 channels.
[0020] In a preferred embodiment of the present invention, step S12 further includes the following steps:
[0021] Step S121: The feature map with size 11×11 and number of channels 256 is passed through three layers of convolution kernels with different numbers of sizes of 3×3 and number of channels 32. The three feature maps with the same size and number of channels are then concatenated to obtain a feature map with size 11×11 and number of channels 96.
[0022] Step S122: Perform identity mapping on the feature map using a convolution kernel of size 1×1 and number of channels 256 to ensure that the size is consistent with the initial feature map.
[0023] In a preferred embodiment of the present invention, in step S13, the method further includes: calculating the IOU of each of the detected results, and selecting the one with the largest IOU value as the final detection result.
[0024] In a preferred embodiment of the present invention, step S2 further includes the following steps:
[0025] Step S21: Given the currently detected result X i The model predicts a set of associated keywords as K. i The probability distribution is modeled using the GPT-2 model based on the Transformer architecture, where:
[0026]
[0027] Where K <j Let p represent all predicted keywords up to the j-th keyword, where p is the probability distribution and q is the sequence length.
[0028] Step S22: Optimize the keyword predictor using maximum likelihood training and next token prediction loss, and train it from real knowledge using the RAKE algorithm. i Extract keywords for K i Provide labels to train this keyword predictor.
[0029] In a preferred embodiment of the present invention, step S21 further includes: learning to infer the next most relevant keyword from the predicted keywords based on the GPT-2 model. Through continuous iteration, the model can generate a keyword sequence related to the initial keywords.
[0030] In a preferred embodiment of the present invention, step S22 further includes: using the RAKE algorithm to extract keywords from real text related to the target detection category as labels for model training.
[0031] In a preferred embodiment of the present invention, step S3 further includes the following steps:
[0032] Step S31: Use a knowledge retrieval tool to match the query information with nodes in the knowledge graph;
[0033] Step S32: After a successful match, the subgraph containing the query information is transformed into a sequence and returned to the knowledge retrieval device;
[0034] Step S33: Input the transformed subgraph sequence and keywords into the subgraph knowledge evaluator for evaluation.
[0035] In a preferred embodiment of the present invention, step S31 further includes the following steps:
[0036] Step S311: Use the Boyer-Moore matching algorithm to match keywords with nodes in the knowledge graph;
[0037] Step S312: After a successful match, locate the keyword nodes in the knowledge graph.
[0038] In a preferred embodiment of the present invention, step S32 further includes the following steps:
[0039] Step S321: The located keyword is used as the center triple. Other triples related to the center triple are used to generate a context subgraph;
[0040] Step S322: Use the KG2SEQ module to perform sequence transformation on the context subgraph and return it to the knowledge retrieval device.
[0041] In a preferred embodiment of the present invention, step S4 further includes the following steps:
[0042] Step S41: Use the BERT model to encode the detected result X i and each knowledge vector Get X i and The representation of:
[0043] V x =W1BERT CLS (X i ),
[0044]
[0045] Where W1 and W2 are learnable weights, V x and V j It is a vector representation after linear transformation through the BERT model;
[0046] Step S42: Use V x and V j The result of the inner product calculation between them is X. i and each knowledge vector Correlation score C between them:
[0047]
[0048] Among them, the subset R i As a positive sample As a negative sample Represents the sample difference set, used to train the evaluator; given a positive sample and a negative sample r p and r n The ranking loss is defined as:
[0049] L = max{0, MC(X) i-1 ,r p )+C(X i-1 ,r n )}
[0050] Where M is an empirically determined boundary, calculated during inference. The fraction C j Based on C j Sort the scores and select the top N most relevant pieces of knowledge as R. i Input it into the description text generation.
[0051] In a preferred embodiment of the present invention, step S41 further includes the following steps:
[0052] Step S411: In X i Each R in j and s j Add a [SEP] identifier after it, in the X i Add a [CLS] identifier to the header to accommodate BERT's format;
[0053] Step S412: For the paragraph identifier, mark the token from the context subgraph as 0, mark the token of the detection result as 1, and finally apply a linear layer to the embedding of the [CLS] token.
[0054] In a preferred embodiment of the present invention, step S5 further includes the following steps:
[0055] By connecting the detection results with external knowledge as input, the conditional probability distribution is modeled using a GPT-2 model based on the Transformer structure, and introductory text is generated accordingly.
[0056] The technical solution of the present invention has the following advantages compared with the prior art:
[0057] 1. When performing deep feature extraction in the target detection module, a residual block structure and identity mapping are adopted to solve the gradient vanishing problem. At the same time, a scaling factor is used to control the intensity and speed of the residual transformation, so that the model can learn features more stably.
[0058] 2. In order to obtain more knowledge related to the target keywords, a related keyword prediction model is introduced to enrich the target detection results and refine the description text. The predicted keywords do not directly predict knowledge triples, but are generated in an interpretable way, which greatly reduces the space for searching knowledge triples.
[0059] 3. This paper proposes the KG2SEQ module, which converts the context subgraph into a text sequence as local structural information. Therefore, when generating introductory text, it is not necessary to have all the information of the knowledge graph, but only to focus on the target detection results and the text sequence after the conversion of the keyword-related subgraph, thus reducing computational redundancy. Attached Figure Description
[0060] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.
[0061] Figure 1 This is a first flowchart of the image description method provided in the embodiments of the present invention.
[0062] Figure 2 This is a flowchart of the target detection method provided in the embodiments of the present invention.
[0063] Figure 3 This is a second flowchart of the image description method provided in the embodiments of the present invention.
[0064] Figure 4 This is a flowchart of the image target detection method provided in the embodiments of the present invention.
[0065] Figure 5 This is a flowchart of the image feature pooling method provided in an embodiment of the present invention.
[0066] Figure 6 This is a flowchart of the image feature convolution method provided in the embodiments of the present invention.
[0067] Figure 7 This is a flowchart of the training keyword prediction method provided in the embodiments of the present invention.
[0068] Figure 8 This is a flowchart of the keyword and subgraph node matching method provided in the embodiments of the present invention.
[0069] Figure 9 This is a flowchart of the keyword matching method provided in the embodiments of the present invention.
[0070] Figure 10 This is a flowchart of keyword knowledge sequence generation provided in an embodiment of the present invention.
[0071] Figure 11 This is a flowchart of the keyword knowledge sequence sorting method provided in the embodiments of the present invention.
[0072] Figure 12 This is a flowchart of the keyword knowledge sequence processing method provided in the embodiments of the present invention.
[0073] Figure 13 This is a feature extraction map of the target detection method provided in the embodiments of the present invention.
[0074] Figure 14 This is a feature dimension diagram of the target detection method provided in the embodiments of the present invention.
[0075] Figure 15 This is a flowchart of the feature stitching process for the target detection method provided in this embodiment of the invention.
[0076] Figure 16 This is a residual block structure diagram of the standard detection method provided in the embodiment of the present invention. Detailed Implementation
[0077] Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the present invention, and should not be construed as limiting the present invention.
[0078] refer to Figures 1-3As shown, in some embodiments, one aspect of the present invention proposes an image description method based on an external knowledge graph, the method comprising the following steps:
[0079] Step S1: Perform object detection on the image to be introduced. Object detection includes image preprocessing, image feature extraction, object detection, post-processing optimization, and outputting the detection results.
[0080] Specifically, refer to Figure 4 , Figures 13-16 As shown, in step S1, the method can be further subdivided into the following steps:
[0081] Step S11: Extract features from the image by performing convolution operations through convolutional layers and downsampling through pooling operations to reduce the number of parameters and computational cost.
[0082] Specifically, in step S11, refer to Figure 5 , Figures 13-14 As shown, the method can be further divided into the following steps:
[0083] Step S111: Extract features from the image by passing it through convolutional kernels of size 3×3 and number of channels 32, 64 and 128 respectively;
[0084] Step S112: The extracted features are downsampled through a max pooling layer with a stride of 2 to obtain feature maps with a size of 11×11 and 32, 64 and 128 channels.
[0085] Step S12: Repeat the residual block connection operation 20 times to extract features at a deeper level. Skip connections are used to solve the gradient vanishing problem, and a scaling factor is used for identity mapping to control the intensity and speed of residual transformation, so that the model can learn features more stably.
[0086] Specifically, in step S12, refer to Figure 6 , Figure 16 As shown, the method can be further divided into the following steps:
[0087] Step S121: The feature map with size 11×11 and number of channels 256 is passed through three layers of convolution kernels with different numbers of sizes of 3×3 and number of channels 32. The three feature maps with the same size and number of channels are then concatenated to obtain a feature map with size 11×11 and number of channels 96.
[0088] Step S122: Perform identity mapping on the feature map using a convolution kernel of size 1×1 and number of channels 256 to ensure that the size is consistent with the initial feature map.
[0089] Step S13: For all detected results, post-processing optimization is performed using the NMS algorithm to remove redundant detection results.
[0090] Specifically, in step S13, the IOU of each of the detected results is calculated, and the one with the largest IOU value is selected as the final detection result.
[0091] Step S2: Use a text generation model as a keyword predictor to predict keywords from the image detection results, and then input the predicted keywords into a knowledge retrieval tool to retrieve knowledge information from an external knowledge graph;
[0092] Specifically, refer to Figure 7 As shown, in step S2, the method can be further subdivided into the following steps:
[0093] Step S21: Given the currently detected result X i The model predicts a set of associated keywords as K. i The probability distribution is modeled using the GPT-2 model based on the Transformer architecture, where:
[0094]
[0095] Where K <j Let p represent all predicted keywords up to the j-th keyword, where p is the probability distribution and q is the sequence length.
[0096] Specifically, in step S21, based on the GPT-2 model, the model learns to infer the next most relevant keyword from the predicted keywords. Through continuous iteration, the model can generate a keyword sequence related to the initial keywords.
[0097] Step S22: Optimize the keyword predictor using maximum likelihood training and next token prediction loss, and train it from real knowledge using the RAKE algorithm. i Extract keywords for K i Provide labels to train this keyword predictor.
[0098] Specifically, in step S22, the RAKE algorithm is used to extract keywords from real text related to the object detection category as labels for model training.
[0099] Step S3: Use the knowledge retrieval tool to search the external knowledge graph, find the knowledge subgraphs associated with the keywords, and input the knowledge subgraphs and keywords together into the subgraph knowledge evaluator for knowledge evaluation.
[0100] Specifically, in step S3, refer to Figure 8 As shown, the method can be further divided into the following steps:
[0101] Step S31: Use a knowledge retrieval tool to match the query information with nodes in the knowledge graph.
[0102] Specifically, in step S31, refer to Figure 9 As shown, the method can be further divided into the following steps:
[0103] Step S311: Use the Boyer-Moore matching algorithm to match keywords with nodes in the knowledge graph;
[0104] Step S312: After a successful match, locate the keyword nodes in the knowledge graph.
[0105] Step S32: After a successful match, the subgraph containing the query information is transformed into a sequence and returned to the knowledge retrieval device.
[0106] Specifically, in step S31, refer to Figure 10 As shown, the method can be further divided into the following steps:
[0107] Step S321: The located keyword is used as the center triple. Other triples related to the center triple are used to generate a context subgraph;
[0108] Step S322: Use the KG2SEQ module to perform sequence transformation on the context subgraph and return it to the knowledge retrieval device.
[0109] Step S33: Input the transformed subgraph sequence and keywords into the subgraph knowledge evaluator for evaluation.
[0110] Step S4: Use a subgraph knowledge evaluator to convert the knowledge subgraph into a text sequence while preserving the subgraph structure and node information. Then match it with keywords to obtain the knowledge most relevant to the target detection results.
[0111] Specifically, in step S4, refer to Figure 11 As shown, the method can be further divided into the following steps:
[0112] Step S41: Use the BERT model to encode the detected result X i and each knowledge vector Get X i and The representation of:
[0113] V x =W1BERT CLS (X i ),
[0114]
[0115] Where W1 and W2 are learnable weights, V x and V j It is a vector representation after linear transformation by the BERT model.
[0116] Specifically, in step S41, refer to Figure 12 As shown, the method can be further divided into the following steps:
[0117] Step S411: In X i Each R in j and s j Add a [SEP] identifier after it, in the X i Add a [CLS] identifier to the header to accommodate BERT's format;
[0118] Step S412: For paragraph identifiers, tokens from the context subgraph are marked as 0, tokens from the detected results are marked as 1, and finally, a linear layer is applied to the embedding of the [CLS] token.
[0119] Step S42: Use V x and V j The result of the inner product calculation between them is X. i and each knowledge vector Correlation score C between them:
[0120]
[0121] Among them, the subset R i As a positive sample As a negative sample Represents the sample difference set, used to train the evaluator; given a positive sample and a negative sample r p and r n The ranking loss is defined as:
[0122] L = max{0, MC(X) i-1 ,r p )+C(X i-1 ,r n )}
[0123] Where M is an empirically determined boundary, calculated during inference. The fraction C j Based on C j Sort the scores and select the top N most relevant pieces of knowledge as R. i Input it into the description text generation.
[0124] Step S5: Use a language model to generate text containing relevant knowledge content.
[0125] Specifically, in step S5, the method can be further subdivided into the following steps:
[0126] Step S51: Transfer the detected result X i and external knowledge R i Connect them together as input to generate S i The conditional probability distribution is modeled using the GPT-2 model based on the Transformer architecture, and the introductory text is generated accordingly.
[0127] In this specification, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Furthermore, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described herein, as well as the features of those different embodiments or examples.
[0128] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.
Claims
1. An image description method based on an external knowledge graph, characterized in that, Includes the following steps: Step S1: Perform object detection on the image to be introduced, including: image preprocessing, image feature extraction, object detection, post-processing optimization, and outputting the detection results; Step S2: Use a text generation model as a keyword predictor to predict keywords in the detection results, and then input the predicted keywords into the knowledge retrieval machine to retrieve knowledge information from the external knowledge graph; Step S21: Given the currently detected results The model predicts a set of associated keywords as follows: The probability distribution is modeled using the GPT-2 model based on the Transformer architecture, where: in Indicates until the All predicted keywords up to the specified number of keywords. It is a probability distribution. It is the sequence length; Step S22: Optimize the keyword predictor using maximum likelihood training and next token prediction loss, training from real-world knowledge using the RAKE algorithm. Extract keywords to Provide labels to train this keyword predictor; Step S3: Use a knowledge retrieval tool to search the external knowledge graph, obtain the knowledge subgraph associated with the keyword, and then input the knowledge subgraph and the keyword into the subgraph knowledge evaluator for knowledge evaluation; Step S31: Use a knowledge retrieval tool to match the query information with nodes in the knowledge graph; Step S32: After a successful match, the subgraph containing the query information is transformed into a sequence and returned to the knowledge retrieval device; Step S33: Input the transformed subgraph sequence and keywords into the subgraph knowledge evaluator for evaluation; Step S4: Use a subgraph knowledge evaluator to convert the knowledge subgraph into a text sequence while preserving the subgraph structure and node information, and then match it with the keywords to obtain the knowledge with the highest correlation to the target detection result; Step S41: Use the BERT model to encode the detected results. and each knowledge vector ,get and The representation of: in, and These are learnable weights. and It is a vector representation after linear transformation through the BERT model; Step S42: Use and The results of the inner product calculation between them have been detected. and each knowledge vector Correlation score between : Among them, the subset As a positive sample As a negative sample This represents the difference set of samples, used to train the evaluator; given a positive sample and a negative sample. and The ranking loss is defined as: in, It is an empirically determined boundary; during reasoning, all... fractions ,based on Sort the scores and select the top N most relevant pieces of knowledge as... Input it into the description text generation; Step S5: Use a language model to generate text containing relevant knowledge content.
2. The image description method based on an external knowledge graph according to claim 1, characterized in that, In step S1, the method further includes the following steps: Step S11: Extract features from the image to be introduced by performing convolution operations through convolutional layers and downsampling through pooling operations to reduce the number of parameters and computational cost; Step S12: Repeat the residual block connection operation a preset number of times to extract features at a deeper level. Solve the gradient vanishing problem by skip connections and use a scaling factor to perform identity mapping to control the intensity and speed of residual transformation. Step S13: For all detected results, post-processing optimization is performed using the NMS algorithm to remove redundant detection results.
3. The image description method based on an external knowledge graph according to claim 2, characterized in that, In step S11, the method further includes the following steps: Step S111: Extract features from the images to be introduced using convolutional kernels of size 3×3 and number of channels 32, 64 and 128 respectively; Step S112: The extracted features are downsampled through a max pooling layer with a stride of 2 to obtain feature maps with a size of 11×11 and 32, 64 and 128 channels.
4. The image description method based on an external knowledge graph according to claim 2, characterized in that, In step S12, the method further includes the following steps: Step S121: The feature map with size 11×11 and number of channels 256 is passed through three layers of convolution kernels with different numbers of sizes of 3×3 and number of channels 32. The three feature maps with the same size and number of channels are then concatenated to obtain a feature map with size 11×11 and number of channels 96. Step S122: Perform identity mapping on the feature map using a convolution kernel of size 1×1 and number of channels 256 to ensure that the size is consistent with the initial feature map.
5. The image description method based on an external knowledge graph according to claim 2, characterized in that, In step S13, the method further includes: calculating the IOU of each of the detected results, and selecting the one with the largest IOU value as the final detection result.
6. The image description method based on an external knowledge graph according to claim 1, characterized in that, In step S31, the method further includes the following steps: Step S311: Use the Boyer-Moore matching algorithm to match keywords with nodes in the knowledge graph; Step S312: After a successful match, locate the keyword nodes in the knowledge graph.
7. The image description method based on an external knowledge graph according to claim 1, characterized in that, In step S32, the method further includes the following steps: Step S321: The located keyword is used as the central triple, and other triples related to the central triple are used to generate a contextual subgraph; Step S322: Use the KG2SEQ module to perform sequence transformation on the context subgraph and return it to the knowledge retrieval device.