An image description generation method based on target detection and knowledge enhancement

Abstract

This invention discloses an image description generation method based on object detection and knowledge enhancement. It combines region bounding boxes and category labels from the object detection stage, utilizing advanced tools such as face recognition and product recognition for deeper information mining, thereby obtaining fine-grained background knowledge. In the decoding stage, an LBPF attention mechanism is introduced to enhance the connections between words, thus generating coherent, fine-grained descriptions rich in background knowledge, providing a new approach to image description generation incorporating knowledge graphs. Compared with traditional methods of incorporating knowledge graphs, this invention can generate descriptions with fine-grained background knowledge and relationships between fine-grained entities; the described targets are more specific, the content is richer, and the readability is higher.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and more specifically to an image description generation method based on object detection and knowledge enhancement. Background Technology

[0002] Natural image description generation is a challenging research area. While relatively easy for humans, it presents a significant challenge for machines, requiring not only models to understand the content of images but also the ability to express the relationships between them using natural language. Image description solves the problem of automatically generating corresponding descriptive text for a given image. It is an interdisciplinary field combining computer vision, natural language processing, and machine learning, and represents a highly challenging research problem in artificial intelligence. Since most human communication relies on natural language, both written and spoken, enabling computers to describe the visual world will offer numerous applications. For example, it has significant implications for image information retrieval on the internet, early childhood education, and assistive technologies for the visually impaired.

[0003] In recent years, there has been a growing interest in describing visual scenes. Traditional image captioning has been largely straightforward: given an image, its features are extracted, fed into a translation model, and a caption is generated. The specific object the caption focuses on and its style are determined by the training data, offering little control. We aim to control the generated caption, such as the key object and region being described. For instance, we can decide whether the caption describes the background or a foreground object, and also determine the level of detail.

[0004] Traditional text generation tasks rely solely on input images, lacking richer "knowledge" information. Consequently, the generated text is often dull and lacks engaging content. In text generation tasks, "knowledge" serves as a supplement to the input text and its context, and can be obtained through various methods and information sources, including but not limited to keywords, topics, linguistic features, knowledge bases, and knowledge graphs.

[0005] Current popular methods for image description generation using knowledge graphs can only obtain coarse-grained category labels for objects through object detection tools, and the number of detected categories is limited. In terms of generated image descriptions, the sentences lack coherence, and the descriptions cannot fully summarize the meaning expressed by the image. For example, the knowledge graph-based image description generation model CNet-NIC incorporates the embeddings of knowledge graph entities into the model, but it does not learn the fine-grained relationships between entities, only describing the coarse-grained category labels in the image. CNet-NIC does not generate fine-grained descriptions focused on salient regions of the image, and the generated descriptive language lacks coherence. Summary of the Invention

[0006] To address the shortcomings of existing technologies, this invention provides an image description generation method based on object detection and knowledge enhancement, which solves the problems of low readability of image title description languages, only having coarse-grained surface information, and lacking fine-grained relationships between entities in existing technologies.

[0007] This invention provides an image description generation method based on object detection and knowledge enhancement, comprising the following steps:

[0008] Step 1: Obtain an existing image description dataset with annotations. Identify each word in the title text of the dataset, obtain fixed-length word vectors, and form a corresponding vocabulary. Extract feature vectors of regions of interest from the image, identify image entity bounding boxes and image entity categories, and obtain fixed-length category feature word vectors. Recognize the image and obtain fixed-length image entity feature vectors. Detect the image and obtain the target region bounding boxes and target labels contained in the image.

[0009] Step 2: The target label and target region bounding box are used as inputs to the classification and detection method. The classification and detection method first matches the target label with the target region bounding box using face recognition and general product recognition methods to obtain the target feature label set within the region. Then, based on the matching, the classification and detection method uses face recognition and general product recognition methods to obtain the specific information of the target, namely face feature label and object feature label, based on the target region bounding box.

[0010] Step 3: Using knowledge graph terminology retrieval methods, obtain the background information tag set of the target in the image by combining the target tag, face feature tag, and object feature tag;

[0011] Step 4: Use the target feature label set and the background information label set as inputs to the deep learning RNN network to obtain the target feature vector and the background information feature vector, respectively.

[0012] Alternatively, the target feature label set and the background information label set can be used as inputs to a deep learning RNN network to obtain the image features generated in the deep learning RNN network. Based on the image entity feature vector, the background information feature vector, and the image features generated in the deep learning RNN network, the fused feature vector of the image can be obtained.

[0013] Step 5: Use the target feature vector and background information feature vector as input to the Long Short-Term Memory (LSTM) network, and the category feature word vector as output to train the LSM network.

[0014] Alternatively, the fused feature vectors can be used as the input to the Long Short-Term Memory (LSTM) network, and the category feature word vectors can be used as the output of the LSM network to achieve training of the LSM network.

[0015] Step 6: Use the trained Long Short-Term Memory network to generate image descriptions.

[0016] Furthermore, the specific method for obtaining the fused feature vector of the image in step 4 is to sequentially concatenate the target feature vector, the background information feature vector, and the image features generated in the deep learning RNN network to obtain the fused feature vector of the image.

[0017] Furthermore, the image features generated in the deep learning RNN network are:

[0018] r = F-RCNN(I)

[0019] Where r is the image entity feature vector, F-RCNN() is the Faster-RCNN tool function, and I is the image in the training set.

[0020] Furthermore, the fused feature vector of the image is:

[0021] V = Concatenate(i, d, r)

[0022] Where V is the fusion feature vector of the image, i is the image entity feature vector, and d is the background information feature vector.

[0023] Furthermore, the image entity feature vector is:

[0024] i = RNN(r0)

[0025] The background information feature vector is:

[0026] d = RNN(e0)

[0027] Where r0 is the label set of the target features and e0 is the label set of background information.

[0028] The beneficial effects of this invention are:

[0029] 1. This invention uses global image features, target knowledge features within the image, and descriptive text features to generate image descriptions. It employs object detection and knowledge enhancement methods to improve the richness and diversity of language generated by the model. Compared to models without knowledge graph integration, the descriptions generated by the method described in this paper are more consistent with everyday expressions, contain background information about objects in the image, and have richer semantic information; they improve the semantic-focused evaluation metrics SPICE and METEOR by 1% and 0.6%, respectively.

[0030] 2. This invention combines the bounding boxes and category labels in the target detection stage, and utilizes advanced tools such as face recognition and product recognition to perform deeper information mining; it uses a ResNet model to extract global features from the image and encode them into feature vectors, while for text, it uses a Long Short-Term Memory network to extract features from the text; in the decoding stage, it introduces the LBPF attention mechanism to enhance the connection between words, thereby generating a coherent description rich in fine-grained background knowledge. Attached Figure Description

[0031] The features and advantages of the invention will be more clearly understood by referring to the accompanying drawings, which are schematic and should not be construed as limiting the invention in any way. In the drawings:

[0032] Figure 1 This is a general framework diagram of the image description generation method in an embodiment of the present invention;

[0033] Figure 2 This is an example diagram illustrating the model generation effect without incorporating a knowledge graph in an embodiment of the present invention;

[0034] Figure 3 This is an example diagram illustrating the model generation effect of incorporating knowledge graphs in an embodiment of the present invention. Detailed Implementation

[0035] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0036] The present invention will be further illustrated below with reference to specific embodiments. Those skilled in the art should understand that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. Modifications to the present invention in various equivalent forms all fall within the scope defined by the appended claims.

[0037] This invention provides step 1: acquiring an existing image description dataset with annotations, recognizing each word in the title text of the dataset, obtaining fixed-length word vectors and forming a corresponding vocabulary; obtaining fixed-length category feature word vectors by extracting feature vectors of regions of interest in the image, recognizing image entity region boxes and image entity categories; recognizing the image to obtain fixed-length image entity feature vectors; detecting the image to obtain target region boxes and target labels contained in the image;

[0038] Step 2: The target label and target region bounding box are used as inputs to the classification and detection method. The classification and detection method first matches the target label with the target region bounding box using face recognition and general product recognition methods to obtain the target feature label set within the region. Then, based on the matching, the classification and detection method uses face recognition and general product recognition methods to obtain the specific information of the target, namely face feature label and object feature label, based on the target region bounding box.

[0039] Step 3: Using knowledge graph terminology retrieval methods, obtain the background information tag set of the target in the image by combining the target tag, face feature tag, and object feature tag;

[0040] Step 4: Use the target feature label set and the background information label set as inputs to the deep learning RNN network to obtain the target feature vector and the background information feature vector, respectively.

[0041] Alternatively, the target feature label set and the background information label set can be used as inputs to a deep learning RNN network to obtain image features generated by the deep learning RNN network. The image entity feature vector, background information feature vector, and image features generated by the deep learning RNN network can be concatenated sequentially to obtain the image fusion feature vector.

[0042] The image features generated by the deep learning RNN network are as follows:

[0043] r = F-RCNN(I)

[0044] In the formula, r is the image entity feature vector, F-RCNN() is the Faster R-CNN tool function, and I is the image in the training set;

[0045] The fused feature vector of the image is:

[0046] V = Concatenate(i, d, r)

[0047] In the formula, V is the fused feature vector of the image, i is the image entity feature vector, and d is the background information feature vector.

[0048] i = RNN(r0)

[0049] d = RNN(e0)

[0050] In the formula, r0 is the label set of the target features, and e0 is the label set of background information;

[0051] Step 5: Use the target feature vector and background information feature vector as input to the Long Short-Term Memory (LSTM) network, and the category feature word vector as output to train the LSM network.

[0052] Alternatively, the fused feature vectors can be used as the input to the Long Short-Term Memory (LSTM) network, and the category feature word vectors can be used as the output of the LSM network to achieve training of the LSM network.

[0053] Step 6: Use the trained Long Short-Term Memory network to generate image descriptions.

[0054] The embodiments of the present invention will be further described below:

[0055] Step 1: Find an open-source image description dataset with well-annotated descriptions, and split the dataset into a training set, a validation set, and a test set. The image description dataset is the mscoco2014 dataset, which is divided into a training set of 113,287 images, a validation set of 5,000 images, and a test set of 5,000 images.

[0056] Step 11: Obtain the descriptive text corresponding to each image, identify each word in the image description, obtain fixed-length word vectors, and construct the corresponding vocabulary. Specifically:

[0057] Step 12: Convert all descriptive text in the image description dataset into a vocabulary of words;

[0058] Step 13: Identify each word in the sentence description, and the resulting word vectors will have 1024 dimensions, each representing a word feature.

[0059] Step 14: For the image, use the Faster R-CNN tool to extract the feature vector of the region of interest and identify the image entity region bounding box and the image entity category. The resulting image region of interest feature vector has each dimension representing the image feature, with a dimension of 2048.

[0060] Step 15: For the image entity categories in Step 14, identify the entity category names and obtain fixed-length category feature word vectors;

[0061] Step 16: For the image entity region bounding boxes in Step 14, use the ResNet tool to identify the images in the entity regions and obtain fixed-length image entity feature vectors;

[0062] Further, steps 14-16 for extracting the feature vector of the region of interest in the image are as follows: Image features r of image I are generated by a pre-trained Faster R-CNN. For input image I, each image uses the feature map extracted by ResNet-101 and the region R extracted by Faster R-CNN.

[0063] r = F-RCNN(I)

[0064] Step 17: Use the YOLO object detection tool to detect objects in the image and obtain the bounding boxes and labels of the objects contained in the image;

[0065] The steps to obtain the target label are as follows: For image I in the dataset, use the YOLO object detector to identify the objects in the image, i.e., O = YOLO(I);

[0066] Step 2: The target label obtained in Step 17 is sent to the classification detector. The classification detector selects to use the face recognition module or the general commodity recognition module based on the information of the object label to recognize the region box obtained in Step 17.

[0067] Step 21: If the target detection result is Person, then this region map is sent to the face and celebrity detection tool to identify the specific person's information; if the target detection result is an object, then it is further sent to JD Cloud general product detection to identify the product name; if there is no product name, then the object name detected by the target is used.

[0068] Step 3: Based on the target region bounding box from Step 21, use the face recognition module and the general product recognition module to obtain specific information about the object, namely, face feature labels and object feature labels; the steps for obtaining face feature labels and object feature labels in Step 3 are as follows:

[0069] Step 31: Obtain the object name in the image through O = YOLO(I), determine whether the object set is a person. If it is a person, send it to the face and celebrity detection module to obtain the person and face information labels; if it is an object, send it to the product recognition module; if there is no output through the celebrity recognition and product recognition modules, output the original object name, that is, finally obtain the output name set of the classification detector.

[0070] Step 4: Input the feature label list obtained in Step 17.3 into the ConceptNet knowledge graph related term retrieval module to obtain the background information label set of objects in the image;

[0071] The steps for obtaining the background information label set of objects in the image are as follows:

[0072] Step 41: Query the relevant entity terms in the ConceptNet commonsense knowledge graph by querying the relevant term API to obtain the term set e0 = ConceptNet(m) related to m and the term set r0 = ConceptNet(M) related to the entire set of objects M in the graph;

[0073] Step 5: Map the background information label set obtained in Step 4 to the corresponding vector space embedding through deep learning RNN pre-training to obtain the background information feature vector;

[0074] The steps for obtaining the background information feature vector are as follows:

[0075] Step 51: Generate vector space embeddings for e0 and r0 using a pre-trained RNN.

[0076] i = RNN(r0)

[0077] d = RNN(e0)

[0078] Step 6: Create the decoder module of the Look Back and Predict Forward method Long Short-Term Memory network by adding a fully connected layer between the input of the convolutional neural network in the encoder module and the input of the recurrent neural network in the decoder module.

[0079] Step 61, the Look Back and Predict Forward method of creating a Long Short-Term Memory (LSTM) network includes: given a k-dimensional image feature vector... and the current hidden state Predict y t+1 and y t+2 Let the attention function be f. att If LSTM2 is F2, then

[0080]

[0081]

[0082]

[0083] Predict the next word, and so, Continue through att t And LSTM2

[0084]

[0085]

[0086]

[0087] Using the above formula, the final predicted word is calculated by summing the two results:

[0088]

[0089] Step 8: Input the word vectors of the image description text obtained in Step 2, the image features extracted in Step 5, and the background information feature vectors obtained in Step 5 into the Long Short-Term Memory network, and train the image features and the corresponding image description text in the Long Short-Term Memory network.

[0090] The training steps for the image features and corresponding image description text in the Long Short-Term Memory network are as follows:

[0091] Step 81: Concatenate the background information features obtained in Step 5 with the image features obtained from Faster R-CNN in Step 16.

[0092] Input: V = Concatenate(i, d, r)

[0093] The concatenated features obtained in step 81 are used as input to the descriptive generative model. The descriptive generative model uses a Long Short-Term Memory (LSTM) network with a Look Back and Predict Forward method.

[0094] Soft Vector:w=softmax(LSTM(V))

[0095] Step 82: In the decoder module of the Long Short-Term Memory Network, predict and generate words for each sentence starting from the preset start flag.

[0096] Step 83: Using the beam search algorithm, we use a linear layer to convert the decoder output into a score for each word in the vocabulary. We then select the words with the highest probability distribution from the best candidate word set to form a sentence, thus obtaining the optimal title sequence.

[0097] The following are specific examples to illustrate this:

[0098] This example compares an image caption generation model with an image description model based on object detection and knowledge enhancement, FTCD+KG. The example images were randomly selected from the test set of the MSCOCO2014 dataset. Figure 2As shown, the description without object detection and knowledge graphs, "a woman is holding a frisbee in a room," accurately describes the content of the image, including information such as the person, actions, and scene. Figure 3 As shown, the description using object detection and knowledge graph integration is "a young girl with long hair was holding a hunt mice cat with a bicycle behind her." Compared to the model without knowledge graph integration, this description is more in line with everyday expression, contains more information about objects in the image, and has richer semantic information. To compare the performance of these two models, this example uses BLEU, METEOR, CIDER, and SPICE automatic evaluation metrics for comparison, as shown in Table 1.

[0099]

[0100] Table 1

[0101] The experiments yielded the following conclusions: This invention has a good effect on image descriptions integrated with knowledge graphs, improving indicators such as BLEU and METEOR. It has a wide range of applications and can effectively compensate for the shortcomings of previous image description methods that incorporate knowledge graphs.

[0102] Although embodiments of the invention have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the invention, and such modifications and variations all fall within the scope defined by the appended claims.

Claims

1. An image description generation method based on object detection and knowledge enhancement, characterized in that, Includes the following steps: Step 1: Obtain an existing image description dataset with annotations, identify each word in the title text of the dataset, obtain fixed-length word vectors, and form a corresponding vocabulary; extract the feature vectors of the regions of interest in the image, identify the image entity bounding boxes and image entity categories to obtain fixed-length category feature word vectors; perform image recognition to obtain fixed-length image entity feature vectors; Detect the image and obtain the bounding boxes and labels of the target regions contained in the image; Step 2: The target label and target region bounding box are used as inputs to the classification and detection method. The classification and detection method first matches the target label with the target region bounding box using face recognition and general product recognition methods to obtain the target feature label set within the region. Then, based on the matching, the classification and detection method uses face recognition and general product recognition methods to obtain the specific information of the target, namely face feature label and object feature label, based on the target region bounding box. Step 3: Using knowledge graph terminology retrieval methods, obtain the background information tag set of the target in the image by combining the target tag, face feature tag, and object feature tag; Step 4: Use the target feature label set and the background information label set as inputs to the deep learning RNN network to obtain the target feature vector and the background information feature vector, respectively. Alternatively, the target feature label set and the background information label set can be used as inputs to a deep learning RNN network to obtain the image features generated in the deep learning RNN network. Based on the image entity feature vector, the background information feature vector, and the image features generated in the deep learning RNN network, the fused feature vector of the image can be obtained. Step 5: Use the target feature vector and background information feature vector as input to the Long Short-Term Memory (LSTM) network, and the category feature word vector as output to train the LSM network. Alternatively, the fused feature vectors can be used as the input to the Long Short-Term Memory (LSTM) network, and the category feature word vectors can be used as the output of the LSM network to achieve training of the LSM network. Step 6: Use the trained Long Short-Term Memory network to generate image descriptions.

2. The image description generation method based on target detection and knowledge enhancement as described in claim 1, characterized in that, The specific method for obtaining the fused feature vector of the image in step 4 is to sequentially concatenate the target feature vector, the background information feature vector, and the image features generated in the deep learning RNN network to obtain the fused feature vector of the image.

3. The image description generation method based on target detection and knowledge enhancement as described in claim 1 or 2, characterized in that, The image features generated by deep learning RNN networks are: r = F-RCNN(I) Where r is the image entity feature vector, F-RCNN() is the Faster-RCNN tool function, and I is the image in the training set.

4. The image description generation method based on target detection and knowledge enhancement as described in claim 3, characterized in that, The fused feature vector of the image is: V = Concatenate(i, d, r) Where V is the fusion feature vector of the image, i is the image entity feature vector, and d is the background information feature vector.

5. The image description generation method based on target detection and knowledge enhancement as described in claim 4, characterized in that, The image entity feature vector is: i = RNN(r0) The background information feature vector is: d = RNN(e0) Where r0 is the label set of the target features and e0 is the label set of background information.

Images