Image natural language description method combined with question answering

Abstract

A kind of image natural language description method of joint question answering, including three steps: step one, first using image segmentation model to extract the features of image target and image background, obtain the division of different categories at pixel level, obtain the segmentation feature map of target and background;Step two, question generation module generates a relationship feature map containing the information of the target of interest by constructing an implicit scene type representation, and generates several semantically related guide questions in multiple granularities;Step three, the loss function of joint question answering module introduces contrast learning, and the relationship feature map and guide question are jointly multi-modal embedded representation, the model can generate long text answers related to the question as the fine semantic description of image content by training.

Description

Technical Field

[0001] This invention belongs to the fields of computer vision and natural language processing. Background Technology

[0002] Image captioning generation is a multimodal task spanning text and images, aiming to generate corresponding natural language descriptions from images. This task is quite easy for humans but very challenging for computers. With the rise of deep learning, more and more people are attempting to use neural networks to solve the image captioning problem for machines.

[0003] However, due to the diversity of natural language descriptions, they possess varying standard forms. Training with a specific dataset can only yield image content descriptions conforming to the distribution of that dataset, limiting its applicability. Furthermore, most image description methods passively generate monotonous sentences without considering the relationship between the scene and the object of interest within the image. Therefore, they struggle to describe different, multi-scale objects simultaneously and easily overlook potential connections. This mechanical image description deviates significantly from human semantic cognition of images, often resulting in a lack of mutual understanding when interacting with real humans.

[0004] Image caption generation generally consists of two sub-modules: image feature extraction and text generation. A common method for image feature extraction is to use image recognition neural network models to extract the target; however, this can lead to missing image information and biased feature extraction. Furthermore, sentence generation based on feature extraction often focuses only on specific targets and cannot generate multi-granular descriptions from the rich semantic information of the image, easily resulting in significant loss of cross-modal information. Therefore, machine-generated image captions still lag far behind human natural image perception. Summary of the Invention

[0005] This invention addresses the shortcomings of the prior art by providing a joint question-answering method for image natural language description. Based on a visual question-answering model, it generates image content descriptions by designing an image segmentation module, a question generation module, and a joint question-answering module. It generates multiple semantically related questions by utilizing different scale regions in the relational feature map, with a question-answer correspondence. Using guiding questions with multi-granular features, it generates a refined description of the image scene. This method can capture the probability of implicit facts occurring in the image and generate natural language descriptions that are more consistent with human semantic cognition of images.

[0006] The present invention adopts the following technical solution:

[0007] A joint question-answering method for image natural language description is characterized by generating refined descriptions of image content based on a visual question-answering model. First, an image segmentation module obtains segmentation feature maps for classifying image targets and backgrounds. Second, a question generation module constructs implicit scene type representations, generating multi-granularity guiding questions centered on the target of interest. Finally, a joint question-answering module introduces a contrastive learning loss function to perform joint multimodal embedding representations of the relational feature maps and guiding questions. This method, based on visual question answering, generates natural language descriptions of image content according to the question-answer correspondence.

[0008] A joint question-answering method for image natural language description includes three steps:

[0009] Step 1: First, use an image segmentation model to extract features from the image target and background to obtain pixel-level classifications of different categories and acquire segmentation feature maps of the target and background.

[0010] Step 2: The question generation module generates a relational feature map containing information about the target of interest by constructing an implicit scene type representation, and generates several semantically related guiding questions at multiple granularities.

[0011] Step 3: The joint question answering module introduces a contrastive learning loss function to perform joint multimodal embedding representation of the relation feature map and the guiding question. Through training, this model can generate long text answers related to the question, serving as a refined semantic description of the image content.

[0012] Regarding step one, the present invention provides a preferred scheme for extracting image features.

[0013] Regarding step two, this invention discloses a question generation model based on the LSTM model. Its feature is that it can process the segmentation feature map, and by constructing an implicit scene type representation, it first generates a relational feature map containing information about the target of interest. Then, with the target of interest as the center, it establishes connections between the target of interest in the image and the target of interest, as well as between the target of interest and the background, at multiple scales. The generated multi-granularity guiding questions serve as a link in the subsequent joint question answering.

[0014] Regarding step three, this invention discloses a joint question-answering model based on the BUTD (bottom-up, top-down) model. Its feature is that it introduces a contrastive learning loss function, which can combine relational feature maps and guiding questions to improve the model's cross-modal learning ability, enhance the model's understanding of the semantic relationship between images and question answers, and generate a refined description of image content.

[0015] Specifically,

[0016] Step 1: Image Segmentation

[0017] 1.1 We utilize publicly available image semantic segmentation datasets, where batch images have pixel-level category annotations.

[0018] 1.2 Deep learning methods are used to train the image segmentation dataset and construct an image segmentation neural network model. The task of image segmentation is to perform dense prediction on an image. By labeling different targets with specific colors, each pixel can be assigned a category to its target or closed region.

[0019] 1.3 Save the model weights of the trained image segmentation neural network. This network model can process the original image, distinguish different targets and backgrounds in the image, and finally output segmentation feature maps between targets and between targets and backgrounds.

[0020] Step Two: Question Generation

[0021] 2.1 This method processes existing visual question generation datasets, classifying the question categories within the datasets. Different question categories view the target and its relationships from multiple perspectives. Multiple question categories for the same image not only focus on different targets but also on image regions of the same target at different scales. Simultaneously, the answers and questions in the dataset are merged to generate a complete natural language description.

[0022] 2.2 Deep learning methods are used to train the processed visual problem generation dataset to construct a problem generation neural network model. The problem generation model constructs implicit scene type representations and initially generates relational feature maps containing information about the target of interest. Then, with the target of interest as the center, it learns the correlation between the problem category and different granular regions of the image, and generates different problems related to the context of the target of interest at multiple scales.

[0023] Step 3: Joint Q&A

[0024] 3.1 The image segmentation module, question generation module, and joint question answering module are integrated. Using the guiding question as context, a top-down attention mechanism is employed for learning, and a contrastive learning loss function is introduced to perform joint multimodal embedding representation of the guiding question and relational feature maps. Based on the trained network, candidate answers and their confidence scores are provided, generating a natural language description of the image content.

[0025] The beneficial effects of this invention are:

[0026] 1. Image segmentation (using existing segmentation models) includes not only explicit pixel-level information of the target and background, but also hidden subordinate, parallel and logical relationships between objects, providing comprehensive and detailed image feature information.

[0027] 2. Compared to directly generating image descriptions, this design scheme, which generates multi-granularity feature questions from images in step two, can guide more refined and specific multi-scale answers. At the same time, it can also capture the possibility of implicit facts occurring in the image and provide the ability to describe simple events.

[0028] 3. This design provides a novel method for image description, which uses multi-granularity feature questions as the guiding core for generating image content descriptions. It uses a visual question-answering model to construct the correspondence between questions and answers (this application is the first to propose using a visual question-answering model to generate image descriptions). The introduction of a contrastive learning loss function improves the model's cross-modal learning ability and can provide natural language descriptions that are more in line with human semantic cognition of images, which is conducive to improving the processing efficiency of human-computer interaction. Attached Figure Description

[0029] Figure 1 Image segmentation model diagram

[0030] Figure 2 The problem generation model designed in this invention

[0031] Figure 3 Joint Question Answering Model Diagram

[0032] Figure 4 Example of image description for joint question answering Detailed Implementation

[0033] Example

[0034] Examples of image descriptions for combined question answering, such as Figure 4 As shown.

[0035] A segmentation-based image content description method based on visual question answering includes the following steps:

[0036] Step 1: Image Segmentation

[0037] 1.1 In this embodiment, the dataset used is the Cityscapes dataset, which focuses on visual understanding of complex urban street scenes. It contains a variety of stereo video sequences recorded from street scenes in 50 different cities, with 20,000 frames of weak annotations and 5,000 frames of high-quality pixel-level annotations, providing a total of 30 categories of annotations including pedestrians, vehicles, roads, buildings, traffic lights and traffic signs.

[0038] 1.2 In this embodiment, a deep learning method is used to train the image segmentation dataset, constructing a Deeplab image segmentation network model with an encoder-decoder structure. The Deeplab series is a fully convolutional dilated convolution semantic segmentation model. Its proposed dilated convolution can recover the feature map resolution that continuously decreases during convolution operations without increasing the number of parameters. The Deeplab v3+ model used in this embodiment is a segmentation network model with an encoder-decoder structure, which has high computational speed and prediction accuracy. The encoder module is used to extract high-level semantic information from the image. The decoder module is used to recover low-level spatial information, obtaining clear and complete category segmentation boundaries and the relationships between targets and between targets and the background. The network model is as follows: Figure 1 As shown.

[0039] In the encoder structure stage, the Xception network, which incorporates depthwise separable convolutions, is first used as the backbone network to extract initial features from the input image. The Xception network uses residual connections and depthwise separable convolutions to downsample the feature map. The subsequent Atrous Spatial Pyramid Pooling (ASPP) module is used to further extract multi-scale feature information. The module combines 1×1 convolutions, global average pooling, and dilated convolutions with dilation rates (6, 12, and 18) into a vertically parallel structure. Then, the multi-scale feature map is processed by 1×1 convolutions to compress the number of channels to 256, reducing the feature map resolution to 1 / 16 of the original image, which serves as the feature map output of the encoder structure.

[0040] In the decoder structure stage, the feature map output by the encoder structure is upsampled by 4 times bilinear interpolation, and then concatenated with the shallow feature map of the corresponding layer on the backbone network after channel adjustment. The feature map is then refined by two 3×3 convolutional layers, and finally upsampled by 4 times bilinear interpolation to obtain a segmentation prediction result with rich details and global information of the same size as the original image.

[0041] 1.3 Save the model weights of the trained Deeplab v3+ image segmentation neural network. This network model can process the original image, distinguish different targets and backgrounds in the image, and finally output segmentation feature maps between targets and between targets and backgrounds. It extracts the pixel-level features of the image in detail.

[0042] Step Two: Question Generation

[0043] 2.1 In this embodiment, publicly available visual question generation datasets (including the SQuAD dataset, etc.) are processed. The question categories in the dataset are classified, and different question categories view the target and the relationships between them from multiple perspectives. Multiple question categories of the same image not only focus on different targets, but also on image regions of the same target at different scales. At the same time, the answers and questions in the dataset are merged to generate a complete natural language description.

[0044] The problem categories include objects, attributes, relationships, counting, behaviors, and so on.

[0045] 2.2 In this embodiment, a deep learning method is used to train the processed visual problem generation dataset to construct a problem generation neural network model.

[0046] To generate vision-based questions directly from feature images, a residual-connected MLP (Multilayer Perceptron) is constructed. It takes a segmentation feature map containing category information as input and outputs a relation feature map containing information about the main target of interest, which is used to guide subsequent question generation.

[0047] A typical MLP consists of three layers: an input layer, a hidden layer, and an output layer. The different layers in an MLP neural network are fully connected. The hidden layers are trained to construct scene type representations of the images, and the calculation formula for each layer is as follows:

[0048] H = XW h +b h

[0049] O = HW o +b o

[0050] Y = σ(O)

[0051] Where X is the feature input, Y is the feature output, and W... h and W o These are the weights of the hidden layer and the output layer, respectively, b h and b o These represent the deviations between the hidden layer and the output layer, respectively, and σ represents the activation function, which is the sigmoid function used here.

[0052] The segmentation feature map generates a relation feature map through residual connections. The channels representing relation weights represent the salience of the target of interest. Each closed target region in the segmentation feature map has a corresponding attention coefficient, denoted by u, representing the relation feature layer. Subsequently, the question generation model, based on LSTM (Long Short-Term Memory Recurrent Neural Network), establishes the association between each target of interest and the surrounding closed range category in the relation feature map in a multi-granular and hierarchical manner. At different scales, it generates relevant guiding questions based on the predicted highest priority question category.

[0053] LSTM is used to learn the relationship between the target and the problem, allowing the model to target the relevant regions for training based on the problem. LSTM involves the transfer of short-term memory (h) and long-term memory (C), including input gates, output gates, and forget gates, as shown in the following formula (current technology):

[0054]

[0055]

[0056]

[0057]

[0058]

[0059]

[0060] c t =z t ⊙i t +c t-1 ⊙f t

[0061]

[0062]

[0063] y t =(c t )⊙o t

[0064] Where, x t It is the input at the current moment, y t-1 It is short-term memory from the previous moment, c t-1 It is the long-term memory of the previous moment, W f W i W z and W o These are the corresponding input weights, R f R i R z and R oIt is the corresponding recursive weight, p i This represents the peephole weight matrix, used to update the granularity, b f b i b z and b o This represents the corresponding computational bias, where σ represents the sigmoid activation function.

[0065] The loss function for the problem-generating neural network model is shown below:

[0066]

[0067] in, It is the problem vector generated by the model, q i is the actual question vector in the dataset, and u represents the predicted relation weight value, which is a positive number.

[0068] The network model for problem generation is as follows Figure 2 As shown, the relational feature map provides contextual information about the entire image. By constructing an implicit scene type representation, the focus of attention efficiently concentrates question generation on several targets, better defining the focus of attention in the image scene and providing a more abstract representation of the target focus in the image. Centered on the focus of attention, several questions of different scales are generated at different granularity levels according to the predicted highest priority question category. These questions cover important information in the core area of ​​the image, making the guided questions processed by the question generation neural network model more comprehensive.

[0069] Step 3: Joint Q&A

[0070] 3.1 In this embodiment, the image segmentation module of step one, the question generation module of step two, and the joint question answering module of step three are integrated. The guiding question is used as the context, and a top-down attention mechanism is used for learning to perform joint multimodal embedding representation of the guiding question and the relation feature map.

[0071] First, the problem feature vector q generated from the problem and the segmentation feature map v obtained from image segmentation are concatenated as context to guide the model's weights W for v and q. q and W v Conduct training;

[0072] f q (q)=W q q

[0073] f v (v)=W v v

[0074] For f q (q) and f v(v) Perform multimodal embedding, computed using the Halma product:

[0075]

[0076] p(y)=σ(h)

[0077] Among them, f q f represents the output of the problem stream. v The output of the visual flow is represented by p(y), which is the final output result. The feature vector q and segmentation feature map v that are closer in matching degree have higher scores. q and W v Let represent the weight matrix, and σ represent the linear activation function. The loss function for joint embedding is the contrastive loss:

[0078]

[0079] Among them, y T The output represents the correctly matched problem feature vector q and segmentation feature map v.

[0080] Network models for joint question answering, such as Figure 3 As shown, this method integrates image segmentation, question generation, and joint question answering modules. Based on the trained network, it can provide the highest confidence answer prediction. This method, based on visual question answering, can provide a refined description of image content.

Claims

1. A joint question-answering method for image natural language description, characterized in that, It includes three steps: Step 1: First, use an image segmentation model to extract features from the image target and background to obtain pixel-level classifications of different categories and acquire segmentation feature maps of the target and background. Step 2: The question generation module based on the LSTM model generates a relational feature map containing information about the target of interest by constructing an implicit scene type representation, and generates several semantically related guiding questions at multiple granularities. Step 3: The joint question answering module based on the BUTD model introduces a contrastive learning loss function to perform joint multimodal embedding representation on the relation feature map and the guiding question. Through training, the BUTD model generates long text answers related to the question based on the joint multimodal embedding representation, which serve as a refined semantic description of the image content. Step two, problem generation, specifically includes: 2.1 The dataset is classified into question categories. Different question categories view the target and the relationship between them from multiple perspectives. Multiple question categories of the same image not only focus on different targets, but also on image regions of the same target at different scales. At the same time, the answers and questions of the dataset are merged to generate a complete natural language description. 2.2 Use deep learning methods to train the visual problem generation dataset processed in step 2.1 to construct a problem generation neural network model; A multilayer perceptron (MLP) neural network with residual connections is constructed. It takes a segmentation feature map containing category information as input and outputs a relation feature map containing information about the main target of interest, which is used to guide the subsequent question generation. A multilayer perceptron neural network consists of three layers: an input layer, a hidden layer, and an output layer. The different layers in a multilayer perceptron neural network are fully connected. The hidden layers are trained to construct scene type representations of the images, and their weights are calculated using the following formulas: in, It is a feature input. It is the feature output. and These are the weights for the hidden layer and the output layer, respectively. and These are the deviations between the hidden layer and the output layer, respectively. This represents the sigmoid activation function; Subsequently, the question-generating neural network model, based on LSTM, establishes the association between each interest target and the surrounding closed range category in the relation feature map in a multi-granular and hierarchical manner, and generates relevant guiding questions at different scales based on the predicted highest priority question category.

2. The method described in claim 1, characterized in that: Step 1: Image Segmentation 1.1 We utilize publicly available image semantic segmentation datasets, where batch images are labeled with pixel-level categories; 1.2 Deep learning methods are used to train the image segmentation dataset and construct an image segmentation neural network model. The task of image segmentation is to make dense predictions for images by labeling different targets with specific colors so that each pixel has its own target or closed region category. 1.3 Save the model weights of the trained image segmentation neural network. This network model can process the original image, distinguish different targets and backgrounds in the image, and finally output segmentation feature maps between targets and between targets and backgrounds.

3. The method described in claim 1, characterized in that: For step two, the question generation module based on the LSTM model processes the segmentation feature map. By constructing an implicit scene type representation, it first generates a relational feature map containing information about the target of interest. Then, with the target of interest as the center, it establishes connections between the target of interest in the image and between the target of interest and the background at multiple scales. The generated multi-granularity guiding questions serve as a link in the subsequent joint question answering.

4. The method of description as described in claim 1, characterized in that: Step three: The algorithm integrates image segmentation, question generation, and joint question answering modules. Using the guiding question as context, it employs a top-down attention mechanism for learning and introduces a contrastive learning loss function to perform joint multimodal embedding representation of the guiding question and relational feature maps. Based on the trained network, it provides candidate answers and their confidence scores to generate a natural language description of the image content.

5. The description method as described in claim 1, characterized in that: The loss function for the problem-generating neural network model is shown below: in, It is the problem vector generated by the model. These are the actual problem vectors in the dataset. This represents the predicted relational weight value, which is a positive number.

6. The description method as described in claim 1, characterized in that: Loss function for contrastive learning: in, The output represents the problem feature vector q and the segmentation feature map v that are correctly matched.

Images