A parallel LSTM image description generation method that synchronizes semantic and visual information

Abstract

The invention discloses a parallel LSTM for synchronizing semantic and visual information, and an image description generation method based on the parallel LSTM. The method proposes a parallel LSTM for synchronizing semantic and visual information, extracts attribute features and visual features respectively through MIML and ResNet101, and uses the attribute features and visual features as the input of the parallel LSTM, and generates a description statement of the image on this basis. The parallel LSTM structure includes an attribute LSTM and a visual LSTM for synchronously performing data processing and information transmission. The initialization input of the attribute LSTM is the attribute feature, and the hidden layer state of each time step is obtained by iteration, which is regarded as a dynamic semantic feature. The visual LSTM dynamically fuses the semantic features output by the attribute LSTM and the visual features generated by itself at each time step through an attention mechanism, and provides richer information for the next time step. This synchronous information fusion method enables the method to synchronously pay attention to important semantic and visual information at each time step, thereby generating a more accurate image description statement.

Description

Technical Field

[0001] The present invention mainly relates to the field of image understanding, and relates to a parallel LSTM for synchronizing semantic and visual information, and an image description generation method based on the parallel LSTM. Background Art

[0002] Image description is one of the important research contents of computer vision, the basis of image understanding, and one of the research hotspots. The process is to generate a sentence for a given image, which can accurately describe the image content.

[0003] The image description method refers to machine translation. It is based on the encoder-decoder framework, uses a convolutional neural network as an encoder to extract features, and then uses a recurrent neural network to decode and generate a description statement of the image. Summary of the invention

[0004] The purpose of the present invention is to propose a parallel LSTM that synchronizes semantic and visual information, and an image description generation method based on the parallel LSTM. The image description generation method first uses a multi-instance multi-label MIML network to extract attribute features, then extracts visual features through a convolutional neural network ResNet101, and finally generates a description sentence of the image through the parallel LSTM proposed in this paper.

[0005] The technical solution of the present invention is as follows:

[0006] (1) Dynamic semantic feature extraction: After the image attribute features are extracted through MIML, the features are input into the attribute LSTM in the parallel LSTM, and the hidden layer state at each time step is used as the dynamic semantic feature;

[0007] (2) Visual feature extraction: The image is passed through the ResNet101 network to extract visual features;

[0008] (3) Fusion of semantic and visual information: The dynamic semantic features output by the attribute LSTM in the parallel LSTM structure at each time step are combined with the visual characteristics of the visual LSTM at the corresponding time step through the attention mechanism to obtain weighted contextual features;

[0009] (3) Synchronizing semantic and visual information: After the attribute features are input for the first time, the attribute LSTM itself contains semantic features. Each time a word is input, it combines its own information with the current word and changes its own information. Therefore, the hidden layer state of its output can be regarded as a dynamic semantic feature. In addition, the attribute LSTM in the parallel LSTM structure leads the visual LSTM by one time step in time step, so that the dynamic semantic features provided to the visual LSTM are exactly the semantic information required by the visual LSTM in the current time step, thereby achieving synchronization.

[0010] (4) Generate image description sentences: The context features obtained by fusing the synchronous semantic and visual information and the words generated in the previous time step are used as the input of the visual LSTM in the parallel LSTM structure, and each word in the sentence is obtained through iteration. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 Flowchart of the parallel LSTM image description generation method that synchronizes semantic and visual information

[0012] Figure 2 Framework diagram of the parallel LSTM image description generation method that synchronizes semantic and visual information

[0013] Figure 3 Parallel LSTM framework diagram DETAILED DESCRIPTION

[0014] The present invention will be further described in detail below in conjunction with the accompanying drawings.

[0015] The flowchart of the parallel LSTM image description generation method for synchronizing semantic and visual information of the present invention is shown in Figure 1. Figure 1 It consists of 4 units: Multi-Instance Multi-Label Network (MIML), Convolutional Neural Network (Resnet 101), Parallel LSTM, and Description Generation.

[0016] The framework diagram of the parallel LSTM image description generation method for synchronizing semantic and visual information of the present invention is shown in Figure 2, and the specific framework diagram of the parallel LSTM is shown in Figure 2. Figure 3 shown.

[0017] Unit 101 is a multi-instance multi-label network MIML, such as Figure 2 As shown in Figure 1, it is used to generate object and attribute information in an image as attribute features. Its input is the original image I, and its output is the attribute feature V att (I), whose dimension is 1024.

[0018] Unit 102 is a convolutional neural network based on ResNet101, such as Figure 2 As shown, it is used to generate visual features of an image. Its input is the original image I, and the output dimension is [14,14,2048] visual features. The basic network part of MIML of unit 101 and ResNet101 of unit 102 share the parameters of the first two bottleneck layers.

[0019] Unit 103 is a parallel LSTM, such as Figure 2 As shown in Figure 1, it mainly includes two parts: attribute LSTM and visual LSTM. The specific implementation details are as follows Figure 3As shown in Figure 1, the attribute LSTM is used to generate dynamic semantic features, and the visual LSTM is used to fuse semantic features and visual features and generate words. The initial input of the attribute LSTM is the attribute feature V att (I), each subsequent step input is the word embedding feature x generated in the previous step t , extract the hidden layer state h of the current time step t , the relevant calculation is:

[0020] h1=LSTM(V att (I),h0,m0)

[0021] h t =LSTM(x t ,h t-1 ,m t-1 )

[0022] Among them, h0 and m0 are the initial hidden layer state and cell state of LSTM. Since the attribute features are input for the first time and the word information related to the text will be input in the subsequent time steps, the currently generated hidden layer state contains dynamic semantic features. For visual LSTM, the attention mechanism is used to fuse dynamic semantic features and visual features to obtain context features. The dynamic semantic features lead the visual features by one time step and can provide the information required for the current time step to achieve synchronization. The calculation formula of context features is as follows:

[0023]

[0024] α t =softmax(z t )

[0025]

[0026] Where V is the visual feature, is a dynamic semantic feature, is the hidden layer state of the visual LSTM, w z , W v , is the training parameter, α h is the adjustment factor, c t is the context feature. Weighting factor α t The calculation of is determined by the visual features, dynamic semantic features and the hidden layer state of the visual LSTM. Therefore, after the weighted visual features and the adjusted dynamic semantic features are fused, the important information that should be paid attention to in the current time step can be focused on.

[0027] Unit 104 is a word generation module. At time t, after obtaining the hidden layer state of the visual LSTM, the generated word probability distribution can be expressed by the following formula:

[0028]

[0029]

[0030] Where f is a nonlinear mapping function and the output dimension is the dictionary size. t |y1,…,y t-1 ,I) is the probability distribution of the words generated at time t. Take the index of the maximum value and find its position in the dictionary. The corresponding word is the generated word.

Claims

1. A parallel LSTM image description generation method that synchronizes semantic and visual information, Its characteristics include the following steps: (1) Extract attribute features through multi-instance multi-label MIML network; (2) Extract visual features through the ResNet101 network; (3) The attribute features and visual features are synchronously input into the attribute LSTM and visual LSTM of the parallel LSMT; (4) The dynamic semantic features output by the attribute LSTM lead the visual features by one time step and can provide the semantic information required by the current visual LSTM time step. The visual LSTM uses the attention mechanism to fuse the dynamic semantic features and visual features to obtain the contextual features. (5) The context features and the words generated in the previous time step are input into the visual LSTM of the parallel LSMT to obtain the words generated in the current time step.

2. The method according to claim 1, characterized in that In steps (1) and (2), the basic network part of MIML and the ResNet101 network share the parameters of the first two bottleneck layers, thereby ensuring the same underlying features, making the image content represented by attributes and visual features basically consistent, and reducing network parameters.

Images