A method for generating image description sentences based on convolutional neural networks

Abstract

The present invention discloses a method for generating image description sentences based on a convolutional neural network, comprising: obtaining data in a preset MSCOCO data set, and preprocessing the obtained data; using a pre-trained convolutional neural network combined with an attention mechanism strategy to extract image information from the preprocessed data, and determining feature vectors of important images, merging the feature vectors of the important images through a fully connected layer, obtaining a fused feature vector and inputting it into a decoder; processing the fused feature vector through an adaptive attention mechanism and a bidirectional long short-term memory network model, and generating an image description sentence. The present invention realizes high-quality image description through an adaptive attention mechanism and a bidirectional long short-term memory network model, so that the forward and backward directions of the sequence can be calculated separately, improving the decoding ability of the bidirectional long short-term memory network model, and thus improving the overall performance of the bidirectional long short-term memory network model.

Description

Technical Field

[0001] The present invention relates to the field of image recognition processing, and in particular to a method for generating image description sentences based on a convolutional neural network. Background Art

[0002] With the rapid development of technology, smartphones are becoming more and more popular among the public, and selfies and casual photos are gradually becoming a mainstream way of socializing. Therefore, images are growing at an exponential rate. The image description task is to use images as input, and use mathematical models and calculations to make the computer output natural language description text corresponding to the image, so that the computer has the ability to "see pictures and speak". For each image, the computer not only needs to identify the objects and attributes of the objects input, but also identify the relationship between different objects and express them in the correct natural language. This is also the difficulty of this technology.

[0003] Among them, patent number CN110390363A discloses an image description method, which uses VGG convolutional neural network to extract global image features in the picture, uses Faster R-CNN network to extract local image features in the picture, fuses global image features and local image features through global-local feature fusion algorithm to obtain image fusion features, processes image fusion features through bidirectional long short-term memory network with attention mechanism, generates preliminary image description sentences, reduces the influence of useless information, enhances the embodiment of key information, and increases the accuracy of description sentences. However, in the process of generating image description, the traditional attention mechanism model has non-visual words such as pronouns and quantifiers, which causes the model to reduce the effectiveness of visual information. At the same time, for the Faster R-CNN network, the extracted feature maps are all single-layer and the resolution is relatively small, which makes it difficult for the model to translate high-quality image descriptions.

[0004] Currently, no effective solution has been proposed for the problems in the related technologies. Summary of the invention

[0005] In view of the problems in the related art, the present invention proposes a method for generating image description sentences based on convolutional neural networks to overcome the above-mentioned technical problems existing in the existing related art.

[0006] To this end, the specific technical solution adopted by the present invention is as follows:

[0007] A method for generating image description sentences based on a convolutional neural network, comprising:

[0008] S1. Obtain data from the preset MSCOCO dataset and preprocess the acquired data;

[0009] S2, using a pre-trained convolutional neural network combined with an attention mechanism strategy to extract image information from the pre-processed data and determine the feature vectors of important images;

[0010] S3, merging the feature vectors of the important images through a fully connected layer to obtain a fused feature vector and inputting it into a decoder;

[0011] S4. Process the fused feature vector through an adaptive attention mechanism and a bidirectional long short-term memory network model, and generate an image description sentence.

[0012] Furthermore, the preprocessing of the acquired data includes text data preprocessing and image data preprocessing;

[0013] Among them, text data preprocessing,processes the encoded version of the title;

[0014] Image data preprocessing, uniformly adjust the image size, and save the adjusted image.

[0015] Furthermore, the method of extracting image information from the preprocessed data using a pre-trained convolutional neural network combined with an attention mechanism strategy and determining feature vectors of important images includes the following steps:

[0016] S21, calculating the initial weight of each extracted image feature vector;

[0017] S22, sequentially selecting images with the largest initial weights from a preset threshold number;

[0018] S23. According to the spatial position information of the selected image, feature vectors of all images adjacent to the selected image are selected as feature vectors of the important image.

[0019] Furthermore, the calculation formula of the initial weight of each image feature vector is as follows:

[0020]

[0021] Among them, b t,i,j represents the weight of the feature vector of the image at position (i, j) when the decoder generates the tth word, and 1≤i≤8, 1≤j≤8;

[0022] exp(·) represents the exponential function;

[0023] e t,i,j Represents the intermediate state of the weight.

[0024] Furthermore, the number of thresholds in the images with the largest initial weights of the preset threshold numbers selected in sequence is set to 3.

[0025] Furthermore, the decoder in S3 is a bidirectional long short-term memory network model.

[0026] Furthermore, processing the fused feature vector through an adaptive attention mechanism and a bidirectional long short-term memory network model and generating an image description sentence includes the following steps:

[0027] S41. Train the bidirectional long short-term memory network model through the cross entropy loss function;

[0028] S42, calculating the hidden layer states of the forward and reverse long short-term memory networks at time m;

[0029] S43, using a summation method to combine the output of the bidirectional operation to obtain the hidden layer state at the current moment;

[0030] S44, the word probability is obtained by combining the fusion feature vector and the output of the bidirectional long short-term memory network through the adaptive attention mechanism;

[0031] S45. Select the words with the highest probability at time m in turn as the result at time m, and connect the above words in the order of generation as the final output of the network to complete the image description.

[0032] Furthermore, the calculation formula for training the bidirectional long short-term memory network model through the cross entropy loss function is as follows:

[0033]

[0034] Among them, E represents the loss value;

[0035] Ω represents the number of categories;

[0036] k represents each category;

[0037] W(k) represents a one-hot vector, which represents the feature vector of an attribute;

[0038] pc (k) Indicates that the predicted sample belongs to category c (k) probability.

[0039] Furthermore, the calculation formula for calculating the hidden layer state of the forward and reverse long short-term memory networks at time m is as follows:

[0040]

[0041]

[0042] Among them, x m Represents the input of the long short-term memory network at time m;

[0043] x n-(m-1) Represents the input of the reverse long short-term memory network at time m;

[0044] LSTM stands for bidirectional long short-term memory network model;

[0045] Respectively represent the hidden layer states of the positive and reverse long short-term memory networks at time m.

[0046] Furthermore, the calculation formula for obtaining the hidden layer state at the current moment by combining the output of the bidirectional operation with the summation method is as follows:

[0047]

[0048] in, The operation represents the addition of corresponding elements.

[0049] The beneficial effects of the present invention are:

[0050] 1. The present invention combines a convolutional neural network with an attention mechanism strategy to perform feature selection on images, and can extract the most important image features at each moment, thereby improving the accuracy of subsequent generation of image description sentences.

[0051] 2. The present invention realizes high-quality image description through an adaptive attention mechanism and a bidirectional long short-term memory network model. The bidirectional long short-term memory network model can be used to calculate the forward and backward directions of the sequence respectively, thereby improving the decoding capability of the bidirectional long short-term memory network model, thereby improving the overall performance of the bidirectional long short-term memory network model.

[0052] 3. The present invention trains the bidirectional long short-term memory network model through the traditional cross-entropy loss function, so that the bidirectional long short-term memory network model can reach a stable state, thereby providing a guarantee for the generation of image description sentences and enabling high-quality image description. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0054] Figure 1 It is a method flow chart of a method for generating image description sentences based on a convolutional neural network according to an embodiment of the present invention. DETAILED DESCRIPTION

[0055] To further illustrate each embodiment, the present invention provides drawings, which are part of the disclosure of the present invention and are mainly used to illustrate the embodiments and can be used in conjunction with the relevant descriptions in the specification to explain the operating principles of the embodiments. With reference to these contents, ordinary technicians in the field should be able to understand other possible implementations and advantages of the present invention. The components in the figures are not drawn to scale, and similar component symbols are generally used to represent similar components.

[0056] According to an embodiment of the present invention, a method for generating image description sentences based on a convolutional neural network is provided.

[0057] The present invention will now be further described with reference to the accompanying drawings and specific embodiments. Figure 1 As shown, the method for generating an image description sentence based on a convolutional neural network according to an embodiment of the present invention includes:

[0058] S1. Obtain data from the preset MSCOCO dataset and preprocess the acquired data;

[0059] S2, using a pre-trained convolutional neural network combined with an attention mechanism strategy to extract image information from the pre-processed data and determine the feature vectors of important images;

[0060] Specifically, the convolutional neural network includes a first convolutional layer, a second convolutional layer, a third convolutional layer, a fourth convolutional layer and a fully connected layer connected in sequence;

[0061] Among them, in the process of extracting image information from data, the ResNet-101 model is used to extract image features. This model belongs to a standard size specification of the residual network;

[0062] Due to the continuous development of deep learning technology and the improvement of computer performance, the number of layers of convolutional neural networks has continued to increase, and the problem of network training failure caused by gradient dissipation and network degradation has followed. The residual module in the residual network avoids this problem very well. The residual network is composed of several interconnected residual modules. The main idea is to make subtle adjustments inside the residual module, so that the main body of the network does not change too much, and the overall result can be updated to an ideal state.

[0063] S3, merging the feature vectors of the important images through a fully connected layer to obtain a fused feature vector and inputting it into a decoder;

[0064] S4. Process the fused feature vector through an adaptive attention mechanism and a bidirectional long short-term memory network model, and generate an image description sentence.

[0065] In one embodiment, the preprocessing of the acquired data includes text data preprocessing and image data preprocessing;

[0066] Among them, text data preprocessing,processes the encoded version of the title;

[0067] Image data preprocessing, uniformly adjusting the image size, and saving the adjusted image;

[0068] Specifically, the above image data preprocessing first divides the pixel values ​​of all images by 255 to scale them to [0, 1], and then standardizes the images by subtracting the mean and dividing by the standard deviation. The image sizes in the MSCOCO image description dataset are different. We input the images of the original size into the ResNet-101 network and use adaptive average pooling to uniformly set the size of the feature maps of all images to 14x14x2048.

[0069] Specifically, most of the current image description datasets are described in English, such as the common UIUCpascal Dataset, Flickr8k, Flickr30k, MSCOCO, etc.

[0070] Among them, the MSCOCO dataset is the largest and most popular open dataset launched by Microsoft in 2015. It contains 123,287 3-channel RGB images, each of which is marked by about 5 image descriptions by staff. These image descriptions explain the content of the image from different angles, dividing about 90% of the dataset into training sets, about 5% into validation sets, and about 5% into test sets. In addition, the MSCOCO dataset file records each entity and its boundary coordinates in detail, with values ​​accurate to two decimal places. Due to the complex image background, many targets and small size of this dataset, the training process on this dataset is more complicated, but due to its huge and accurate data, the MSCOCO dataset has also become the most commonly used dataset in the field of computer vision.

[0071] In one embodiment, the method of extracting image information from preprocessed data using a pretrained convolutional neural network combined with an attention mechanism strategy and determining feature vectors of important images includes the following steps:

[0072] S21, calculating the initial weight of each extracted image feature vector;

[0073] S22, sequentially selecting images with the largest initial weights from a preset threshold number;

[0074] S23. According to the spatial position information of the selected image, feature vectors of all images adjacent to the selected image are selected as feature vectors of the important image.

[0075] In one embodiment, the calculation formula of the initial weight of each image feature vector is as follows:

[0076]

[0077] Among them, b t,i,j represents the weight of the feature vector of the image at position (i, j) when the decoder generates the tth word, and 1≤i≤8, 1≤j≤8;

[0078] exp(·) represents the exponential function;

[0079] e t,i,j Represents the intermediate state of the weight.

[0080] In one embodiment, the number of thresholds in the images with the largest initial weights of the preset threshold number selected in sequence is set to 3.

[0081] In one embodiment, the decoder in S3 is a bidirectional long short-term memory network model.

[0082] In one embodiment, processing the fused feature vector through an adaptive attention mechanism and a bidirectional long short-term memory network model and generating an image description sentence includes the following steps:

[0083] S41. Train the bidirectional long short-term memory network model through the cross entropy loss function;

[0084] S42, calculating the hidden layer states of the forward and reverse long short-term memory networks at time m;

[0085] S43, using a summation method to combine the output of the bidirectional operation to obtain the hidden layer state at the current moment;

[0086] S44, the word probability is obtained by combining the fusion feature vector and the output of the bidirectional long short-term memory network through the adaptive attention mechanism;

[0087] S45. Select the words with the highest probability at time m in turn as the result at time m, and connect the above words in the order of generation as the final output of the network to complete the image description.

[0088] In one embodiment, the calculation formula for training the bidirectional long short-term memory network model using the cross entropy loss function is as follows:

[0089]

[0090] Among them, E represents the loss value;

[0091] Ω represents the number of categories;

[0092] k represents each category;

[0093] W(k) represents a one-hot vector, which represents the feature vector of an attribute;

[0094] pc (k) Indicates that the predicted sample belongs to category c (k) probability.

[0095] In one embodiment, the calculation formula for calculating the hidden layer state of the forward and reverse long short-term memory networks at time m is as follows:

[0096]

[0097]

[0098] Among them, x m Represents the input of the long short-term memory network at time m;

[0099] x n-(m-1) Represents the input of the reverse long short-term memory network at time m;

[0100] LSTM stands for bidirectional long short-term memory network model;

[0101] Respectively represent the hidden layer states of the positive and reverse long short-term memory networks at time m.

[0102] In one embodiment, the calculation formula for obtaining the hidden layer state at the current moment by combining the output of the bidirectional operation with the summation method is as follows:

[0103]

[0104] in, The operation represents the addition of corresponding elements.

[0105] In summary, with the help of the above-mentioned technical scheme of the present invention, the present invention combines a convolutional neural network with an attention mechanism strategy to perform feature selection on an image, and can extract the most important image features at each moment, thereby improving the accuracy of subsequent generation of image description sentences; the present invention realizes high-quality image description through an adaptive attention mechanism and a bidirectional long short-term memory network model, and can use the bidirectional long short-term memory network model to calculate the forward and backward directions of the sequence respectively, thereby improving the decoding ability of the bidirectional long short-term memory network model, and thus improving the overall performance of the bidirectional long short-term memory network model; the present invention trains the bidirectional long short-term memory network model through a traditional cross-entropy loss function, thereby enabling the bidirectional long short-term memory network model to reach a stable state, thereby providing a guarantee for the generation of image description sentences, and enabling high-quality image description.

[0106] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for generating image description sentences based on convolutional neural networks, characterized in that: include: S1. Obtain data from the preset MSCOCO dataset and preprocess the acquired data; S2, using a pre-trained convolutional neural network combined with an attention mechanism strategy to extract image information from the pre-processed data and determine the feature vectors of important images; S3, merging the feature vectors of the important images through a fully connected layer to obtain a fused feature vector and inputting it into a decoder; S4, processing the fused feature vector through an adaptive attention mechanism and a bidirectional long short-term memory network model, and generating an image description sentence; Processing the fused feature vector through an adaptive attention mechanism and a bidirectional long short-term memory network model and generating an image description sentence includes the following steps: S41. Train the bidirectional long short-term memory network model through the cross entropy loss function; S42, calculating the hidden layer states of the forward and reverse long short-term memory networks at time m; S43, using a summation method to combine the output of the bidirectional operation to obtain the hidden layer state at the current moment; S44, the word probability is obtained by combining the fusion feature vector and the output of the bidirectional long short-term memory network through the adaptive attention mechanism; S45, sequentially selecting the words with the highest probability at time m as the result at time m, and connecting the above words in the order of generation as the final output of the network to complete the image description; The calculation formula for training the bidirectional long short-term memory network model through the cross entropy loss function is as follows: Among them, E represents the loss value; Ω represents the number of categories; k represents each category; W(k) represents a one-hot vector, which represents the feature vector of an attribute; pc (k) Indicates that the predicted sample belongs to category c (k) probability; The calculation formula for calculating the hidden layer state of the forward and reverse long short-term memory networks at time m is as follows: Among them, xm represents the input of the long short-term memory network at time m; xn-(m-1) represents the input of the reverse long short-term memory network at time m; LSTM stands for bidirectional long short-term memory network model; Respectively represent the hidden layer states of the positive and reverse long short-term memory networks at time m.

2. The method for generating image description sentences based on convolutional neural network according to claim 1, characterized in that: The preprocessing of the acquired data includes text data preprocessing and image data preprocessing; Among them, text data preprocessing,processes the encoded version of the title; Image data preprocessing, uniformly adjust the image size, and save the adjusted image.

3. The method for generating image description sentences based on convolutional neural network according to claim 1, characterized in that: The method of extracting image information from preprocessed data using a pretrained convolutional neural network combined with an attention mechanism strategy and determining feature vectors of important images includes the following steps: S21, calculating the initial weight of each extracted image feature vector; S22, sequentially selecting images with the largest initial weights from a preset threshold number; S23. According to the spatial position information of the selected image, feature vectors of all images adjacent to the selected image are selected as feature vectors of the important image.

4. The method for generating image description sentences based on convolutional neural network according to claim 3, characterized in that: The calculation formula of the initial weight of each image feature vector is as follows: Where bt,i,j represents the weight of the feature vector of the image at position (i, j) when the decoder generates the tth word, and 1≤i≤8, 1≤j≤8; exp(·) represents the exponential function; e t,i,j Represents the intermediate state of the weight.

5. The method for generating image description sentences based on convolutional neural network according to claim 3, characterized in that: The number of thresholds selected in sequence from the image with the largest initial weight of the preset threshold number is set to 3.

6. The method for generating image description sentences based on convolutional neural network according to claim 1, characterized in that: The decoder in S3 is a bidirectional long short-term memory network model.

7. The method for generating image description sentences based on convolutional neural network according to claim 6, characterized in that: The calculation formula for obtaining the hidden layer state at the current moment by combining the output of the bidirectional operation with the summation method is as follows: in, The operation represents the addition of corresponding elements.

Images