An image description generation method based on network structure search

Abstract

The application discloses an image description generation method based on network structure search, comprising the following steps: searching a convolution unit based on an image classification task by using a network structure optimization method to obtain a general convolution and a reduction pooling unit structure; constructing a model structure by using the searched unit structure, and training the model structure based on an image classification dataset; constructing a visual attention model by using the convolution unit model structure obtained by searching and training, and applying the visual attention model to an image description generation task; decoding and evaluating the visual attention model obtained by applying to the image description generation task based on beam search, and realizing image description generation. The application solves the problem of insufficient application of current network structure search technology in heavyweight tasks in various fields, successfully applies the convolution structure unit searched by the network structure optimization method based on a lightweight image classification task to a more complex image understanding task, effectively reduces the parameter quantity of the model, and improves the performance of the corresponding task.

Description

TECHNICAL FIELD

[0001] The present application relates to an image description generation method in the field of natural language processing, in particular to an image description generation method based on network structure search. BACKGROUND

[0002] Neural Architecture Search (NAS) is an important branch of Automated Machine Learning (AutoML), and its development has a unique significance especially in the period when artificial intelligence is moving towards high efficiency and intelligence. In the field of automated machine learning, researchers spend less time on tedious manual adjustment and instead use data-driven decision-making to automatically determine the best learning strategy.

[0003] In the traditional machine learning process, researchers have to complete a series of work such as feature extraction and model design for different tasks and different data before proceeding to the next step, i.e. model training and optimization. With the birth and development of deep learning research direction, machine learning practitioners gradually get rid of the work of feature design and extraction, which is more delivered to the model to process. However, for the work of model design, relevant researchers still need to spend a lot of valuable time to design suitable neural networks and constantly experiment to adjust hyperparameters and optimize details. At the same time, this manual model design method depends heavily on the researchers' understanding of the task and is limited by their knowledge and ideas. Even after a large number of experimental tests and verifications, the designed model cannot be guaranteed to be optimal, so many deep learning-based methods still have a strong dependence on experts in related fields.

[0004] Network structure search can eliminate the excessive dependence of model design work on manual work in the above process and effectively reduce the unacceptable manual trial and error and time cost. It adopts the research idea of automatic network structure search and can achieve the effect of accurate modeling for specific tasks without excessive human intervention.

[0005] Although the network structure search technology has achieved good results in various basic tasks, at present, in order to design and search a breakthrough structure different from the traditional artificial design structure, the search space of the network structure search task is often relatively large, and the current advanced method still pays great attention to the optimization of the topological structure, which greatly improves the flexibility of the search process and the diversity of the structure, but also leads to a large consumption of computing resources by the network structure search, so most of the work in the field of network structure search is focused on relatively lightweight tasks (such as image classification tasks in the field of image processing). The purpose of this is to use more lightweight tasks to verify the proposed method to reduce the requirement of network structure search for computing power. Based on the above series of reasons, the processing and application of network structure search for complex tasks, i.e. heavy tasks (such as image description generation tasks in the field of image processing) still need continuous exploration and attempt of researchers. SUMMARY

[0006] In view of the problem that the current network structure search technology is scarce in the application of complex tasks in various fields, the application provides an image description generation method based on network structure search, which can apply the structure units searched on basic tasks to complex tasks and improve the performance of the tasks.

[0007] To solve the above technical problems, the technical scheme adopted by the application is:

[0008] The application provides an image description generation method based on network structure search, comprising the following steps:

[0009] 1) using a network structure optimization method to search a convolution unit based on an image classification task, obtaining an ordinary convolution and a reduction pooling unit structure;

[0010] 2) using the searched unit structure to construct a model structure, and training the model structure based on an image classification data set;

[0011] 3) using the convolution unit model structure obtained by search training to construct a visual attention model, and applying it to an image description generation task;

[0012] 4) decoding and evaluating the visual attention model obtained by the above image description generation task based on beam search, to realize the generation of image description.

[0013] The specific process of step 1) is:

[0014] 101) Randomly generate multiple convolutional structures and add them to the structure pool, use the single-layer LSTM encoder of the search controller to encode each convolutional structure, map the discrete convolutional structure to the continuous space, and train each discrete convolutional structure and evaluate the model performance;

[0015] 102) Use the evaluated convolutional structure and model performance data to train the search controller;

[0016] 103) Select multiple high-performance convolutional structures, and use the performance predictor function f of the controller to derive new convolutional structure embedding representations e based on the encoder's hidden layer state x′ :

[0017]

[0018] where h t is the hidden layer state of the encoder, e x′ is the derived new structure embedding, h' t is the component of the new structure embedding in each dimension, η is the hyperparameter used for derivation, T is the size of the hidden layer of the encoder, h' T is the last component of the derived convolutional structure embedding representation;

[0019] 104) Decode the representation of the new convolutional structure and add it to the structure pool, which is equivalent to the inverse operation of the encoding structure process, restore the structure in the structure pool from the continuous representation to the character sequence; repeat the iteration process of evaluation, training and derivation until the search controller converges, select the optimal structure in the structure pool as the search result, and obtain the ordinary convolution and reduction pooling unit structure.

[0020] Step 3) The specific process is as follows:

[0021] 301) Pass the picture into the convolutional neural network trained by the search to process it, use the average pooling unit to encode pictures of different sizes into feature vectors of the same size, scale to a uniform size, and send it to the decoder for flattening operation;

[0022] 302) Use the attention mechanism to calculate the relative weight of the picture vector feature, get the context vector, and use the gating unit to weight it for better target object recognition, while performing word embedding on the picture description;

[0023] 303) Input the context vector, word embedding vector of the picture description, and hidden layer state of the encoder into the decoding unit for decoding, and train it through the cross-entropy loss of the decoding result and the true label.

[0024] Step 4) is to extend the flattening and previous processing in step 3) to K dimensions to realize a beam search method with a beam size of K, each prediction sequence is decoded from the beginning, and after decoding, it is removed from the decoding queue, and the diversity of the generated picture description is increased.

[0025] The present application has the following advantages and benefits:

[0026] 1. The present application solves the problem of insufficient application of current network structure search technology in heavy tasks in various fields, and successfully applies the network structure optimization method based on the convolution structure unit searched by the lightweight image classification task to more complex image understanding tasks.

[0027] 2. In addition, the method proposed in the present application can effectively reduce the parameter quantity of the model and improve the performance of the corresponding task, and automatically obtain a more optimal model structure adapted to the task through searching. BRIEF DESCRIPTION OF DRAWINGS

[0028] Figure 1 The flowchart of the image description generation method based on network structure search of the present application;

[0029] Figure 2 The flowchart of the network structure search algorithm searching for the convolution unit of the present application;

[0030] Figure 3 The convolution unit structure searched based on the image classification task of the present application. DETAILED DESCRIPTION

[0031] The present application will be further described below in conjunction with the drawings of the specification.

[0032] The present application provides an image description generation method based on network structure search, and the specific structure flow is as shown in Figure 1 The method comprises the following steps:

[0033] 1) using a network structure optimization method to search for a convolution unit based on an image classification task, obtaining a general convolution and a reduction pooling unit structure;

[0034] 2) training the searched convolution structure based on an image classification dataset;

[0035] 3) using the convolution unit model structure searched and trained to construct a visual attention model, and applying it to an image description generation task;

[0036] 4) decoding and evaluating the obtained image understanding generation model based on beam search.

[0037] In step 1), the network structure optimization method is used to search for a convolution unit based on an image classification task, and the overall flow is as shown in Figure 2 Specifically,

[0038] 101) Using various fine-grained operations such as convolution operation with kernel size 3x3, average pooling operation with kernel size 2x2, etc. to combine, randomly generate 1000 random convolutional structures to join the structure pool, use the single-layer LSTM encoder of the search controller to embed and encode each structure, convert the input structure into a character sequence, which can be regarded as a mapping from discrete structure space to continuous space E: X→ε, the embedding of the discrete structure is e x = E(x), at this time the hidden layer state h t of the LSTM can be used as the continuous space representation e x = {h1, h2, …, h T}, where T is the dimension size of the LSTM hidden layer, and train each structure based on the image classification task and evaluate the model performance.

[0039] 102) After model evaluation, sort the model and performance data pairs according to the accuracy of the model performance and write them to a file, use this data pair as input data, and train the search controller combined with the joint loss of the performance predictor and the decoder.

[0040] 103) According to the accuracy performance of the model structure, select 100 optimal structures from the structure pool, and based on these current optimal structures, use the performance predictor function f in the controller to derive new structure embedding representations based on the gradient of the encoder LSTM hidden layer state. The performance predictor is responsible for mapping the continuous embedding representation e x of each structure x to its performance on the validation set s x . It first performs average pooling on the embedding representation e x = {h1, h2, …, h T} to obtain Then use the feedforward neural network composed of fully connected layers to convert into a scalar value as the prediction of the structure performance, and the objective function of the performance predictor is the least squares regression loss between the true performance and the predicted performance of the structure. The embedding representation of the new structure is as follows:

[0041]

[0042] Where h t is the hidden layer state of the encoder, e x′ is the derived new structure embedding;

[0043] 104) Add the embedding of these derived new structures to the structure pool after decoding them into the original structure, and the decoder also uses e xAs input, and in a self-recursive manner for decoding. As the inverse operation of the encoder, the decoder can be seen as a mapping function D: ε→ X from discrete space to continuous space, decoding the character sequence from the continuous representation of the structure, i.e. x = D(e x ). The decoder is an LSTM model with attention mechanism for the initial hidden layer state s0= h T (x). The above evaluation, training and derivation iteration process is repeated for about 4 times, and when the encoder and the decoder are trained to convergence, the structure with the best performance is selected from the structure pool as the final searched structure.

[0044] In this step, the search space of the search structure is composed of meta-structures, and these meta-structures are stacked as directed acyclic graphs, with the nodes representing local calculations to be searched and the information flow represented by the edges of the graph. The independent parameters of the local calculation are activated only after being selected by the controller in the corresponding node. Based on the idea of weight sharing, all sub-graphs in the graph share parameters corresponding to the model structure, so as to speed up the search process. During the training of the controller, its joint loss function L can be composed of two parts, namely the performance prediction loss L pp of the performance predictor in the controller and the structure reconstruction loss L rec of the decoder, as follows:

[0045]

[0046] where λ∈[0, 1] is a weighting parameter, X is the overall structure set, x is a structure in the set, E(x) is the continuous embedding of the structure, P D (x|E(x)) is the probability of decoding from the continuous embedding to the discrete structure.

[0047] In step 2), the search results of step 1) are trained on the Cifar or Imagenet dataset of the image classification task, and the final searched convolutional unit structure for subsequent image understanding tasks is as shown in Figure 3 .

[0048] In step 3), the convolutional unit model structure searched and trained is used to construct a visual attention model, which is applied to the image description generation task, and the specific operation steps are as follows:

[0049] 301) The picture is input into the convolutional neural network searched and trained, and the average pooling unit is used to encode pictures of different sizes into feature vectors of the same size, facilitating subsequent processing. The picture after the above downsampling will be scaled to a uniform size, and the picture features of the uniform size will be received together with the standard picture description in the dataset by the decoder, and the decoder will flatten the picture features extracted by the encoder;

[0050] 302)The attention model implemented by using the multi-layer perception is used to calculate the relative weight of each position of the picture feature vector, and the context vector is calculated by combining the attention mechanism and the extracted picture vector representation, and the context vector is weighted by using the gating unit, so that better target object recognition is realized. Meanwhile, the picture description is embedded from the size of the word table dimension into a 512-dimensional vector space, and the multi-layer perception is used to initialize the initial hidden layer state and cell state of the LSTM model by using the encoder output as the input;

[0051] 303)The decoding stage is based on time step iteration, and the context vector combined with the attention mechanism, the word embedding vector of the picture description and the hidden layer state of the encoder are input into the decoding unit for decoding, and the training is realized by the cross entropy loss of the decoding result and the real label.

[0052] In step 4), the obtained image understanding generation model is decoded and evaluated based on beam search, and the flattening and previous processing in step 3) need to be expanded to K dimensions to realize the beam search method with a beam size of K, each predicted sequence is decoded from the beginning, and after decoding, it is removed from the decoding queue, and the diversity of the generated picture description is increased.

[0053] In order to verify the effectiveness of the method, the image description generation method based on network structure search proposed in the application is used in the image description generation task. Specifically, the experiment is carried out on the Microsoft Common Objects in Context (MSCOCO) dataset, which is commonly used in image recognition, segmentation and detection and other tasks. The baseline of the experiment uses an encoder-decoder model with a VGG19 structure encoder, and the convolutional structure unit searched by the method is used to replace the convolutional unit in VGG19 for testing. The experimental results are shown in the following table, wherein the model parameter unit is million, abbreviated as M, the smaller the better; BLEU represents the quality of the image understanding generation system on the test set, the higher the better.

[0054]

[0055] Experiments show that the image description generation method based on network structure search proposed in the application can effectively reduce the model parameters while significantly improving the quality of the image understanding generation system. After replacing the convolutional unit, the method can reduce the parameter amount of the encoder by 16.6%, and realize a performance improvement of 4.8 BLEU.

[0056] The present application is based on an image classification task, adopts a network structure optimization method to search for an adaptive convolution unit, and successfully applies it to a relatively complex image understanding generation task, thereby solving the problem of insufficient application of current network structure search technology to complex tasks in various fields. Meanwhile, the method proposed in the present application can effectively reduce the parameter quantity of the model and improve the performance of the corresponding task.

Claims

1. A method for generating an image description based on network structure search, characterized by The method comprises the following steps: 1) searching a convolution unit based on an image classification task using a network structure optimization method to obtain a general convolution and reduction pooling unit structure; 2) constructing a model structure using the searched unit structure and training the model structure based on an image classification dataset; 3) constructing a visual attention model using the convolution unit model structure obtained by searching and training, and applying the visual attention model to an image description generation task; 4) decoding and evaluating the visual attention model obtained by searching based on a beam search to implement image description generation; Step 1) has the following specific process: 101) randomly generating a plurality of convolution structures and adding the structures to a structure pool, using a single-layer LSTM encoder of a search controller to embed and encode each convolution structure, mapping the discrete convolution structure to a continuous space, and training each discrete convolution structure and evaluating the model performance; 102) using the convolution structure and model performance data after evaluation to train the search controller; 103) selecting a plurality of well-performing convolutional structures, and using a performance predictor function of the controller to derive a new convolutional structure embedding representation based on the encoder's hidden layer state : wherein is a hidden layer state of an encoder, is a derived new structure embedding, is a component of the new structure embedding on each dimension, is a hyperparameter for the derivation, T is a hidden layer dimension size of the encoder, is a last component of the derived convolutional structure embedding representation; 104) decoding the representation of the new convolution structure and adding the structure to the structure pool, restoring the structure in the structure pool from the continuous representation to a character sequence; repeating the iteration process of evaluation, training and derivation until the search controller converges, selecting the optimal structure in the structure pool as the search result, and obtaining the general convolution and reduction pooling unit structure.

2. The method for generating image description based on network structure search according to claim 1, characterized in that: Step 3) has the following specific process: 301) inputting a picture into the convolutional neural network obtained by searching and training to process the picture, using an average pooling unit to encode pictures of different sizes into feature vectors of the same size, scaling to a uniform size, and sending the feature vectors to a decoder for flattening operation; 302) using an attention mechanism to calculate the relative weight of the picture vector features to obtain a context vector, and using a gating unit to weight the context vector for better target object recognition, while performing word embedding on the picture description; 303) inputting the context vector, word embedding vector of the picture description and hidden layer state of the encoder into the decoding unit for decoding, and training the decoding result by cross-entropy loss with the true label.

3. The method for generating image description based on network structure search according to claim 1, characterized in that: Step 4) is to extend the flattening and previous processing in step 3) to K dimensions to implement a beam search method with a beam size of K, each predicted sequence is decoded from the beginning, and is removed from the decoding queue after decoding is completed, thereby increasing the diversity of generated picture descriptions.

Images