A method and system for generating image description based on curriculum learning

Abstract

The present invention proposes a method and system for generating image description based on course learning, including: obtaining an image training set with annotated description information, constructing an initial image description model as the current image description model; inputting the features of all images in the image training set into the current image description model in sequence to obtain the description of each image, and obtaining the difficulty of each image in the image training set according to the image description and its corresponding description information; judging whether the performance index of the current image description model reaches a preset value, if so, using the entire image training set to iteratively train the current image description model, otherwise executing the step of selecting an input ratio according to the performance index, and selecting the features of the image with the lowest difficulty in the image training set to train the current image description model with the input ratio. The image description generator trained by the present invention has better generalization performance.

Description

Technical Field

[0001] The present invention relates to the technical field of image description in image recognition analysis, and in particular to a method and system for generating image description based on curriculum learning. Background Art

[0002] In recent years, the fields of natural language processing and computer vision have continued to develop, greatly promoting the development and application scope of related technologies in the field of artificial intelligence. In order to adapt to complex application scenarios, the image description generation task generates a natural language sentence for a given image to describe the image content. The image description method based on deep learning is the current mainstream solution. This method is based on a large-scale image description dataset, builds an image description generation model and optimizes its implicit parameters. The model first uses an encoder (such as a convolutional neural network) to extract image features, and then uses a decoder and attention mechanism to generate text. The existing model training framework is mainly divided into two stages: cross-entropy-based training and further optimization of model parameters using reinforcement learning.

[0003] Although some progress has been made using deep learning methods, there are still some problems with existing methods in model training and data utilization. The main reasons are as follows:

[0004] On the one hand, this is because the model requires multiple network modules and a large number of learnable parameters to master cross-modal knowledge. The deeper the model architecture becomes, the more difficult it is to optimize.

[0005] On the other hand, data is highly heterogeneous in terms of quality, difficulty, and noise. Noise may lead to performance degradation, which means that although all samples participate in model training, different samples contribute differently to the model. Summary of the invention

[0006] The purpose of the present invention is to solve the above-mentioned problems of difficulty in optimizing the prior art model and insufficient utilization of training data, and proposes a method and system for generating image descriptions based on curriculum learning. The generator trained by this method has better generalization performance and can generate image descriptions more effectively.

[0007] In view of the shortcomings of the prior art, the present invention proposes an image description generation method based on curriculum learning, which includes:

[0008] Step 1: Obtain an image training set with labeled description information and construct an initial image description model as the current image description model;

[0009] Step 2: Input the features of all images in the image training set into the current image description model in sequence to obtain the description of each image, and obtain the difficulty of each image in the image training set according to the image description and its corresponding description information; determine whether the performance index of the current image description model reaches the preset value, if so, execute step 4, otherwise execute step 3;

[0010] Step 3, selecting an input ratio according to the performance indicator, selecting the features of the image with the lowest difficulty in the image training set with the input ratio to train the current image description model once, and executing step 2 again after completing the training once;

[0011] Step 4: Use the entire image training set to iteratively train the current image description model until a preset number of training iterations is reached or the loss of the current image description model converges, and save the current image description model as the final image description model to generate an image description result for the image to be described.

[0012] The curriculum-based image description generation method, wherein the difficulty is a cross-entropy loss or a reinforcement learning reward.

[0013] In the image description generation method based on curriculum learning, the difficulty of each image is obtained in step 2 by the following formula:

[0014] In the tth iteration, the difficulty d of the current image sample s is:

[0015]

[0016] Where s represents the current image sample, θ represents the model parameters, |D train | represents the number of all image samples in the image training set, e(s; θ t-1 ) represents the cross entropy loss or reinforcement learning reward value of the previous iteration of the image description model, e(s; θ t ) represents the cross entropy loss or reinforcement learning reward value of the current iteration of the model, and ε is the regularization term.

[0017] In the image description generation method based on curriculum learning, the performance index c of the current image description model is obtained by the following formula:

[0018]

[0019]

[0020] In iteration round t, c(t)∈(0,1], p is the parameter that controls the course learning progress, c0 represents the initial capability value of the model; C t is the CIDEr value of the current image description model, β is a parameter that controls the learning speed of the course; CP and C T are the best CIDEr values ​​on the dataset in the cross entropy training and reinforcement learning stages, respectively.

[0021] The present invention also proposes an image description generation system based on curriculum learning, which includes:

[0022] The initial module is used to obtain an image training set with labeled description information and build an initial image description model as the current image description model;

[0023] The difficulty calculation module is used to input the features of all images in the image training set into the current image description model in sequence to obtain the description of each image, and obtain the difficulty of each image in the image training set according to the image description and its corresponding expression information; determine whether the performance index of the current image description model reaches a preset value, if so, call the second training module, otherwise call the first training module;

[0024] The first training module is used to select an input ratio according to the performance indicator, select the features of the image with the lowest difficulty in the image training set according to the input ratio to train the current image description model once, and then execute the difficulty measurement module after completing the training once;

[0025] The second training module uses the entire image training set to iteratively train the current image description model until a preset number of training iterations is reached or the loss of the current image description model converges, and saves the current image description model as the final image description model to generate an image description result for the image to be described.

[0026] The curriculum learning-based image description generation system, wherein the difficulty is a cross entropy loss or a reinforcement learning reward.

[0027] In the curriculum learning-based image description generation system, the difficulty calculation module is used to obtain the difficulty of each image through the following formula:

[0028] In the tth iteration, the difficulty d of the current image sample s is:

[0029]

[0030] Where s represents the current image sample, θ represents the model parameters, |D train | represents the number of all image samples in the image training set, e(s; θ t-1 ) represents the cross entropy loss or reinforcement learning reward value of the previous iteration of the image description model, e(s; θ t ) represents the cross entropy loss or reinforcement learning reward value of the current iteration of the model, and ε is the regularization term.

[0031] In the curriculum learning-based image description generation system, the performance index c of the current image description model is obtained by the following formula:

[0032]

[0033]

[0034] In iteration round t, c(t)∈(0,1], p is the parameter that controls the course learning progress, c0 represents the initial capability value of the model; C t is the CIDEr value of the current image description model, β is a parameter that controls the learning speed of the course; C P and C T are the best CIDEr values ​​on the dataset in the cross entropy training and reinforcement learning stages, respectively.

[0035] The present invention also proposes a storage medium for a program of any one of the image description generation methods based on curriculum learning.

[0036] The present invention also proposes a client for use in any one of the image description generation systems based on curriculum learning.

[0037] It can be seen from the above scheme that the advantages of the present invention are:

[0038] like Figure 3 As shown in FIG. 1 , the present invention conducts experiments on the MS COCO dataset, which is the most commonly used dataset in image description generation tasks, containing 123,000 images, each with 5 descriptions. The results are evaluated using the indicators BLEU, METEOR, ROUGE, CIDEr, and SPICE.

[0039] The results on the validation set show that based on the same model, our method outperforms the previous training method in all six indicators, and the results on the test set further demonstrate its effectiveness. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 is a block diagram of the method of the present invention;

[0041] Figure 2 A framework diagram for the image description generation method based on curriculum learning;

[0042] Figure 3 It is a comparison diagram of the technical effects of the present invention and the prior art. DETAILED DESCRIPTION

[0043] The image description generation model has a large number of parameters that make optimization difficult and data noise causes performance degradation. The present invention adopts a curriculum learning method to guide the training from simple to complex, making the training smoother, while reducing the interference of noise data, making the training more generalized, and thus improving the performance of the image description generation task.

[0044] Specifically, in view of the complexity of the image description task model and the heterogeneity of the data, the present invention proposes to evaluate the sample difficulty at each stage of training. The main idea is to consider the decrease in loss or reward value and look for samples with a large learning space. At the same time, the model capability of the present invention is continuously enhanced, and the gap between the current model and the trained model is measured to decide how many samples to learn. When the model capability reaches a certain level, all samples are learned.

[0045] In order to make the above features and effects of the present invention more clearly understood, embodiments are given below and described in detail with reference to the accompanying drawings.

[0046] The method of the present invention is specifically as follows Figure 1 As shown, the method includes:

[0047] Process the image description dataset and extract the feature data in the image. Image feature extraction takes a lot of time. If the features are not extracted, each iteration needs to be extracted repeatedly, which reduces the training speed. After extracting the features, you only need to save the extracted features, and you can use them directly in future training, reducing repeated calculations.

[0048] Build an image description generation network model;

[0049] At the beginning of the training phase, each iteration determines whether the maximum round has been reached. If it is greater than or equal to the maximum round, the model state is saved and the training ends; otherwise, the training continues;

[0050] When the number of iterations is less than the maximum, the difficulty of the samples is evaluated first. When training based on cross entropy, the cross entropy loss is used to calculate the difficulty of each sample and sort it (from simple to complex). When training using reinforcement learning, the reward is used as the evaluation criterion.

[0051] The indicators of the current model are compared with the indicators of the model that has completed all the training before or the preset model indicators to measure the ability level of the current model. If the threshold is reached, the model ability is considered to be up to standard, and all samples will be put into training; that is, all samples need to calculate the difficulty of learning, and gradually increase the difficulty of samples until all samples participate in training.

[0052] Determine the proportion of training samples in the current iteration to the total samples based on the current model capability, and select samples that meet the requirements for training;

[0053] Continue to the next iteration.

[0054] Figure 2 A framework diagram of the image description generation method based on curriculum learning is given: the core technical points of the entire method lie in sample difficulty evaluation and model capability measurement.

[0055] Sample difficulty evaluation. This part measures the difficulty of each image-description pair before each model training starts, and quantifies the difficulty value of the current sample. The present invention believes that the sample difficulty depends on the current model parameters. After the image description generation model parameters are updated, the sample difficulty will be recalculated to ensure that the system always selects training data with appropriate difficulty for training. The two training stages of the model use cross-entropy loss and reward to optimize parameters respectively. The present invention defines the difficulty of the sample according to the decrease in cross-entropy loss or reward value to consider the model changes between two adjacent iterations. The present invention uses t (t≥1) as the iteration round, s represents the current sample, and θ represents the model parameters. In the tth round of iteration, the difficulty of the current sample s is:

[0056]

[0057] Among them, |D train | represents the number of all samples. The first iteration (t=1) assumes that all samples have the same difficulty. When t>1, the present invention calculates the decrease in loss or reward value, e(s; θ t-1 ) represents the cross entropy loss or reward value of the previous iteration of the model, e(s; θ t ) represents the cross entropy loss or reward value of the current iteration of the model, and ε is a regularization term to prevent overflow due to the denominator being too small as training progresses. The general value is 10 -5 .

[0058] Model capability measurement. This paper uses CIDEr (Consensus-based Image Description Evaluation) as an indicator to measure model performance. Each training selects |D train |·c(t) the simplest samples for the current model, and simple samples are continuously added in each iteration until the model is trained with all samples. In addition, the order of all samples is shuffled in each iteration to maintain local randomness. At iteration round t, the present invention defines the model capacity c(t)∈(0,1], p is a preset parameter for controlling the course learning progress, c0 represents the initial capacity of the model, and takes 0.01.

[0059]

[0060] The above formula ensures that as the training progresses, more and more samples will participate in the training. At the same time, considering the current state of the model, the present invention introduces M(t) as the capability of the current state of the model. P and C T The best CIDEr values ​​on the two training phase data sets are recorded as the evaluation benchmark. This best CIDEr value is the CIDEr value obtained through the practice of the model in the prior art, that is, the present invention first uses the prior art mature model to train the set and records the CIDEr value. P and C T , for subsequent use.

[0061]

[0062] C t is the current CIDEr indicator of the model, and β is the parameter that controls the learning speed of the course. t ≥C P β or C t -C P ≥(C T -C P )·β, it means that the model is ready to be trained on the entire dataset, and the model is trained on the entire training set until convergence.

[0063] Sample difficulty evaluation and model capability measurement function design ensure that the current training sample is selected to prevent the model from falling into a bad local optimum. In the assumption of the present invention, the ability to generate descriptions is weak when the model is initialized, so the difficulty of the sample and the ability of the model at each iteration are estimated to determine the training sample. The model continues to learn until it has enough ability to process the entire training set. The curriculum learning-based method proposed in the present invention can be easily integrated into the training strategy of the mainstream model.

[0064] The following is a system embodiment corresponding to the above method embodiment. This embodiment can be implemented in conjunction with the above embodiment. The relevant technical details mentioned in the above embodiment are still valid in this embodiment. In order to reduce repetition, they are not repeated here. Accordingly, the relevant technical details mentioned in this embodiment can also be applied in the above embodiment.

[0065] The present invention also proposes an image description generation system based on curriculum learning, which includes:

[0066] The initial module is used to obtain an image training set with labeled description information and build an initial image description model as the current image description model;

[0067] The difficulty calculation module is used to input the features of all images in the image training set into the current image description model in sequence to obtain the description of each image, and obtain the difficulty of each image in the image training set according to the image description and its corresponding expression information; determine whether the performance index of the current image description model reaches a preset value, if so, call the second training module, otherwise call the first training module;

[0068] The first training module is used to select an input ratio according to the performance indicator, select the features of the image with the lowest difficulty in the image training set according to the input ratio to train the current image description model once, and then execute the difficulty measurement module after completing the training once;

[0069] The second training module uses the entire image training set to iteratively train the current image description model until a preset number of training iterations is reached or the loss of the current image description model converges, and saves the current image description model as the final image description model to generate an image description result for the image to be described.

[0070] The curriculum learning-based image description generation system, wherein the difficulty is a cross entropy loss or a reinforcement learning reward.

[0071] In the curriculum learning-based image description generation system, the difficulty calculation module is used to obtain the difficulty of each image through the following formula:

[0072] In the tth iteration, the difficulty d of the current image sample s is:

[0073]

[0074] Where s represents the current image sample, θ represents the model parameters, |D train | represents the number of all image samples in the image training set, e(s; θ t-1 ) represents the cross entropy loss or reinforcement learning reward value of the previous iteration of the image description model, e(s; θ t ) represents the cross entropy loss or reinforcement learning reward value of the current iteration of the model, and ε is the regularization term.

[0075] In the curriculum learning-based image description generation system, the performance index c of the current image description model is obtained by the following formula:

[0076]

[0077]

[0078] In iteration round t, c(t)∈(0,1], p is the parameter that controls the course learning progress, c0 represents the initial capability value of the model; C tis the CIDEr value of the current image description model, β is a parameter that controls the learning speed of the course; C P and C T are the best CIDEr values ​​on the dataset in the cross entropy training and reinforcement learning stages, respectively.

[0079] The present invention also proposes a storage medium for a program of any one of the image description generation methods based on curriculum learning.

[0080] The present invention also proposes a client for use in any one of the image description generation systems based on curriculum learning.

Claims

1. A method for generating image description based on curriculum learning, characterized in that: include: Step 1: Obtain an image training set with labeled description information and construct an initial image description model as the current image description model; Step 2: Input the features of all images in the image training set into the current image description model in sequence to obtain the description of each image, and obtain the difficulty of each image in the image training set according to the image description and its corresponding description information; determine whether the performance index of the current image description model reaches the preset value, if so, execute step 4, otherwise execute step 3; Step 3, selecting an input ratio according to the performance indicator, selecting the features of the image with the lowest difficulty in the image training set with the input ratio to train the current image description model once, and executing step 2 again after completing the training once; Step 4: iteratively train the current image description model using the entire image training set until a preset number of training iterations is reached or the loss of the current image description model converges, and save the current image description model as the final image description model to generate an image description result for the image to be described; Among them, the difficulty is the cross entropy loss or reinforcement learning reward; In step 2, the difficulty of each image is obtained by the following formula: In the tth iteration, the difficulty d of the current image sample s is: Where s represents the current image sample, θ represents the model parameters, |D train | represents the number of all image samples in the image training set, e(s; θ t-1 ) represents the cross entropy loss or reinforcement learning reward value of the previous iteration of the image description model, e(s; θ t ) represents the cross entropy loss or reinforcement learning reward value of the current iteration of the model, and ε is the regularization term; The performance index c of the current image description model is obtained by the following formula: In iteration round t, c(t)∈(0,1], p is the parameter that controls the course learning progress, c0 represents the initial capability value of the model; C t is the CIDEr value of the current image description model, β is a parameter that controls the learning speed of the course; C P and C T are the best CIDEr values ​​on the dataset in the cross entropy training and reinforcement learning stages, respectively.

2. A curriculum learning-based image description generation system, characterized in that: include: The initial module is used to obtain an image training set with labeled description information and build an initial image description model as the current image description model; The difficulty calculation module is used to input the features of all images in the image training set into the current image description model in sequence to obtain the description of each image, and obtain the difficulty of each image in the image training set according to the image description and its corresponding expression information; determine whether the performance index of the current image description model reaches a preset value, if so, call the second training module, otherwise call the first training module; The first training module is used to select an input ratio according to the performance indicator, select the features of the image with the lowest difficulty in the image training set according to the input ratio to train the current image description model once, and then execute the difficulty measurement module after completing the training once; The second training module uses the entire image training set to iteratively train the current image description model until a preset number of training iterations is reached or the loss of the current image description model converges, and saves the current image description model as the final image description model to generate an image description result for the image to be described; Where the difficulty is the cross entropy loss or reinforcement learning reward; The difficulty calculation module is used to obtain the difficulty of each image through the following formula: In the tth iteration, the difficulty d of the current image sample s is: Where s represents the current image sample, θ represents the model parameters, |D train | represents the number of all image samples in the image training set, e(s; θ t-1 ) represents the cross entropy loss or reinforcement learning reward value of the previous iteration of the image description model, e(s; θ t ) represents the cross entropy loss or reinforcement learning reward value of the current iteration of the model, and ε is the regularization term; The performance index c of the current image description model is obtained by the following formula: In iteration round t, c(t)∈(0,1], p is the parameter that controls the course learning progress, c0 represents the initial capability value of the model; C t is the CIDEr value of the current image description model, β is a parameter that controls the learning speed of the course; C P and C T are the best CIDEr values ​​on the dataset in the cross entropy training and reinforcement learning stages, respectively.

3. A storage medium for storing a program for executing the image description generation method based on curriculum learning as claimed in claim 1.

4. A client, used in the image description generation system based on course learning as described in claim 2.

Images