An image caption generation model based on dynamic early exit and a model training method

Abstract

The present invention provides an image caption generation model based on dynamic early exit, comprising: a visual encoder for extracting visual features of an image, which includes a plurality of encoding layers connected in series in sequence; a text decoder for decoding the visual features output by the visual encoder and sequentially outputting words in a plurality of prediction vocabulary lists to form a natural language description text of the image, which includes a plurality of decoding layers connected in series in sequence, and each decoding layer is configured with a dynamic early exit decision module and an imitation learning network; wherein: each of the dynamic early exit decision modules is used to determine whether to exit early during the word prediction process according to the word prediction probability in the prediction vocabulary list, and output the word corresponding to the maximum probability when the maximum probability in the word prediction probabilities in the prediction vocabulary list exceeds the confidence threshold; each of the imitation learning networks is used to imitate and predict the output hidden layer state vector of its corresponding decoding layer according to the input.

Description

Technical Field

[0001] The present invention relates to the field of multimedia data processing. Specifically, it relates to the technology of generating image descriptions in the multimedia field. More specifically, it relates to an image description generation model based on early exit and a model training method. Background Art

[0002] The technology of generating image descriptions is used to generate a text description for a given picture. The text description contains multiple words. The image description generation model identifies the image features and outputs words in sequence to form the text description. This not only requires the image description generation model to be able to identify the objects contained in the picture, but also to describe the relationships between the objects in natural language. This technology can realize the generation of natural language descriptions for natural scene pictures in life, and convey the visual information of the image in an intuitive text expression method, skillfully connecting computer vision and natural language processing technologies.

[0003] Image captioning technology has a wide range of applications. For example, on e-commerce websites, it can be used to automatically generate captions for product images to meet user search needs. In early childhood education, it can be used to automatically generate captions for images, leveraging speech synthesis technology to implement autonomous "picture-speaking" teaching systems, serving both early childhood education and early childhood development. In the medical field, it can be used to recognize case images and automatically generate medical diagnostic reports, saving radiologists time in writing reports. In the field of visual aids, image captioning technology can be used to convert visual images into text, and speech synthesis technology can be used to broaden perception for the visually impaired. However, for image captioning models to be widely used in production, they must address the issue of slow caption generation speed. Existing image captioning models often use an autoregressive generation strategy, where each word in the text is sequentially passed through all decoding layers from left to right before being output, resulting in slow generation speed. To accelerate model decoding, a non-autoregressive generation strategy was proposed, where all words in a sentence are output in parallel. While this approach improves generation speed, since each word is independent when predicting the output, dependencies are not established, resulting in poor coherence and accuracy in the generated text. Later, an iteratively optimized non-autoregressive generation strategy was proposed. This strategy re-inputs the entire sentence generated by the previous non-autoregressive generation and iterates the output text multiple times. While this approach effectively improves the quality of the non-autoregressive generated text, it also suffers from computational redundancy. Alternatively, a two-stage semi-autoregressive generation strategy has been proposed to address the trade-off between generation quality and speed. However, to meet varying generation speeds, it cannot be adjusted in real time and requires retraining the model through hyperparameter adjustments, which undoubtedly increases training costs. Therefore, to accelerate text generation, accelerating the text generation process in image caption generation tasks and adjusting the speedup ratio in real time are important and pressing issues to be addressed. Summary of the Invention

[0004] Therefore, the purpose of the present invention is to overcome the defects of the above-mentioned prior art and provide an image description generation model based on dynamic early retirement and a model training method to achieve accelerated text generation.

[0005] According to a first aspect of the present invention, there is provided an image description generation model based on dynamic early exit, which is used to output a natural language description text of an image according to an input image. The image description generation model includes: a visual encoder for extracting visual features of the image, which includes a plurality of encoding layers connected in series; a text decoder for decoding the visual features output by the visual encoder and sequentially outputting words in a plurality of prediction vocabularies to form a natural language description text of the image, which includes a plurality of decoding layers connected in series, and each decoding layer is configured with a dynamic early exit decision module and an imitation learning network; wherein: each of the dynamic early exit decision modules is used to judge whether the current prediction needs to exit early according to the prediction probability of the words in the prediction vocabulary during the word prediction process, and output the word corresponding to the maximum probability when the maximum probability in the prediction probabilities of the words in the prediction vocabulary exceeds the confidence threshold. The input end of each dynamic early exit decision module is connected to the outputs of its corresponding decoding layer and all the decoding layers before it, and the outputs of the imitation learning networks corresponding to all the decoding layers after its corresponding decoding layer; the input end of each imitation learning network is connected to the outputs of its corresponding decoding layer and all the decoding layers before it, and is used to imitate and predict the output hidden layer state vector of its corresponding decoding layer according to the input.

[0006] Preferably, each of the encoding layers includes a self-attention layer and a feed-forward neural network connected in sequence. Each of the decoding layers includes a self-attention layer, an encoder-decoder self-attention layer, and a feed-forward neural network connected in sequence. Each of the dynamic early exit decision modules includes a shallow feature fusion layer, a deep feature fusion layer, a fusion gating layer, and a classification layer, wherein: the shallow feature fusion layer is used to perform feature fusion on the hidden layer state vectors of its corresponding decoding layer in the dynamic early exit decision module and all the decoding layers before this decoding layer to obtain a shallow fusion feature vector; the deep feature fusion layer is used to perform feature fusion on the hidden layer state vectors of the imitation learning networks corresponding to all the decoding layers after its corresponding decoding layer in the dynamic early exit decision module to obtain a deep fusion feature vector; the fusion gating layer is used to perform feature fusion on the shallow fusion feature vector and the deep fusion feature vector to obtain a final fusion feature vector; the classification layer is configured as a fully connected layer and is used to output the prediction probabilities of each word in the prediction vocabulary according to the final fusion feature vector. In some embodiments of the present invention, the shallow feature fusion layer performs feature fusion in any one of the following ways: splicing, attention weighting, and time series model. The deep feature fusion layer performs feature fusion in any one of the following ways: splicing, attention weighting, and time series model.

[0007] Preferably, each of the imitation learning networks includes a feed-forward neural network.

[0008] Preferably, the visual encoder includes 6 encoding layers, and the text decoder includes 6 decoding layers.

[0009] According to a second aspect of the present invention, there is provided a method for training an image caption generation model as described in the first aspect of the present invention. The method includes: S1. Obtain an image set and natural language description texts corresponding to all images in the image set. Combine each image with its corresponding natural language description text into a sample to form a data set, and divide the data set into a training set and a test set. The words in the natural language texts corresponding to all images form a prediction vocabulary; S2. Use the training set to train the image generation description model until convergence; S3. Use the test set to test the trained image generation description model, and set a confidence threshold for the dynamic early exit decision module corresponding to each decoding layer, so that the dynamic early exit decision module determines whether the current prediction needs to exit early based on the word prediction probability in the prediction vocabulary during the word prediction process, and outputs the word corresponding to the maximum probability when the maximum probability in the word prediction probability in the prediction vocabulary exceeds the confidence threshold.

[0010] Preferably, step S1 includes: S11. Obtain an image set and natural language description texts corresponding to all images in the image set. Duplicate the images whose corresponding natural language description texts exceed one sentence according to the number of sentences of their corresponding natural language description texts, so that one text corresponds to one image to form an image-text pair, and all the image-text pairs are used as samples to form a data set; S12. Preprocess all the natural language texts corresponding to all images in the image set to extract all the words in the text, count the word occurrence frequencies and delete the words whose occurrence times are less than the preset frequency to obtain a prediction vocabulary; S13. Divide the data set into a training set and a test set.

[0011] In some embodiments of the present invention, the following loss function is used to train the model in step S2:

[0012] L = λ·L ce +(1 - λ)·L imit

[0013] where λ is a balance factor for adjusting the influence of the dynamic early exit decision module and the imitation learning network, L ce is the cross-entropy loss between the predicted word and the correct word, and L imit is the imitation loss between the predicted hidden layer state vector imitated by the imitation learning network and the true hidden layer state vector output by the decoding layer.

[0014] Preferably, the cross-entropy loss between the predicted word and the correct word is calculated in the following manner:

[0015]

[0016] p m = softmax(zm )

[0017] Among them, N is the total number of decoding layers, y i represents the predicted word, softmax represents the activation function, and z m represents the final fused feature vector input to the m-th classification module.

[0018] Preferably, the final fused feature vector is calculated in the following manner:

[0019] z m = αh shallow +(1 - α)h deep

[0020] α = σ(FFN(h shallow ))

[0021] where h shallow is the shallow fused feature vector obtained by fusing the hidden layer state vectors of the decoding layer corresponding to the m-th dynamic early exit decision module and all decoding layers before this decoding layer, and h deep is the deep fused feature vector obtained by fusing the hidden layer state vectors of the imitation learning network corresponding to all decoding layers after the decoding layer corresponding to the m-th dynamic early exit decision module. FFN(·) represents the feedforward neural network, and σ represents the sigmoid activation function.

[0022] Preferably, the imitation loss between the predicted hidden layer state vector imitated by the imitation learning network and the true hidden layer state vector output by the decoding layer is calculated in the following manner:

[0023]

[0024] where h k represents the true hidden layer state vector output by the k-th decoding layer, represents the hidden layer state vector imitated and output by the k-th imitation learning network when exiting early at the m-th classification layer, represents the pre-computed similarity between h k and .

[0025] Preferably, the confidence threshold is set to any value within the range of [0.5, 1) according to the requirements of the application scenario.

[0026] According to the third aspect of the present invention, there is provided an image description generation method, the method comprising: T1, obtaining an image to be processed; T2, using the image description generation model trained by the method according to the second aspect of the present invention to identify the image to be processed to generate a text description.

[0027] Compared with the prior art, the advantages of the present invention are as follows: The present invention designs an image description generation model and method based on dynamic early exit, which can generate a natural language text that can describe the visual content for a given image, and can achieve the acceleration of the generation process and the real-time adjustment of the acceleration ratio. The present invention effectively solves the problems that the conventional picture description generation methods cannot adjust the acceleration ratio of the model generation process in real time and the text generation speed is slow. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The embodiments of the present invention will be further described below with reference to the accompanying drawings, wherein:

[0029] Figure 1 is a schematic diagram of the system framework of the image description generation model according to the embodiment of the present invention;

[0030] Figure 2 is a schematic diagram of the structure of the encoding layer according to the embodiment of the present invention;

[0031] Figure 3 is a schematic diagram of the structure of the decoding layer according to the embodiment of the present invention;

[0032] Figure 4 is a schematic diagram of the structure of the dynamic early exit decision module according to the embodiment of the present invention;

[0033] Figure 5 is a schematic diagram of the structure of the imitation learning network according to the embodiment of the present invention;

[0034] Figure 6 is a schematic diagram of the construction of the feature vector of the imitation learning network when there is dynamic early exit according to the embodiment of the present invention;

[0035] Figure 7 is a schematic diagram of the feature fusion of the dynamic early exit decision module according to the embodiment of the present invention;

[0036] Figure 8 is a schematic diagram of the comparison of the experimental result data according to the embodiment of the present invention;

[0037] Figure 9 is a schematic diagram of the comparison of the model performance of the early exit models using the fused features and not using the fused features under different acceleration ratio conditions according to the embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0038] In order to make the objectives, technical solutions and advantages of the present invention clearer and more understandable, the present invention will be further described in detail below through specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0039] As described in the background art, the goal of the image description generation task is to generate a natural language text that can describe the visual content for a given image. The image description generation model predicts and outputs a text composed of multiple words. Existing general image description generation technologies have problems such as slow text generation speed and inability to adjust the model acceleration ratio in real time (the acceleration ratio is the speed-up multiple of the model decoding speed. If it was originally 200 ms and now it is 50 ms, the acceleration ratio is 4).

[0040] The inventors have found through research that the reason for the slow text generation speed of existing image description generation models is that when generating words in the encoding and decoding process of existing image description generation models, each generated word is output after passing through all decoding layers, which greatly reduces the generation speed. To address the above problems, the present invention proposes an image description generation model architecture based on dynamic early exit. The image description generation model of the present invention includes a visual encoder and a text decoder. Different from the prior art, the present invention configures a dynamic early exit decision module in each decoding layer of the text decoder, so that each generated word does not need to pass through all decoding layers. Instead, in each dynamic early exit decision module of a certain decoding layer, it determines whether the current prediction needs to exit early according to the prediction probability of words in the prediction vocabulary, and outputs the word corresponding to the maximum probability when the maximum probability among the prediction probabilities of words in the prediction vocabulary exceeds the confidence threshold, thereby achieving an improvement in the decoding speed by saving the forward propagation calculation process of subsequent decoding layers.

[0041] To better understand the present invention, the present invention will be described in detail below with reference to the accompanying drawings and embodiments.

[0042] According to an embodiment of the present invention, there is provided an image description generation model based on dynamic early exit, as Figure 1As shown in the figure, the image description generation model is implemented using the Transformer architecture, including a visual encoder and a text decoder. The visual encoder includes a plurality of sequentially connected encoding layers (for convenience of description, in subsequent embodiments, the visual encoder is described by taking the example of including N encoding layers, and is sequentially numbered as encoding layer 1, encoding layer 2, …, encoding layer N from front to back). The text decoder includes a plurality of sequentially connected decoding layers (for convenience of description, in subsequent embodiments, the text decoder is described by taking the example of including N decoding layers, and is sequentially numbered as decoding layer 1, decoding layer 2, …, decoding layer N from front to back). Among them, each decoding layer is configured with a dynamic early exit decision module and an imitation learning network. The input end of each dynamic early exit decision module is connected to the outputs of its corresponding decoding layer and all the decoding layers before it, and the outputs of the imitation learning networks corresponding to all the decoding layers after its corresponding decoding layer. The input end of each imitation learning network is connected to the outputs of its corresponding decoding layer and all the decoding layers before it, and is used to imitate and predict the output hidden layer state vector of its corresponding decoding layer according to the input. According to an embodiment of the present invention, as Figure 2 shown, each of the encoding layers includes a self-attention layer and a feed-forward neural network connected in sequence. Among them, for each encoding layer, the input of the self-attention layer in the current encoding layer is the input of the current encoding layer, the output of the self-attention layer in the current encoding layer is the input of the feed-forward neural network in the current encoding layer, and the output of the feed-forward neural network in the current encoding layer is the output of the current encoding layer and serves as the input of the next encoding layer. According to an embodiment of the present invention, as Figure 3 shown, each of the decoding layers includes a self-attention layer, an encoder-decoder self-attention layer, and a feed-forward neural network connected in sequence. Among them, for each decoding layer, the input of the self-attention layer in the current decoding layer is the input of the current decoding layer, the output of the self-attention layer in the current decoding layer is the input of the encoder-decoder self-attention layer in the current decoding layer, the output of the encoder-decoder self-attention layer in the current decoding layer is the input of the feed-forward neural network in the current decoding layer, and the output of the feed-forward neural network in the current decoding layer is the output of the decoding layer and serves as the input of the next decoding layer. According to an embodiment of the present invention, as Figure 4As described above, each dynamic early-exit decision module includes a shallow feature fusion layer, a deep feature fusion layer, a fusion gating layer, and a classification layer, where: the shallow feature fusion layer is used to fuse the hidden layer state vectors of the decoding layer corresponding to the dynamic early-exit decision module where it is located and all decoding layers before this decoding layer to obtain a shallow fusion feature vector; the deep feature fusion layer is used to fuse the hidden layer state vectors of the imitation learning networks corresponding to all decoding layers after the decoding layer corresponding to the dynamic early-exit decision module where it is located to obtain a deep fusion feature vector; the fusion gating layer is used to fuse the shallow fusion feature vector and the deep fusion feature vector to obtain a final fusion feature vector; the classification layer is configured as a fully connected layer, and is used to output the prediction probability of each word in the prediction vocabulary according to the final fusion feature vector. According to an embodiment of the present invention, as Figure 5 shown, each of the imitation learning networks includes a feedforward neural network. Different from the traditional Transformer model, in the text decoder of the model of the present invention, a dynamic early-exit decision module including a fully connected layer as a classifier is configured after each decoding layer, which is used to judge whether to directly output the current predicted word at the current layer and omit the forward propagation process of the subsequent decoding layers, and when the prediction word probability of the classifier after the decoding layer reaches a certain confidence threshold (the output of the classifier is the prediction probability of all words, and the confidence threshold is the probability value corresponding to the word with the highest probability, and this threshold can be set as any value in the range of [0.5, 1) as a hyperparameter, such as 0.5 / 0.6 / 0.7 / 0.8 / 0.9), the word is output in advance, and the decoding speed is improved by saving the forward propagation calculation process of the subsequent decoding layers. Among them, it should be noted that the data transmitted between different encoding layers, between the last encoding layer and the first decoding layer, between different decoding layers, between the decoding layer and its corresponding dynamic decision module, between the decoding layer and its corresponding imitation learning network, and between the imitation learning network and the dynamic early-exit decision module are the hidden layer state vectors output by different layers.

[0043] According to an embodiment of the present invention, a method for training the image description generation model described in the foregoing embodiment is provided. The method includes steps S1, S2, and S3, and each step is described in detail below.

[0044] In step S1, an image set and the natural language description texts corresponding to all the images in the image set are obtained. The data set is preprocessed to form a sample by combining each image with its corresponding natural language description text, thus constituting the data set. The data set is divided into a training set and a test set, and the words in the natural language texts corresponding to all the images form a prediction vocabulary. According to an embodiment of the present invention, the publicly available MS COCO Captions data set is adopted in the present invention. This data set includes 123,287 pictures, and each picture has 5 corresponding natural language description texts. In the data preprocessing stage, for the picture data, open-source visual features are used to omit the operation of extracting video features; for the text data, all the words are converted to lowercase and then special symbols are deleted. The frequency of each word is counted, and the words that appear less than 5 times are deleted to obtain the prediction vocabulary. Each picture is copied 4 times to form a picture-text pair with each corresponding text as a sample, thus constituting the data set. The data set is divided into a training set and a test set. According to an embodiment of the present invention, the data set is divided into a training set and a test set in a ratio of 7:3.

[0045] In step S2, the image generation description model is trained using the training set until convergence. According to an embodiment of the present invention, during training, a batch of picture-text pairs are input into the model to update the model parameters. Among them, the pictures in the samples are input into the visual encoder to obtain the predicted text, and the texts in the samples are used as reference texts and input into the text decoder together with the predicted text. All the predicted words are output after the last decoding layer, and the total training loss function is used for optimization. According to an embodiment of the present invention, the number of encoding layers in the visual encoder is 6, the number of decoding layers in the text decoder is 6, the dimension of the hidden layer state vector is 512, and the input of the fully connected layer is 2048 dimensions. During the training process, the initial learning rate of 3e-5 is used to train for 25 rounds, and then it continues to be trained with a 90% decay every 5 rounds. The optimizer uses Adam, and the prediction decoding time is calculated using the graphics card NVIDIA GTX 2080 Ti. All the operation results are obtained by taking the average of running three times to obtain the final calculation result.

[0046] In step S3, the trained image generation description model is tested using a test set. The pictures corresponding to the samples in the test set are input to obtain the predicted text, which is then input to the text decoder. The credibility threshold of the dynamic early exit decision module corresponding to each decoding layer is set, so that the dynamic early exit decision module determines whether to exit early during the word prediction process based on the word prediction probability in the prediction vocabulary, and outputs the word corresponding to the maximum probability when the maximum probability in the word prediction probability in the prediction vocabulary exceeds the credibility threshold. According to an embodiment of the present invention, during model testing, the credibility thresholds of each layer classifier ({0.5, 0.6, 0.7, 0.8, 0.9}) are adjusted. When the maximum probability among the prediction probabilities of all words output by the classifier exceeds the set credibility threshold, the word corresponding to the maximum probability exits at the current layer.

[0047] However, when the model exits the forward propagation process prematurely, the feature vectors of the deep information of the uncalculated decoding layers will be missing, which will cause the classifier to predict words inaccurately, thereby reducing the quality of the generated text. To solve this problem, the present invention has modified the feature vectors input to the classification layer when exiting the decoding at a certain decoding layer, and uses the method of feature fusion to fuse the shallow features and deep features and use them as the input of the classification layer. Taking the image generation model exiting the decoding at the m-th layer as an example, the deep feature vectors of the subsequent decoding layers from the m + 1-th layer to the N-th layer will be lost. The present invention uses an imitation learning network to predict the deep feature vectors of the decoding layers from the m + 1-th layer to the N-th layer, and fuses the shallow feature vectors and deep feature vectors respectively and then performs fusion to obtain the final fused feature vector, which is used as the input of the classification layer in the dynamic early exit decision module corresponding to the m-th decoding layer. Still taking Figure 1 the image description generation model structure shown in Figure 6 as an example, assuming that the decoding exits at the m-th decoding layer, the deep information of the decoding layers from the m + 1-th layer to the N-th layer will be missing. As shown in For example Figure 6 the h shown in m and Among them, imitation learning refers to a learning mode characterized by imitating the behavior mode of an example. In the present invention, a feedforward neural network is used to imitate the deep decoding layer to calculate the deep features. As shown in Figure 7As shown, in the present invention, all shallow features and deep features are fused to construct the current feature vector, which is used as the input of the classification layer. This not only preserves the existing historical information but also includes the deep information of the future decoding layer obtained by prediction, thus avoiding information loss. Therefore, the feature vector of the m-th decoding layer consists of two parts: shallow features and deep features, which come from the first m decoding layers and the subsequent (N - m) decoding layers respectively. The modeling process is as follows:

[0048] To aggregate all past shallow features {h1,...,h m} from a global perspective, a feature fusion strategy g(·) is used in the shallow feature fusion layer to obtain the shallow fusion state vector h shallow :

[0049] h shallow = g({h1,...,h m})

[0050] Among them, the feature fusion strategy has various implementation forms. The present invention provides the following three:

[0051] 1) Concatenation: All shallow features {h1,...,h m} are concatenated in order into a whole vector, and then input into a fully connected layer for dimensionality reduction. Among them, the concatenated vector is [h1;...;h m , and the fully connected layer is represented by FC(·). Then the fusion state vector is h shallow = FC([h1;...;h m ).

[0052] 2) Attention weighting: m attention weight values with a sum of weights of 1 are used to weight the corresponding hidden layer state vectors. Assume that the attention weight of the i-th hidden layer state vector is w i . Then the weighted fusion state vector is

[0053] 3) Temporal model: A layer of long short-term memory network is used to encode the series of hidden layer state vectors {h1,...,h m}, and the output state vector of the last layer of the network is taken as the fusion state vector. The long short-term memory network is represented by LSTM(·). Then the fusion state vector is h shallow = LSTM({h1,...,h m}).

[0054] However, as mentioned above, if only the shallow features of history are aggregated while ignoring the deep features of the future, information loss will occur, resulting in poor quality of the generated descriptive text. Therefore, the present invention uses imitation learning to predict the deep features and fuse them with the shallow features of history. Specifically, an imitation learning network is equipped for each decoding layer to imitate the calculation process of the decoding layer. During the training process, in the forward propagation process through all N decoding layers of the model, the imitation learning network is encouraged to imitate the hidden layer state vector output by the output decoding layer. Assuming that the decoding process ends at the m-th layer, when any subsequent layer (assumed to be the k-th layer) performs imitation learning, it receives the hidden layer state vector h m as input. Therefore, the output of the imitation learning network, i.e., the predicted deep feature vector can be modeled as:

[0055]

[0056] where FFN k (·) represents the k-th layer imitation learning network, which is implemented using a feed-forward neural network.

[0057] Through imitation learning, an approximate result of the hidden layer state vector output by the uncomputed layer can be obtained at the smallest cost. Among them, the similarity between the true value h k and the predicted value generated by the imitation learning network is calculated using cosine similarity:

[0058]

[0059] Therefore, the loss function of imitation learning can be implemented using the average cosine similarity corresponding to all possible exit positions:

[0060]

[0061] The deep fusion feature vector h deep is also calculated using the feature fusion strategy g(·) in the deep feature fusion layer:

[0062]

[0063] After obtaining the shallow fusion state and the deep fusion state, since their reliabilities are different, they cannot be directly added. The present invention uses an adaptive fusion gating mechanism to weight according to the reliabilities of the shallow fusion state vector and the deep fusion state vector:

[0064] α = σ(FFN(h shallow ))

[0065] z m = αh shallow+(1α)h deep

[0066] where z m is the final feature vector after fusing shallow features and deep features, FFN is a feed-forward neural network, and σ is the sigmoid activation function.

[0067] During the training process, it is not known which layer will exit early. Therefore, the classification layer of each decoding layer uses the final fused feature vector to predict the current word distribution. When this distribution meets the confidence threshold for exit, the predicted text is output. And the cross-entropy loss between the predicted word and the correct word y i is used as the generation loss to train the model:

[0068]

[0069] where p m = softmax(z m ).

[0070] Therefore, the total training loss function adjusted by the parameter λ is:

[0071] L = λ · L ce +(1 - λ) · L imit

[0072] where λ is a balance factor that adjusts the influence of the dynamic early exit decision module and the imitation learning network, and it is a constant within the range of (0, 1).

[0073] To better verify the effectiveness of the present invention, the present invention conducts comparative experiments with different models (represented by ours) on the MS COCO Captions public dataset, and compares the following indicators of different models:

[0074] BLEU-4: A bilingual evaluation metric for 4-grams, which calculates the repeatability of 4-grams in the reference text and the generated text to evaluate the matching degree between the two.

[0075] METEOR: A translation evaluation metric based on explicit ranking, which calculates the accuracy and recall rate of the generated text.

[0076] ROUGE: A recall-based evaluation metric, which calculates the recall rate of the generated text.

[0077] CIDEr: A consensus-based image caption evaluation metric, which calculates the cosine similarity between the reference text and the generated text to evaluate the matching degree between the two.

[0078] SPICE: A semantic-based image caption evaluation metric.

[0079] Acceleration ratio: The multiple of the speedup of the model decoding speed.

[0080] The experimental results are as Figure 8 shown. It can be seen that the model adopting the present invention can achieve an acceleration ratio of 4 - 5 times, and the performance loss of the generated text is small.

[0081] In addition, the present invention also compares the model performance of the variant model (TF-EE) that does not use the fused feature vector for early exit word prediction and the model (DeeCap) that uses the fused feature vector for early exit word prediction under different acceleration ratio conditions. The experimental results are as Figure 9 shown. The experimental results prove that the model proposed by the present invention that uses the fused feature vector for early exit prediction can have a performance loss of less than 5% under the condition that the decoding speed is increased by 4 - 5 times, and the model can realize real-time control of the acceleration ratio by controlling the classifier output confidence threshold (that is, the switching between different points on the solid line performance curve is real-time without retraining the model).

[0082] In summary, the present invention designs an image caption generation model and method based on dynamic early exit, which can generate a natural language text that can describe the visual content for a given image, and can achieve the acceleration of the generation process and the real-time adjustment of the acceleration ratio. The present invention effectively solves the problems that the acceleration ratio of the model generation process and the text generation speed of the previous picture caption generation methods cannot be adjusted in real time.

[0083] It should be noted that although the above steps are described in a specific order, it does not mean that the steps must be executed in the above specific order. In fact, some of these steps can be executed concurrently or even the order can be changed as long as the required functions can be achieved.

[0084] The present invention can be a system, a method, and / or a computer program product. The computer program product can include a computer-readable storage medium having computer-readable program instructions thereon for causing a processor to implement various aspects of the present invention.

[0085] A computer-readable storage medium can be a tangible device that retains and stores instructions for use by an instruction execution device. A computer-readable storage medium may include, for example, but is not limited to, electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. More specific examples (non-exhaustive list) of the computer-readable storage medium include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read-only memory (CD-ROM), digital versatile disks (DVD), memory sticks, floppy disks, mechanically encoded devices such as punch cards or raised structures in grooves having instructions stored thereon, and any suitable combination of the foregoing.

[0086] The embodiments of the present invention have been described above. The above description is exemplary and not exhaustive, and is also not limited to the disclosed embodiments. Many modifications and variations are obvious to those of ordinary skill in the art in the technical field without departing from the scope and spirit of the described embodiments. The selection of the terms used herein is intended to best explain the principles of the embodiments, the practical application, or the improvement of the technology in the market, or to enable other ordinary skill in the technical field to understand the embodiments disclosed herein.

Claims

1. An image caption generation system based on dynamic early exit, which is used to output the natural language description text of an image according to the input image, and is characterized in that The system includes an image caption generation model based on dynamic early exit, and the image caption generation model includes: A visual encoder for extracting visual features of an image, which includes a plurality of encoding layers connected in series; A text decoder for decoding the visual features output by the visual encoder and sequentially outputting words in a plurality of prediction vocabulary to form a natural language description text of the image. The text decoder includes a plurality of decoding layers connected in series, and each decoding layer is configured with a dynamic early exit decision module and an imitation learning network; wherein: Each of the dynamic early exit decision modules is used to judge whether the current prediction needs to exit early according to the prediction probability of the words in the prediction vocabulary during the word prediction process, and output the word corresponding to the maximum probability when the maximum probability in the prediction probability of the words in the prediction vocabulary exceeds the confidence threshold. The input end of each dynamic early exit decision module is connected to the outputs of its corresponding decoding layer and all the decoding layers before it, and the outputs of the imitation learning networks corresponding to all the decoding layers after its corresponding decoding layer; wherein, each dynamic early exit decision module includes a shallow feature fusion layer, a deep feature fusion layer, a fusion gating layer, and a classification layer, where: the shallow feature fusion layer is used to fuse the hidden layer state vectors of its corresponding decoding layer and all the decoding layers before this decoding layer in the dynamic early exit decision module to obtain a shallow fusion feature vector; the deep feature fusion layer is used to fuse the hidden layer state vectors of the imitation learning networks corresponding to all the decoding layers after its corresponding decoding layer in the dynamic early exit decision module to obtain a deep fusion feature vector; the fusion gating layer is used to fuse the shallow fusion feature vector and the deep fusion feature vector to obtain a final fusion feature vector; the classification layer is configured as a fully connected layer, and is used to output the prediction probability of each word in the prediction vocabulary according to the final fusion feature vector; The input end of each imitation learning network is connected to the outputs of its corresponding decoding layer and all the decoding layers before it, and is used to imitate and predict the output hidden layer state vector of its corresponding decoding layer according to the input.

2. The system according to claim 1, wherein Each of the encoding layers includes a self-attention layer and a feed-forward neural network connected in sequence.

3. The system according to claim 2, wherein Each of the decoding layers includes a self-attention layer, an encoder-decoder self-attention layer, and a feed-forward neural network connected in sequence.

4. The system according to claim 3, characterized in that, The shallow feature fusion layer performs feature fusion in any of the following ways: Concatenation, attention weighting, time series model.

5. The system according to claim 3, wherein The deep feature fusion layer performs feature fusion in any of the following ways: Concatenation, attention weighting, time series model.

6. The system according to claim 3, characterized in that, Each of the imitation learning networks includes a feed-forward neural network.

7. The system according to claim 4, characterized in that, The visual encoder includes 6 encoding layers, and the text decoder includes 6 decoding layers.

8. A method for training an image caption generation system according to any one of claims 1-7, characterized in that, The method includes: S1. Obtain an image set and the natural language description texts corresponding to all the images in the image set. Combine each image with its corresponding natural language description text into a sample to form a data set, and divide the data set into a training set and a test set. The words in the natural language texts corresponding to all the images form a prediction vocabulary; S2. Use the training set to train the image generation description model until convergence; S3. Use the test set to test the trained image generation description model, and set the confidence threshold for the dynamic early exit decision module corresponding to each decoding layer, so that the dynamic early exit decision module determines whether to exit early during the word prediction process based on the prediction probability of the words in the prediction vocabulary, and outputs the word corresponding to the maximum probability when the maximum probability in the prediction probability of the words in the prediction vocabulary exceeds the confidence threshold.

9. The method according to claim 8, wherein The step S1 includes: S11. Obtain an image set and the natural language description texts corresponding to all the images in the image set. Duplicate the images whose corresponding natural language description texts exceed one sentence according to the number of sentences of their corresponding natural language description texts, so that one text corresponds to one image to form image-text pairs. All the image-text pairs are used as samples to form a data set. S12. Preprocess all the natural language texts corresponding to all the images in the image set to extract all the words in the texts. Count the word occurrence frequencies and delete the words whose occurrence times are less than the preset frequency to obtain a prediction vocabulary. S13. Divide the data set into a training set and a test set.

10. The method according to claim 8, wherein In the step S2, the following loss function is used to train the model: Among them, is a balance factor for adjusting the influence of the dynamic early exit decision module and the imitation learning network, is the cross-entropy loss between the predicted word and the correct word, is the imitation loss between the predicted hidden layer state vector generated by the imitation learning network and the true hidden layer state vector output by the decoding layer.

11. The method according to claim 10, wherein Calculate the cross-entropy loss between the predicted word and the correct word in the following way: where N is the total number of decoding layers, represents the predicted word, represents the activation function, represents the input to the final fused feature vector of the 12. The method according to claim 11, wherein The final fused feature vector is calculated in the following way: Among them, is the shallow fusion feature vector obtained after fusing the hidden layer state vectors of the decoding layer corresponding to the -th dynamic early exit decision module and all decoding layers before this decoding layer, is the deep fusion feature vector obtained after fusing the hidden layer state vectors of the imitation learning network corresponding to all decoding layers after the decoding layer corresponding to the -th dynamic early exit decision module, represents a feedforward neural network, represents a sigmoid activation function.

13. The method according to claim 10, wherein Calculate the imitation loss between the predicted hidden layer state vector generated by the imitation learning network through imitation and the true hidden layer state vector output by the decoding layer in the following way: Among them, represents the true hidden layer state vector output by the th decoding layer, represents the hidden layer state vector output by imitation of the th imitation learning network when the th classification layer exits early, represents pre - similarity.

14. The method according to claim 9, wherein The confidence threshold is set to any value within the range of [0.5, 1) according to the requirements of the application scenario.

15. An image description generation method, characterized in that, The method includes: T1. Obtain an image to be processed. T2. Use the image description generation model trained by the method according to any one of claims 8-14 to identify the image to be processed to generate a text description.

16. A computer-readable storage medium, characterized in that, A computer program is stored thereon, and the computer program can be executed by a processor to implement the steps of the method according to any one of claims 8 to 14 and 15.

17. An electronic device, characterized in that, It includes: One or more processors; A storage device for storing one or more programs, which when executed by the one or more processors, cause the electronic device to implement the steps of the method according to any one of claims 8 to 14 and 15.

Images