Unsupervised image caption generation method, system and storage medium based on prior probability distribution

Abstract

The present invention discloses an unsupervised image caption generation method, system, and storage medium based on prior probability distribution. The method comprises: constructing a training set based on acquired image and text datasets; samples in the training set include image features of each image and corresponding pseudo-real sentences in a sentence space; inputting the image features into a feature-aggregated long-short-term memory recurrent neural network in an unsupervised image caption generation model to obtain generated sentences, so that the generated sentences are globally aligned with the image features; calculating the predicted distribution of the generated sentences and the prior probability distribution in the semantic space based on the generated sentences; training the unsupervised image caption generation model using the training set, updating the parameters in the unsupervised image caption generation model based on a total loss function, and obtaining a trained unsupervised image caption generation model; and inputting an image for which captions are to be generated into the trained unsupervised image caption generation model to generate captions for the image. The present invention can generate captions that match the image content.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method, system, terminal device and computer-readable storage medium for generating unsupervised image captions based on prior probability distribution. Background Art

[0002] With the significant improvements of deep learning in natural language processing and computer vision, the task of image captioning has attracted widespread attention from researchers. The purpose of image captioning is to convert the visual features of an image into high-level semantic information, so that the machine can generate image descriptions similar to human understanding, which is beneficial for image classification and object / scene recognition, etc. Researchers have proposed various methods for generating image captions, ranging from adversarial active learning to iterative reverse modification, and using shared encoders of visual and language domains to improve intra-modal and inter-modal visual relationships. However, most image captioning models are based on paired image-sentence pairs, which are too laborious to collect. Therefore, many methods have been proposed to utilize unlabeled data, such as unsupervised image captioning, unpaired image captioning, cross-domain captioning, and semi-supervised captioning. Among them, unsupervised image captioning builds a bridge between visual information and language information in an unsupervised manner.

[0003] Most image captioning models use maximum likelihood estimation (MLE) to align unlabeled images and sentences. Although MLE has achieved great success due to its intuitiveness and flexibility in natural language processing, it still suffers from many problems, such as negative sample bias, where all incorrect words are treated equally no matter how similar they are to the correct words. For example, a set of words "sheep, sheep, house" are generated as candidate words, and "sheep" is the target word. Although "goat" is more similar to "sheep" than "house", MLE treats these two words equally because they are incorrect predictions. In addition, during the training process of unsupervised image captioning, since images and sentences do not correspond to each other, it is not enough to adopt only a one-time matching strategy. When generating target words, it is necessary to pay attention to the probability distribution of the target word in the entire sentence corpus and its relationship with other words. Summary of the Invention

[0004] In order to address the above-mentioned deficiencies in the prior art, the present invention provides a method, system, terminal device and computer-readable storage medium for generating unsupervised image subtitles based on prior probability distribution, which can generate subtitles that match the image content.

[0005] The first object of the present invention is to provide an unsupervised image caption generation method based on prior probability distribution.

[0006] The second object of the present invention is to provide an unsupervised image caption generation system based on prior probability distribution.

[0007] The third object of the present invention is to provide a terminal device.

[0008] A fourth object of the present invention is to provide a computer-readable storage medium.

[0009] The first object of the present invention can be achieved by adopting the following technical solutions:

[0010] An unsupervised image caption generation method based on prior probability distribution, the method comprising:

[0011] Constructing a training set based on the acquired image dataset and text dataset; the samples in the training set include image features of each image and corresponding pseudo-real sentences in the sentence space;

[0012] The image features are input into the feature-aggregated long short-term memory recurrent neural network in the unsupervised image caption generation model to obtain a generated sentence so that the generated sentence is globally aligned with the image features; based on the generated sentence, the predicted distribution of the generated sentence and the prior probability distribution in the semantic space are calculated;

[0013] The unsupervised image caption generation model is trained using the training set, and parameters in the unsupervised image caption generation model are updated based on a total loss function to obtain a trained unsupervised image caption generation model; the total loss function includes a cosine loss function and a KL loss function, the cosine loss function compares the prior probability distribution of the generated sentences in the semantic space with the predicted distribution of the generated sentences to increase the diversity of the generated sentences; the KL loss function compares the predicted distribution of the generated sentences with the distribution of the corresponding pseudo-real sentences to improve the accuracy of the unsupervised image caption generation model;

[0014] The image to be captioned is input into the feature aggregation long short-term memory recurrent neural network in the trained unsupervised image caption generation model to generate captions for the corresponding image.

[0015] Furthermore, the calculation process of generating the prior probability distribution of the sentence in the semantic space is as follows:

[0016] Calculate the cosine similarity between each word and every other word in the text dataset;

[0017] Calculate the evaluation score based on the cosine similarity between words;

[0018] According to the evaluation scores, the temperature mechanism is used to calculate the prior probability distribution of words in the semantic space;

[0019] The prior probability distributions of all words in the generated sentence are merged to obtain the prior probability distribution of the generated sentence in the semantic space.

[0020] Furthermore, the image features are input into the feature aggregation long short-term memory recurrent neural network in the unsupervised image caption generation model to obtain the generated sentences, including:

[0021] The image features are input into the feature aggregation long short-term memory recurrent neural network, which is expressed as follows:

[0022]

[0023]

[0024] Where y i represents the word embedding after the i-th iteration; σ represents the Sigmoid activation function; odd numbers i∈[1,…r] and even numbers j∈[2…r] represent the iteration rounds; r represents the total number of iterations; y -1 and h 0 Represent the initialized word embedding and hidden layer state respectively; h j and Represent the hidden layer state and image features after the jth iteration, Indicates F img ; R j and Q i Both represent randomly initialized matrices;

[0025] According to the word embedding after r iterations, the generated sentence is obtained.

[0026] Furthermore, the calculation process of the predicted distribution of the generated sentence is:

[0027] The iterated word embedding and hidden layer state are input into LSTM to obtain the predicted distribution p of the generated sentence θ (y):

[0028] y i =LSTM(h r-1 ,y r )

[0029] p θ (y)=concat(y1,y2,…,y l )

[0030] Where h r-1 and y r They represent the hidden layer state and word embedding after iteration, and l represents the number of words in the sentence.

[0031] Furthermore, the KL loss function is:

[0032]

[0033] Where r(y), pθ (y) denote the distribution of pseudo-true sentences and the predicted distribution of generated sentences, respectively.

[0034] Furthermore, the cosine loss function is:

[0035]

[0036] Where, q(y), p θ (y) represents the prior probability distribution of the generated sentence in the semantic space and the predicted distribution of the generated sentence, g t represents the gate indicator, and l represents the total number of words in the generated sentence.

[0037] Furthermore, the total loss function also includes a gated loss function, which compares the words in the generated sentence with the targets extracted from the image to determine the importance of the words in the entire sentence when describing the image. The specific expression is:

[0038]

[0039] in,

[0040] Where W g ∈R d×d and W t ∈R d×d Both represent linear transformation matrices, d represents the dimension of word vectors; W t V represents the word vector of the target V after dimension transformation to get the same dimension as the word. Indicates the calculation of two word vectors W t Evaluation score between V.

[0041] Furthermore, constructing a training set based on the acquired image dataset and text dataset includes:

[0042] Encode the images in the image dataset to obtain image features and multiple targets for each image;

[0043] Based on multiple targets and text datasets for each image, the visual concept of each image is obtained; the target-concept dictionary is constructed from the visual concepts of all images;

[0044] According to the target-concept dictionary and text dataset, a concept-caption dictionary is obtained;

[0045] Retrieve the visual concept of each image from the concept-caption dictionary to obtain pseudo-true sentences for all images and form a sentence space;

[0046] The image features of each image and the corresponding pseudo-real sentences in the sentence space are used as samples in the training set.

[0047] The second object of the present invention can be achieved by adopting the following technical solutions:

[0048] An unsupervised image caption generation system based on prior probability distribution, the system comprising:

[0049] A construction module is used to construct a training set based on the acquired image dataset and text dataset; the samples in the training set include image features of each image and corresponding pseudo-real sentences in the sentence space;

[0050] The sentence generation module is used to input image features into the feature-aggregated long short-term memory recurrent neural network in the unsupervised image caption generation model to generate sentences so that the generated sentences are globally aligned with the image features. Based on the generated sentences, the predicted distribution of the generated sentences and the prior probability distribution in the semantic space are calculated.

[0051] A training module is configured to train an unsupervised image caption generation model using the training set, and to update parameters in the unsupervised image caption generation model based on a total loss function to obtain a trained unsupervised image caption generation model; the total loss function includes a cosine loss function and a KL loss function, wherein the cosine loss function compares the prior probability distribution of generated sentences in a semantic space with the predicted distribution of generated sentences to increase the diversity of generated sentences; and the KL loss function compares the predicted distribution of generated sentences with the distribution of corresponding pseudo-real sentences to improve the accuracy of the unsupervised image caption generation model;

[0052] The subtitle generation module is used to input the image to be subtitled into the trained unsupervised image subtitle generation model to generate subtitles for the corresponding image.

[0053] The third object of the present invention can be achieved by adopting the following technical solutions:

[0054] A terminal device includes a processor and a memory for storing a program executable by the processor. When the processor executes the program stored in the memory, the above-mentioned unsupervised image caption generation method based on prior probability distribution is implemented.

[0055] The fourth object of the present invention can be achieved by adopting the following technical solutions:

[0056] A computer-readable storage medium stores a program, which, when executed by a processor, implements the above-mentioned unsupervised image caption generation method based on prior probability distribution.

[0057] The present invention has the following beneficial effects compared to the prior art:

[0058] The present invention provides an unsupervised image subtitle generation method, system, terminal device and computer-readable storage medium based on prior probability distribution, comprising: constructing a training set based on an acquired image dataset and a text dataset; the samples in the training set include image features of each image and corresponding pseudo-real sentences in a sentence space; inputting the image features into a feature-aggregated long short-term memory recursive neural network in an unsupervised image subtitle generation model to obtain generated sentences, so that the generated sentences are globally aligned with the image features; calculating the predicted distribution of the generated sentences and the prior probability distribution in the semantic space based on the generated sentences; and using the training set to train the unsupervised image subtitle generation model. The method comprises the following steps: training, updating the parameters of the unsupervised image caption generation model based on the total loss function to obtain a trained unsupervised image caption generation model; the total loss function includes a cosine loss function and a KL loss function, the cosine loss function compares the prior probability distribution of the generated sentences in the semantic space with the predicted distribution of the generated sentences to increase the diversity of the generated sentences; the KL loss function compares the predicted distribution of the generated sentences with the distribution of the corresponding pseudo-real sentences to improve the accuracy of the unsupervised image caption generation model; the image to be captioned is input into the feature aggregation long short-term memory recurrent neural network in the trained unsupervised image caption generation model to generate captions for the corresponding image. The present invention utilizes a feature aggregation long short-term memory network to generate image descriptions to enhance the correlation between the generated sentences and the image; in the total loss function, by combining the comparison of the prior probability distribution of the generated sentences in the semantic space with the predicted distribution of the generated sentences and the comparison of the predicted distribution of the generated sentences with the distribution (posterior probability distribution) of the corresponding pseudo-real sentences, the negative sample bias problem in the traditional text generation field is solved, and the image caption description that conforms to the semantic relationship and has a high degree of matching with the image can be accurately generated. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the structures shown in these drawings without paying any creative work.

[0060] Figure 1 This is a simplified flowchart of the unsupervised image caption generation method based on prior probability distribution according to Example 1 of the present invention;

[0061] Figure 2 Detailed flowchart of the unsupervised image caption generation method based on prior probability distribution according to Example 1 of the present invention;

[0062] Figure 3 This is a schematic diagram of the FM-LSTM process of Example 1 of the present invention;

[0063] Figure 4 Schematic diagram of the semantic space of Example 1 of the present invention;

[0064] Figure 5 This is a structural block diagram of an unsupervised image caption generation system based on prior probability distribution according to embodiment 2 of the present invention;

[0065] Figure 6 This is a structural block diagram of the terminal device of embodiment 3 of the present invention. DETAILED DESCRIPTION

[0066] To make the purpose, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention. It should be understood that the specific embodiments described are only used to explain this application and are not used to limit this application.

[0067] Example 1:

[0068] This example is based on the Pytorch framework and the Pycharm development environment: Pytorch is a deep learning development framework based on the Python language. It provides interfaces for numerous encapsulated functions and image processing functions used in deep learning, making it easy for users to build deep learning networks and enabling GPU-accelerated graphics computing. The Pycharm integrated development environment is one of the primary tools for Python programming, providing programming, testing, and debugging tools, and can call compilers on remote servers to run programs.

[0069] The unsupervised image caption generation method based on prior probability distribution provided by this embodiment mainly includes data construction, model structure design, model training and model application. Among them, data construction is mainly based on unsupervised image and text data, and image features and visual concepts are obtained based on the inherent information of the image; word vectors are obtained based on the contextual relationship between words in the text, and the obtained image and text features are used as the input of the model; the model structure design is mainly based on a long short-term memory neural network framework with feature aggregation, and a prior probability distribution based on gated data dependency and a posterior probability distribution are designed to solve the negative sample bias problem in the field of traditional text generation, accurately generate image caption descriptions that conform to semantic relationships and have a high degree of matching with the image; wherein the prior probability distribution effectively combines the current generated word with its neighboring words by constructing a semantic space, thereby improving the diversity of generated words; the posterior probability distribution compares the distribution similarity between the real sentence and the generated sentence by using KL divergence, thereby achieving more realistic captions; model training is to train the designed model structure using a data set and adjust the model parameters according to the value of the loss function; model application is to apply the trained model to actual testing, that is, output the caption description of the corresponding image based on the input image data.

[0070] Specifically, the image is first feature encoded to obtain the image feature F containing global information img And the target V including local information; build a target-concept dictionary D based on the target V and text information oc , and obtain visual concepts including phrase concepts M and word concepts O, aiming to obtain the implicit latent topic distribution of multiple words and images and the relationship between them; then, the feature aggregation long short-term memory network (FM-LSTM) is used to generate image descriptions, taking the frozen image features and the context composed of word embeddings as input to enhance the correlation between the generated sentences and images; by constructing a semantic space, the distribution of the generated sentences is compared with the prior probability distribution in the semantic space to increase the diversity of the generated sentences; next, the frequency of the previously generated visual concepts in the sentences is used to construct a concept-caption dictionary This algorithm is used to determine the pseudo-true sentence that is most relevant to the current image. KL divergence is used as the posterior probability distribution to compare the difference between the predicted sentence and the pseudo-true sentence. This approach improves the accuracy and diversity of generated sentences by considering both the prior and posterior probability distributions. By building a dictionary and semantic space of visual concepts, it can better understand the relationship between images and text and generate descriptive sentences relevant to the image content.

[0071] like Figure 1 、 2As shown, the unsupervised image caption generation method based on prior probability distribution provided by this embodiment specifically includes the following steps:

[0072] S101. Construct a training set based on the acquired image dataset and text dataset.

[0073] The image dataset in this embodiment is mainly the public MSCOCO-2024 dataset, and the samples are scene images; the text dataset is mainly the public Shutterstock (SS) and Google Conceptual Captions (GCC), and the samples are sentences. SS and GCC are sentence datasets for image description that are unrelated to the MSCOCO-2024 image dataset.

[0074] Furthermore, step S101 includes:

[0075] (1) Encode the images in the image dataset to obtain the image features and multiple targets of each image.

[0076] First, an encoder is used to encode the image to obtain image features containing global information and objects including local information.

[0077] Generally, Faster R-CNN pre-trained on Visual Genome is used to process images and obtain object sets. k Get object v and image features F img :

[0078] V,F img =CNN(i k ),k∈{0,1,…,m}

[0079] Where m represents the number of images in the dataset, V = {v1,v2,…,v n} represents image i k There are n targets.

[0080] (2) Based on multiple targets of each image and text datasets, the visual concept of each image is obtained; the visual concepts of all images constitute the target-concept dictionary.

[0081] The visual concept corresponding to each image: retrieve image i k The probability of adjacent words of multiple targets appearing in the text dataset is calculated and sorted according to the probability. If the probability is greater than the set threshold, the word corresponding to the target and its adjacent words constitute a phrase concept; the word corresponding to the target is a word concept, and the phrase concept and the word concept together constitute a visual concept.

[0082] In this embodiment, the threshold is set to 10%, that is, in all phrases containing the target, if the probability of the neighbor word appearing is greater than 10%, it is a phrase concept.

[0083] The visual concepts corresponding to all images in the image dataset constitute the target-concept dictionary D oc .

[0084] This step builds a bridge between image information and text semantic information.

[0085] (3) Based on the target-concept dictionary and the text dataset, a concept-caption dictionary is obtained.

[0086] From the target-concept dictionary D oc Select k o word concepts and k m phrase concepts, forming k o +k m visual concepts as candidates;

[0087] Retrieve sentences containing candidate items in the text dataset, use visual concepts as keys and retrieved corresponding sentences as values ​​to build a concept-caption dictionary

[0088] (4) Obtain the sentence space based on the visual concepts and concept-caption dictionary of each image.

[0089] exist Retrieve the visual concept of each image from the dataset and obtain pseudo-true sentences for all images to form a sentence space.

[0090] (5) A training set is constructed based on the image features of each image and the corresponding pseudo-real sentences in the sentence space.

[0091] The image i k The corresponding image feature F img And the corresponding pseudo-real sentences in the sentence space are used as samples in the training set.

[0092] S102. Construct an unsupervised image caption generation model based on prior probability distribution.

[0093] The unsupervised image caption generation model with prior probability distribution includes feature aggregation long short-term memory recurrent neural network (FM-LSTM) and gated data-dependent prior probability distribution (G-DPPD).

[0094] (1) Feature aggregation long short-term memory recurrent neural network.

[0095] like Figure 3 As shown, FM-LSTM makes full use of context information and makes the generated sentences consistent with the image features F imgMaintain global alignment.

[0096] (1-1) Using feature aggregation long short-term memory recurrent neural network according to image feature F img , new word embedding and hidden layer states are obtained through iteration.

[0097] In traditional LSTM, memory cells (word embeddings) y and hidden states h are independent of each other and only interact in the LSTM gate, which may lead to the loss of contextual information. In order to better align image features with generated sentences, this embodiment adopts a feature aggregation long short-term memory recurrent neural network (FM-LSTM) to transform image features F img As part of the LSTM hidden layer, and F img Connect it with the output of the current iteration to generate the next word. By embedding the word y i With the hidden state h i and image features Interact, enhance the input features in the context and the alignment of global information with the image, as shown below:

[0098]

[0099]

[0100] Among them, y i represents the word embedding after the i-th iteration; σ represents the Sigmoid activation function, which maps the variable to [0,1]; odd numbers i∈[1,…r] represent the i-th iteration; even numbers j∈[2…r] represent the j-th iteration, and r is a hyperparameter representing the number of iterations; y -1 and h 0 are the initialized word embedding and hidden layer states respectively; represents the image features after the jth iteration, F img ;h j Represents the hidden layer state after the jth iteration; R and Q are both randomly initialized matrices.

[0101] In this embodiment, r is set to 5.

[0102] (1-2) Input the iterated word embedding and hidden layer state into the LSTM model to obtain the probability of the final generated sentence.

[0103] Combine the probabilities of all word predictions to form the predicted distribution p of the generated sentence θ (y):

[0104] y i =LSTM(h r-1 ,y r )

[0105] p θ (y)=concat(y1,y2,…,y l )

[0106] h r-1 and y r They represent the word embedding and hidden layer state after iteration, l represents the number of words in the sentence, and θ is the parameter to be trained.

[0107] (2) Gated data-dependent prior probability distribution (G-DPPD).

[0108] G-DPPD is the predicted distribution p of the generated sentence θ (y) and the prior probability distribution Alignment is performed to increase the diversity of generated sentences.

[0109] (2-1) Calculate the cosine similarity between each word and other words in the text dataset.

[0110] First, each word in the text dataset is encoded into a word embedding E, which extracts word information considering context and semantic features. Assume that the word is the prediction of the next word, then It will be the word embedding space except the target word Other words of ; then calculate the target word With other words The cosine similarity is:

[0111] And j≠i;

[0112]

[0113] Here, m represents the number of all words in the word embedding space.

[0114] (2-2) The evaluation scores are calculated based on the cosine similarity between words to make the words ordered, and the Gaussian probability density function is used to map these ordered words into the common space.

[0115] Introducing an evaluation score As additional supervision, we compare the similarity of the target word with other words. The higher the similarity of other words to the target word, the larger F will be. For example, if the word "sheep" is the target word, the word "goat" will have a higher score than other words. Specifically, for each target word There is an a priori topological order among all known tokens in the vocabulary in is always at the highest priority because it is the most similar to itself; then the Gaussian probability density function G(x) is used to map these ordered words into a common space. Each word prediction in this space carries relevant semantic information by calculating the evaluation score between words:

[0116]

[0117]

[0118] Although the evaluation score can reflect the attributes and positional relationships between words and other adjacent words, it still requires the distribution of all words in the entire space, and the distribution of words in the vocabulary is relatively dense.

[0119] (2-3) The temperature mechanism is used to scale the words in the public space to obtain a semantic space containing the similarity relationship between words.

[0120] Based on the evaluation scores, a softmax temperature mechanism is used to calculate the probability distribution It is used to scale the distance between words in the common space, which is called the Gated Data-Dependent Prior Probability Distribution (G-DPPD).

[0121] The distribution of each word in the text dataset constitutes a semantic space containing the similarity relationship between words, refer to Figure 4 .

[0122] Current word The prior probability distribution of for:

[0123]

[0124] Where T is the temperature parameter.

[0125] The prior probability distributions of all words in the generated sentence are combined to obtain the prior probability distribution q(y) of the generated sentence.

[0126] (3) Total loss function.

[0127] Including cosine loss function, gated loss function and KL loss function.

[0128] (3-1) Gating indicators.

[0129] A gated metric is calculated based on the similarity between the visual concept corresponding to the image and the generated word to determine the importance of the generated word in the entire sentence when describing the image.

[0130] If the semantic information of all words is compared with the predicted distribution without considering the relationship between the semantic information of the text and the image features, the model will lack a "guide word" and will not know whether the word corresponds to the target image. t Determine how important a given visual concept is to the t-th word in the generated sentence:

[0131]

[0132] Among them, W t ∈R d×d It is used to calculate the gating index g t ∈[0,1] linear transformation matrix, σ represents the Sigmoid activation function, d represents the dimension of the word vector, and V is all the targets extracted from the image.

[0133] When the gating index is close to 1, it means that the current word is closely connected to the central word of the entire sentence.

[0134] (3-2) Cosine loss function.

[0135] Compare the predicted distribution of the generated sentences with the prior probability distribution, restrict them using gating indicators, and construct a cosine loss function.

[0136] By generating the predicted distribution p of the sentence θ (y) and the prior probability distribution q(y) can construct the cosine loss function:

[0137]

[0138] (3-3) Gating loss function.

[0139] The gated loss function is calculated based on the similarity between the gated metric and the target.

[0140] The relationship between the gate and the image is compared by minimizing the following gating loss function:

[0141]

[0142] Among them, W g ∈R d×d is the linear transformation matrix, V represents all targets extracted from the current image, g t Represents the gating indicator of the tth word in the generated sentence.

[0143] (3-4)KL loss function.

[0144] Compare the predicted distribution of the generated sentences with the distribution of pseudo-true sentences and calculate the KL loss function.

[0145] The above G-DPPD effectively alleviates the negative sample bias caused by MLE, but MLE still adopts a full matching strategy and does not consider the distribution between the predicted sentence and the ground truth sequence. Therefore, KL divergence is used as the posterior probability distribution (PPD) to replace MLE, which converts the probability distribution p of the source space (generated sentence) into θ (y) and the distribution r(y) of the target space (pseudo-true sentences), where r(y) is the distribution of pseudo-true sentences. This method improves the accuracy of model predictions and measures how much information is lost in the process of choosing an approximation. This loss is called the KL loss function:

[0146]

[0147] The KL divergence can be expressed as:

[0148]

[0149] Among them, θ is the hyperparameter of the model and n is the length of the generated sentence.

[0150] S103. Use the training set to train an unsupervised image caption generation model.

[0151] According to a certain ratio, oc Visual concepts are selected to form pseudo-real sentences. Word concepts and phrase concepts are screened according to a certain ratio, and the cosine loss function and KL loss function are reconstructed by summing them.

[0152] The main task of the model is to generate unsupervised image captions, and the auxiliary task is semantic feature extraction. First, according to the target of the current image, the target-concept dictionary D oc A set of visual concepts, including words and phrases, are selected in a certain proportion. These visual concepts will be used to generate pseudo-true sentences and calculate their predicted distribution and prior probability distribution.

[0153] The loss function of G-DPPD includes word concept loss and phrase concept loss:

[0154]

[0155] Among them, W o and W m is a trainable parameter, b o and b m Indicates a biased term, λ indicates the proportion of selected words and phrases; Y indicates the predicted distribution of sentences generated by FM-LSTM, Q indicates the prior probability distribution of generated sentences, g t Represents the gating indicator.

[0156] Similarly, the posterior probability distribution (PPD) requires the use of KL divergence to evaluate the similarity between the predicted sentence and the true sentence. The loss function of PPD is:

[0157]

[0158] Where R represents the distribution of the set of pseudo-true sentences.

[0159] In experiments, the lambda of the above two loss functions is generally set to 0.4 to achieve the best effect.

[0160] Add the cosine loss function, KL loss function and gate loss function in a certain proportion to get the final loss function:

[0161]

[0162] in, Used to evaluate the authenticity of generated text, used to capture the diversity of negative training predictions, Used to calculate the correspondence between predicted words and image features.

[0163] α is a hyperparameter used to balance the objectives. In this embodiment, α is set to 0.5.

[0164] In this example, the text decoder fastText is used as a pre-trained model to encode image descriptions into word embeddings. These word embeddings are then fed into the decoder using FM-LSTM when processing data from different datasets. For GCC, the hidden layer dimension is set to 512, while for other datasets, it is set to 1024. The extracted visual concepts are matched with the sentences containing these visual concepts by constructing a concept-caption dictionary. For the visual concept k o 、k m , there is a corresponding set of sentences to match it. In the constructed target-concept dictionary, the number of word concepts is 18669 and the number of phrase concepts is 10612. For the evaluation score F, a Gaussian density function is selected to map words to a common space, where the mean and variance are set to 0 and 0.1. The batch size of the training set is set to 16, and the batch size of the test set is set to 32. In the calculation of the loss function, α is set to 0.01. For the optimizer, Adam is used, and the recommended hyperparameter settings are adopted for a maximum of 100 iterations when training G-DPPD, and the EarlyStopping technique is adopted, that is, when the validation loss does not decrease for 20 consecutive training iterations, the training is stopped early.

[0165] Finally, back propagation is used to optimize the model parameters θ and the distribution p of the generated sentences θ(y), by continuously training and optimizing the model’s ability to generate image descriptions, we eventually obtain an image caption generation model with good generalization ability.

[0166] S104: Input the image to be captioned into the trained unsupervised image caption generation model to generate a corresponding caption image.

[0167] The images for which subtitles are to be generated are used as a test set, and subtitles are generated for the input scene image samples using the optimal model parameters obtained through training in step S103 and the pre-trained model.

[0168] Those skilled in the art will appreciate that all or part of the steps in the method for implementing the above embodiments may be completed by instructing related hardware through a program, and the corresponding program may be stored in a computer-readable storage medium.

[0169] It should be noted that although the method operations of the above embodiments are described in a particular order in the accompanying drawings, this does not require or imply that the operations must be performed in this particular order, or that all of the illustrated operations must be performed to achieve the desired results. Rather, the depicted steps may be performed in a different order. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into a single step, and / or a single step may be broken down into multiple steps.

[0170] Example 2:

[0171] like Figure 5 As shown, this embodiment provides an unsupervised image caption generation system based on prior probability distribution, which includes a construction module 501, a sentence generation module 502, a training module 503 and a caption generation module 504, wherein:

[0172] A construction module 501 is configured to construct a training set based on the acquired image dataset and text dataset; the samples in the training set include image features of each image and corresponding pseudo-real sentences in the sentence space;

[0173] The sentence generation module 502 is configured to input image features into a feature-aggregated long short-term memory recurrent neural network in an unsupervised image caption generation model to generate a generated sentence, such that the generated sentence is globally aligned with the image features; and calculate a predicted distribution of the generated sentence and a prior probability distribution in a semantic space based on the generated sentence.

[0174] A training module 503 is configured to train the unsupervised image caption generation model using the training set, and update parameters in the unsupervised image caption generation model based on a total loss function to obtain a trained unsupervised image caption generation model; the total loss function includes a cosine loss function and a KL loss function. The cosine loss function compares the prior probability distribution of generated sentences in the semantic space with the predicted distribution of generated sentences to increase the diversity of generated sentences; the KL loss function compares the predicted distribution of generated sentences with the distribution of corresponding pseudo-real sentences to improve the accuracy of the unsupervised image caption generation model;

[0175] The subtitle generation module 504 is used to input the image for which subtitles are to be generated into the trained unsupervised image subtitle generation model to generate subtitles corresponding to the image.

[0176] The specific implementation of each module in this embodiment can be found in the above-mentioned embodiment 1, and will not be described one by one here; it should be noted that the system provided in this embodiment is only illustrated by the division of the above-mentioned functional modules. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure can be divided into different functional modules to complete all or part of the functions described above.

[0177] Example 3:

[0178] This embodiment provides a computer device, which can be a computer, such as Figure 6 As shown, a processor 602, a memory, an input device 603, a display 604, and a network interface 605 are connected via a system bus 601. The processor is used to provide computing and control capabilities. The memory includes a non-volatile storage medium 606 and an internal memory 607. The non-volatile storage medium 606 stores an operating system, a computer program, and a database. The internal memory 607 provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. When the processor 602 executes the computer program stored in the memory, the unsupervised image caption generation method based on prior probability distribution of the above-mentioned embodiment 1 is implemented as follows:

[0179] Constructing a training set based on the acquired image dataset and text dataset; the samples in the training set include image features of each image and corresponding pseudo-real sentences in the sentence space;

[0180] The image features are input into the feature-aggregated long short-term memory recurrent neural network in the unsupervised image caption generation model to obtain a generated sentence so that the generated sentence is globally aligned with the image features; based on the generated sentence, the predicted distribution of the generated sentence and the prior probability distribution in the semantic space are calculated;

[0181] Using the training set to train the unsupervised image caption generation model, updating the parameters of the unsupervised image caption generation model based on the total loss function, and obtaining a trained unsupervised image caption generation model;

[0182] The image to be captioned is input into the trained unsupervised image caption generation model to generate captions for the corresponding image.

[0183] Example 4:

[0184] This embodiment provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the method for generating unsupervised image captions based on a priori probability distribution according to the first embodiment is implemented as follows:

[0185] Constructing a training set based on the acquired image dataset and text dataset; the samples in the training set include image features of each image and corresponding pseudo-real sentences in the sentence space;

[0186] The image features are input into the feature-aggregated long short-term memory recurrent neural network in the unsupervised image caption generation model to obtain a generated sentence so that the generated sentence is globally aligned with the image features; based on the generated sentence, the predicted distribution of the generated sentence and the prior probability distribution in the semantic space are calculated;

[0187] The unsupervised image caption generation model is trained using the training set, and parameters in the unsupervised image caption generation model are updated based on a total loss function to obtain a trained unsupervised image caption generation model; the total loss function includes a cosine loss function and a KL loss function, the cosine loss function compares the prior probability distribution of the generated sentences in the semantic space with the predicted distribution of the generated sentences to increase the diversity of the generated sentences; the KL loss function compares the predicted distribution of the generated sentences with the distribution of the corresponding pseudo-real sentences to improve the accuracy of the unsupervised image caption generation model;

[0188] The image to be captioned is input into the trained unsupervised image caption generation model to generate captions for the corresponding image.

[0189] It should be noted that the computer-readable storage medium of the present embodiment may be a computer-readable signal medium or a computer-readable storage medium or any combination thereof. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or component, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection having one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0190] In summary, the unsupervised image subtitle generation method, system, terminal device and computer-readable storage medium based on prior probability distribution provided by the present invention are based on prior probability distribution and feature aggregation long short-term memory neural network, and are optimized for the negative sample bias problem that occurs in the existing MLE method for image subtitle generation tasks; the correlation between image features and corresponding words in the generated sentences is calculated, and the prior probability distribution and the correlation between image features and corresponding words are used as a gating mechanism to enhance the distinction between incorrect prediction differences, and the gated loss function is used to alleviate the negative sample bias problem; a feature aggregation long short-term memory recurrent neural network is adopted, and context information is utilized to align the generated sentences with the image features of global information; by constructing a concept-subtitle dictionary, word and phrase expressions can be jointly aligned with the image, so that the model can express more detailed image descriptions; the KL loss function is used to align the predicted distribution of the generated sentences with the distribution of pseudo-true sentences to increase the authenticity of the generated sentences.

[0191] The above is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes based on the technical solution and inventive concept of the present invention within the scope disclosed by the present invention, which falls within the scope of protection of the present invention.

Claims

1. An unsupervised image caption generation method based on prior probability distribution, characterized in that: The method comprises: Based on the acquired image dataset and text dataset, a training set is constructed, including: obtaining the visual concept of each image based on multiple targets of each image and the text dataset; constructing a target-concept dictionary from the visual concepts of all images; obtaining a concept-caption dictionary based on the target-concept dictionary and the text dataset; searching the visual concept of each image in the concept-caption dictionary to obtain pseudo-real sentences for all images and construct a sentence space; using the image features of each image and the corresponding pseudo-real sentences in the sentence space as samples in the training set; The image features are input into the feature-aggregated long short-term memory recurrent neural network in the unsupervised image caption generation model to obtain a generated sentence so that the generated sentence is globally aligned with the image features; based on the generated sentence, the predicted distribution of the generated sentence and the prior probability distribution in the semantic space are calculated; The unsupervised image caption generation model is trained using the training set, and parameters in the unsupervised image caption generation model are updated based on a total loss function to obtain a trained unsupervised image caption generation model; the total loss function includes a cosine loss function and a KL loss function, the cosine loss function compares the prior probability distribution of the generated sentences in the semantic space with the predicted distribution of the generated sentences to increase the diversity of the generated sentences; the KL loss function compares the predicted distribution of the generated sentences with the distribution of the corresponding pseudo-real sentences to improve the accuracy of the unsupervised image caption generation model; The image to be captioned is input into the trained unsupervised image caption generation model to generate captions for the corresponding image.

2. The unsupervised image caption generation method according to claim 1, characterized in that The calculation process of generating the prior probability distribution of a sentence in the semantic space is as follows: Calculate the cosine similarity between each word and every other word in the text dataset; Calculate the evaluation score based on the cosine similarity between words; According to the evaluation scores, the temperature mechanism is used to calculate the prior probability distribution of words in the semantic space; The prior probability distributions of all words in the generated sentence are merged to obtain the prior probability distribution of the generated sentence in the semantic space.

3. The unsupervised image caption generation method according to claim 1, wherein: The image features are input into the feature aggregation long short-term memory recurrent neural network in the unsupervised image caption generation model to obtain the generated sentences, including: The image features are input into the feature aggregation long short-term memory recurrent neural network, which is expressed as follows: Where y i represents the word embedding after the i-th iteration; σ represents the Sigmoid activation function; i and j both represent the iteration round, i is an odd number in [1,…r], and j is an even number in [1,…r]; r represents the total number of iterations; y -1 and h 0 Represent the initialized word embedding and hidden layer state respectively; h j and Represent the hidden layer state and image features after the jth iteration, Represents the image feature F img ; R j and Q i Both represent randomly initialized matrices; According to the word embedding after r iterations, the generated sentence is obtained.

4. The unsupervised image caption generation method according to claim 3, wherein: The calculation process of the predicted distribution of the generated sentence is: The iterated word embedding and hidden layer state are input into LSTM to obtain the predicted distribution p of the generated sentence θ (y): y i =LSTM(h r-1 ,y r ) p θ (y)=concat(y1,y2,…,y l ) Where h r-1 and y r They represent the hidden layer state and word embedding after iteration, and l represents the number of words in the sentence.

5. The unsupervised image caption generation method according to claim 1, wherein: The KL loss function is: Where r(y), p θ (y) denote the distribution of pseudo-true sentences and the predicted distribution of generated sentences, respectively.

6. The unsupervised image caption generation method according to claim 1, wherein: The cosine loss function is: Where, q(y), p θ (y) represents the prior probability distribution of the generated sentence in the semantic space and the predicted distribution of the generated sentence, g t represents the gate indicator, and l represents the total number of words in the generated sentence.

7. The unsupervised image caption generation method according to claim 1, wherein: The total loss function also includes a gated loss function to determine the importance of a word in the entire generated sentence when describing an image. The specific expression is: in, Where W g ∈R d×d and W t ∈R d×d Both represent linear transformation matrices, d represents the dimension of the word vector; V represents all targets extracted from the current image, g t represents the gate index, l represents the total number of words in the generated sentence; W t V represents the word vector of the target V after dimension transformation to get the same dimension as the word. Indicates the calculation of two word vectors W t Evaluation score between V.

8. The unsupervised image caption generation method according to any one of claims 1 to 7, characterized in that: The step of constructing a training set based on the acquired image dataset and text dataset further includes: Images in an image dataset are encoded to obtain image features and multiple targets for each image; the image features and targets respectively contain global information and local information of the image.

9. An unsupervised image caption generation system based on prior probability distribution, characterized in that: The system comprises: A construction module is used to construct a training set based on the acquired image dataset and text dataset, including: obtaining the visual concept of each image based on multiple targets of each image and the text dataset; constructing a target-concept dictionary from the visual concepts of all images; obtaining a concept-caption dictionary based on the target-concept dictionary and the text dataset; searching the visual concept of each image in the concept-caption dictionary to obtain pseudo-true sentences for all images to form a sentence space; and using the image features of each image and the corresponding pseudo-true sentences in the sentence space as samples in the training set; The sentence generation module is used to input image features into the feature-aggregated long short-term memory recurrent neural network in the unsupervised image caption generation model to generate sentences so that the generated sentences are globally aligned with the image features. Based on the generated sentences, the predicted distribution of the generated sentences and the prior probability distribution in the semantic space are calculated. A training module is configured to train an unsupervised image caption generation model using the training set, and to update parameters in the unsupervised image caption generation model based on a total loss function to obtain a trained unsupervised image caption generation model; the total loss function includes a cosine loss function and a KL loss function, wherein the cosine loss function compares the prior probability distribution of generated sentences in a semantic space with the predicted distribution of generated sentences to increase the diversity of generated sentences; and the KL loss function compares the predicted distribution of generated sentences with the distribution of corresponding pseudo-real sentences to improve the accuracy of the unsupervised image caption generation model; The subtitle generation module is used to input the image to be subtitled into the trained unsupervised image subtitle generation model to generate subtitles for the corresponding image.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the unsupervised image caption generation method according to any one of claims 1 to 8 is implemented.

Images