A method, device and storage medium for generating image description model

Abstract

The present application discloses a method, device and storage medium for generating an image description model, specifically, obtaining an image sample and a description sentence corresponding to the image sample; inputting the image sample and the description sentence into a first prediction model to generate a first distribution probability set; obtaining a second distribution probability set corresponding to a target vocabulary corresponding to each entity object in the image sample; assigning weight values ​​to the first distribution probability set and the second distribution probability set respectively, so as to obtain a third distribution probability set containing a third distribution probability corresponding to the target vocabulary at each decoding moment, and outputting a prediction sentence; based on the first distribution probability set, the second distribution probability set and the weight value, respectively determining a sentence entity object coverage loss function and a language sequence loss function, and determining a model loss function, so as to optimize the image description model until the image description model is generated. The embodiment of the present application improves the accuracy of image description by optimizing the image description model.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method, a device and a storage medium for generating an image description model. Background Art

[0002] Image description mainly describes the visual content of an image through natural language expression. In recent years, image description technology has been widely used, and automatic image description generation can be applied to image retrieval. Existing image description technology mainly extracts the feature vector of the input image data through Convolutional Neural Networks (CNN), and then uses Recurrent Neural Network (RNN) or Long Short-Term Memory (LSTM) to decode the extracted features, obtain the output sequence, and generate image description.

[0003] When training the generative model of image description, the existing image description technology usually uses images and corresponding description sentences as sample data sets for training. Therefore, the vocabulary is fixed, and the generated model is generally limited to the use of image data that has been trained, and its application in scenarios where new image data is used as input is limited. In addition, the recognition accuracy of the recognition object in the description sentence output by the above model is not high. Summary of the invention

[0004] An embodiment of the present application provides a method for generating an image description model, which improves the accuracy of the image description of an input image to be recognized by establishing an image description model.

[0005] The method includes:

[0006] Acquire an image sample and a description statement corresponding to the image sample;

[0007] Inputting the image sample and each target word in the description sentence into a pre-trained first prediction model in sequence, and generating a first distribution probability set of first distribution probabilities corresponding to each target word included in the description sentence at each decoding moment;

[0008] Obtain a second distribution probability set of second distribution probabilities corresponding to the target vocabulary corresponding to each entity object in the image sample at each decoding moment;

[0009] Assign weight values ​​to the first distribution probability corresponding to the target word in the first distribution probability set and the second distribution probability corresponding to the second distribution probability set, respectively, to obtain a third distribution probability set including the third distribution probability corresponding to the target word at each decoding moment, and output a predicted sentence;

[0010] Based on the first distribution probability set, the second distribution probability set and the weight value, respectively determine a sentence entity object coverage loss function and a language sequence loss function;

[0011] A model loss function is determined according to the sentence entity object coverage loss function and the language sequence loss function, and the image description model is optimized according to the model loss function until the image description model is generated.

[0012] Optionally, the generating method further comprises:

[0013] Inputting the image sample into the pre-trained convolutional neural network model, and extracting the feature vector of the image sample;

[0014] The feature vector and the target vocabulary corresponding to the description sentence at the last decoding moment are input into the long short-term memory network model.

[0015] Optionally, the generating method includes:

[0016] Inputting the image sample into an object detector pre-trained based on a preset image data set, and generating a set of prediction scores for each entity object contained in the image sample corresponding to the target vocabulary;

[0017] Based on the prediction score set and the hidden state of the long short-term memory network model at the current decoding moment, the target vocabulary is determined to be the correct semantic word of the entity object, and a second distribution probability set of the second distribution probability of the target vocabulary at each decoding moment is generated.

[0018] Optionally, the generating method includes:

[0019] Based on the first weight value assigned to the first distribution probability of the target vocabulary and the second weight value assigned to the second distribution probability, a third distribution probability set including the third distribution probability corresponding to the target vocabulary at each decoding moment is calculated, wherein the sum of the first weight value and the second weight value is 1.

[0020] Optionally, the generating method includes:

[0021] Determine a sentence entity object coverage loss function based on the sum of the second distribution probability corresponding to each target word at each decoding moment in the second distribution probability set and the assigned weight value, and the actual distribution probability of the target word in the description sentence;

[0022] A language sequence loss function is determined based on the prediction sentence and the description sentence for the image sample output by the first prediction model.

[0023] In another embodiment of the present invention, a method for obtaining an image description is provided, the method comprising:

[0024] Obtain an image to be recognized;

[0025] The image to be recognized is input into the image description model in each step of the above-mentioned method for generating an image description model to generate a description sentence of the image to be recognized.

[0026] In another embodiment of the present invention, a device for generating an image description model is provided, the device comprising:

[0027] An acquisition module, used for acquiring an image sample and a description statement corresponding to the image sample;

[0028] A first generating module, configured to sequentially input the image sample and each target word in the description sentence into a pre-trained first prediction model, and generate a first distribution probability set of first distribution probabilities corresponding to each target word included in the description sentence at each decoding moment;

[0029] A second generation module, used for obtaining a second distribution probability set of second distribution probabilities corresponding to the target words corresponding to each entity object in the image sample at each decoding moment;

[0030] A third generation module is used to assign weight values ​​to the first distribution probability corresponding to the target word in the first distribution probability set and the second distribution probability corresponding to the second distribution probability set, respectively, to obtain a third distribution probability set including the third distribution probability corresponding to the target word at each decoding moment, and output a prediction sentence;

[0031] A first determination module, configured to determine a sentence entity object coverage loss function and a language sequence loss function based on the first distribution probability set, the second distribution probability set and the weight value, respectively;

[0032] The second determination module is used to determine the model loss function according to the sentence entity object coverage loss function and the language sequence loss function, and optimize the image description model according to the model loss function until the image description model is generated.

[0033] Optionally, the generating device further includes:

[0034] An extraction module, used for inputting the image sample into the pre-trained convolutional neural network model and extracting a feature vector of the image sample;

[0035] An input module is used to input the feature vector and the target vocabulary corresponding to the description sentence at the last decoding moment into the long short-term memory network model.

[0036] Optionally, the second generating module includes:

[0037] An input unit, used to input the image sample into an object detector pre-trained based on a preset image data set, and generate a set of prediction scores of each entity object contained in the image sample corresponding to the target vocabulary;

[0038] A generating unit is used to determine the target vocabulary as the correct semantic word of the entity object based on the prediction score set and the hidden state of the long short-term memory network model at the current decoding moment, and generate a second distribution probability set of the second distribution probability of the target vocabulary at each decoding moment.

[0039] Optionally, the third generation module is further used for:

[0040] Based on the first weight value assigned to the first distribution probability of the target vocabulary and the second weight value assigned to the second distribution probability, a third distribution probability set including the third distribution probability corresponding to the target vocabulary at each decoding moment is calculated, wherein the sum of the first weight value and the second weight value is 1.

[0041] Optionally, the first determining module includes:

[0042] A first determining unit, configured to determine a sentence entity object coverage loss function based on the sum of the second distribution probability corresponding to each target word at each decoding moment in the second distribution probability set and the assigned weight value, and an actual distribution probability of the target word in the description sentence;

[0043] The second determining unit is used to determine a language sequence loss function based on the prediction sentence and the description sentence for the image sample output by the first prediction model.

[0044] In another embodiment of the present invention, a device for obtaining image description is provided, the device comprising:

[0045] An acquisition module, used for acquiring an image to be recognized;

[0046] A generating module is used to input the image to be identified into the image description model in each step of the above-mentioned method for generating an image description model, so as to generate a description sentence of the image to be identified.

[0047] In another embodiment of the present invention, a non-transitory computer-readable storage medium is provided, wherein the non-transitory computer-readable storage medium stores instructions, and when the instructions are executed by a processor, the processor executes each step in the above-mentioned method for generating an image description model.

[0048] In another embodiment of the present invention, a terminal device is provided, including a processor, wherein the processor is used to execute each step in the above-mentioned method for generating an image description model.

[0049] As can be seen above, based on the above embodiment, firstly, an image sample and a description sentence corresponding to the image sample are obtained. Secondly, each target word in the image sample and the description sentence is sequentially input into the pre-trained first prediction model, and a first distribution probability set of the first distribution probability corresponding to each target word contained in the description sentence at each decoding time is generated. At the same time, a second distribution probability set of the second distribution probability corresponding to each entity object in the image sample corresponding to the target word at each decoding time is obtained. Further, weight values ​​are assigned to the first distribution probability corresponding to the target word in the first distribution probability set and the second distribution probability corresponding to the second distribution probability set, respectively, to obtain a third distribution probability set containing the third distribution probability corresponding to the target word at each decoding time, and a prediction sentence is output. Then, based on the first distribution probability set, the second distribution probability set and the weight value, the sentence entity object coverage loss function and the language sequence loss function are determined respectively. Finally, according to the sentence entity object coverage loss function and the language sequence loss function, the model loss function is determined, and according to the model loss function, the image description model is optimized until the image description model is generated. The embodiment of the present application optimizes the image description model by determining the model loss function, and improves the accuracy of the image description. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0051] Figure 1 A schematic diagram showing a flow chart of a method for generating an image description model provided in Example 10 of the present application;

[0052] Figure 2 A schematic diagram showing a sentence entity object coverage loss function and a language sequence loss function generated in a method for generating an image description model provided in Example 20 of the present application is shown;

[0053] Figure 3 A schematic diagram showing a specific process in a method for generating an image description model in Example 30 provided in the present application;

[0054] Figure 4 A schematic flow chart of a method for obtaining an image description provided in Example 40 of the present application is shown;

[0055] Figure 5 A schematic diagram of a device for generating an image description model provided in Example 50 of the present application is shown;

[0056] Figure 6 A schematic diagram of a device for obtaining image description provided in Embodiment 60 of the present application is shown;

[0057] Figure 7 A schematic diagram of a terminal device provided in Example 70 of the present application is shown. DETAILED DESCRIPTION

[0058] In order to make the purpose, technical solutions and advantages of the present application more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and embodiments.

[0059] Based on the problems in the prior art, the embodiment of the present application provides a method for generating an image description model, by proposing a LSTM-P pointing mechanism based on LSTM for the object to be identified, firstly extracting the visual features of a given image through CNN, and secondly, inputting these visual features into LSTM, and LSTM outputting the first distribution probability of all target words contained in the word list. In addition, the entity objects in the input image are identified by a pre-trained object detector, and the prediction scores of each entity object are obtained, and the prediction scores of each entity object and the hidden state of the current decoding moment of LSTM are used as input in the copy layer, and the second distribution probability of directly using the target word as the description of the entity object is output. Finally, the first distribution probability and the second distribution probability output by the two models are input into the pointing mechanism, and the weights are dynamically assigned by the pointing mechanism, and the final probability distribution of each target word is output, and the corresponding sentence entity object coverage loss function and language sequence loss function are calculated to determine the model loss function of the entire image description model, and optimize the image description model. The image description model based on LSTM-P expands the range of target words that can be identified, while improving the accuracy of the description sentence of the image description.

[0060] The application field of this application is mainly in the field of computer technology, and is applicable to the fields of computer vision technology and natural language processing. Figure 1 FIG. 1 is a schematic diagram of Example 10 of a method for generating an image description model provided in an embodiment of the present application. The detailed steps are as follows:

[0061] S11, obtaining an image sample and a description statement corresponding to the image sample.

[0062] In this step, multiple image samples and description statements of the image samples are obtained as multiple pairs of input data, such as Figure 2 The image sample shown has a description sentence corresponding to "a large dog laying on a blanket on a couch", where the description sentence corresponding to the image sample needs to be pre-set. At the same time, the description sentence is a correct description of the content and meaning of the image sample.

[0063] S12, inputting the image sample and the description sentence into a pre-trained first prediction model, and generating a first distribution probability set of first distribution probabilities corresponding to each target word included in the description sentence at each decoding moment.

[0064] In this step, the first prediction model is mainly composed of CNN and LSTM. The CNN model is mainly used to extract the feature vector of the image sample, input the image sample into the CNN model, and output the feature vector of the image sample. Specifically, the image sample is input into the CNN, and the pre-trained network VGG-16 or ResNet is used, and the network layer other than the softmax classifier that outputs the category score is used to extract the feature vector of the image sample. CNN is usually called an encoder, which encodes a large amount of information contained in the image sample into a feature vector, processes it, and uses the extracted feature vector as the initial input of the subsequent LSTM. The main function of LSTM is to process the feature vector by decoding and convert it into natural language. Among them, the image sample and the corresponding description sentence are input into the LSTM together. Specifically, the features extracted by CNN and the target vocabulary in the description sentence at the previous decoding moment are input into the LSTM, and the LSTM generates prediction sentences one by one according to the order of the words in the sentence. Here, the LSTM finally outputs the first distribution probability set of the first distribution probability of the target vocabulary at each decoding moment according to the order of the words in the sentence. As Figure 2 As shown, It is a first distribution probability set consisting of predicted first distribution probabilities obtained after the first prediction model including CNN and LSTM is trained on the image sample and the corresponding description sentence input sentence.

[0065] S13, obtaining a second distribution probability set of second distribution probabilities corresponding to target words corresponding to each entity object in the image sample at each decoding moment.

[0066] In this step, the image sample is first input into a pre-trained object detector, which identifies each entity object that may be contained in the image sample. The object detector is pre-trained based on an image dataset. The image dataset contains a large amount of image data and corresponding descriptive words. The object detector is used to identify each entity object in the image sample, and a set of prediction scores of the predicted scores of each entity object that may be the corresponding target vocabulary is obtained. Figure 2 As shown in the figure, the object detector is represented as Object Leaner. After the image sample is input into Object Leaner, the prediction score of each entity object contained in the image sample may be the target word is output. Figure 2 When the image sample is shown, Object Leaner outputs the predicted probability that each entity object may be the relevant target word, that is, the prediction score, such as dog: 1.00, couch; 0.21, bed: 0.13, blanket: 0.12.

[0067] Secondly, after obtaining the predicted score set for each entity object, the hidden state of the LSTM at the current decoding time and the obtained predicted score set are simultaneously input into the copying layer. In the copying layer, the target vocabulary of the recognized entity object is determined as the correct semantic word of each entity object in the input image sample, and a second distribution probability set of the second distribution probability corresponding to the target vocabulary at each decoding time is generated.

[0068] S14, respectively assigning weights to the first distribution probability corresponding to the target word in the first distribution probability set and the second distribution probability corresponding to the target word in the second distribution probability set, so as to obtain a third distribution probability set including the third distribution probability corresponding to the target word at each decoding moment.

[0069] In this step, after obtaining the first distribution probability set generated by the first prediction model and the second distribution probability set generated in the Copying Layer, the above two distribution probability sets are input into the created pointing mechanism model. The Pointing Mechanism dynamically assigns weights p to the two probability distribution sets. t and 1-p t , and calculate the weighted sum to determine the third distribution probability set Pr finally generated containing the third distribution probabilities of each target word in the prediction sentence t .

[0070] S15, determining a sentence entity object coverage loss function and a language sequence loss function based on the first distribution probability set, the second distribution probability set and the weight value.

[0071] In this step, because the sentence finally generated by the first prediction model is composed of each word generated one by one in the order of the target words in the description sentence. Therefore, the first distribution rate of the sentence finally generated by the first prediction model is equal to the sum of the logarithmic probabilities of each word being generated in the sentence to represent the language sequence loss function. By minimizing the language sequence loss function, the contextual grammatical dependency of each target word in the final output sentence is guaranteed, so that the generated sentence is grammatically fluent.

[0072] In addition, multi-label classification is performed on the target vocabulary for the final semantic consistency of the predicted sentence. Specifically, in the second distribution probability set generated by the above-mentioned replication layer, the sum of the product of the second distribution probability and the corresponding weight value assigned at each decoding moment is calculated to obtain the word-out probability of each target vocabulary at each decoding moment. Finally, based on the word-out probability, the sentence entity object coverage loss function is obtained. At the same time, by minimizing the sentence entity object coverage loss function, it is ensured that each target vocabulary in the final output sentence is consistent with the correct meaning of the entity object in the corresponding image sample, thereby improving the accuracy of each target vocabulary in the generated sentence.

[0073] S16, determining the model loss function according to the sentence entity object coverage loss function and the language sequence loss function, and optimizing the image description model according to the model loss function until the image description model is generated.

[0074] In this step, after determining the semantic consistency loss function and language sequence loss function, the optimization parameters are selected to determine the model loss function. At the same time, the LSTM-P image description model is continuously modified according to the model loss function until the optimal image description model is generated.

[0075] Based on the above embodiment of the present application, firstly, an image sample and a description sentence corresponding to the image sample are obtained, and then, each target word in the image sample and the description sentence is sequentially input into a pre-trained first prediction model, and a first distribution probability set of the first distribution probability corresponding to each target word contained in the description sentence at each decoding time is generated, and at the same time, a second distribution probability set of the second distribution probability corresponding to the target words corresponding to each entity object in the image sample at each decoding time is obtained, and then, weight values ​​are assigned to the first distribution probability corresponding to the target words in the first distribution probability set and the second distribution probability corresponding to the second distribution probability set, respectively, to obtain a third distribution probability set containing the third distribution probability corresponding to the target words at each decoding time, and a prediction sentence is output, and based on the first distribution probability set, the second distribution probability set and the weight value, the sentence entity object coverage loss function and the language sequence loss function are determined respectively, and finally, according to the sentence entity object coverage loss function and the language sequence loss function, the model loss function is determined, and according to the model loss function, the image description model is optimized until the image description model is generated. The embodiment of the present application designs an image description model based on LSTM-P to realize the image description of the image to be recognized. Through the pointing mechanism, the correct target vocabulary corresponding to the identified entity object is dynamically integrated into the output prediction sentence. The LSTM-P-based image description model first fully explores the contextual relevance between the generated target vocabulary through the conventional CNN&RNN language model. At the same time, the object trainer trained according to the identified data set is used to identify the entity objects in the input image sample. In addition, the vocabulary is directly copied from the target vocabulary corresponding to the identified entity object through the Copying Layer. Furthermore, the pointing mechanism generates prediction sentences for the first prediction model composed of CNN and LSTM, and directly copies the vocabulary from the identified target vocabulary, and dynamically assigns weight values ​​to the target vocabulary through these two methods. The Pointing Mechanism can identify the best time to copy the object sample to the corresponding description sentence in the current context. And by reducing the error rate of the sentence entity object coverage loss and language sequence loss in the output prediction sentence, the LSTM-P-based image description model is globally trained. In addition, the lack of coverage at the sentence level is used as feedback to further improve the coverage of each entity object in the image sample by the output prediction sentence.

[0076] like Figure 3 As shown, it is a schematic diagram of a specific process of a method for generating an image description model in Example 30 provided by the present application. The detailed process of the specific process is as follows:

[0077] S301, obtaining an image sample and a description statement corresponding to the image sample.

[0078] S302: Input the image sample and each target word in the corresponding description sentence into a pre-trained first prediction model in sequence.

[0079] Here, the first prediction training model includes a convolutional neural network (CNN) and a long short-term memory (LSTM) network model. Specifically, the image sample is input into the pre-trained convolutional neural network model to extract the feature vector of the image sample. Furthermore, the feature vector and the target vocabulary corresponding to the description sentence at the last decoding moment are input into the long short-term memory network model.

[0080] S303: Generate a first distribution probability set.

[0081] In this step, the first prediction model outputs the first distribution probability set corresponding to the first distribution probability of each target word contained in the description sentence at each decoding time. For an image sample I, a sentence S is used to describe the image sample I. Where S = {ω1, ω2, ..., ω Ns} by N s words. Dv and Wt∈R Dw denote the Dv-dimensional visual features and the Dw-dimensional text features of the t-th word in the sentence S, respectively. In the first prediction model stage, the target vocabulary ω t+1 The first distribution probability of formula 1 is expressed as follows:

[0082]

[0083] Among them, h t is the output state of LSTM at decoding time t.

[0084] S304: Input the image sample into the object detector and generate a set of prediction scores.

[0085] Here, the image sample is input into an object detector pre-trained based on a preset image data set, and a set of prediction scores corresponding to the target vocabulary for each entity object contained in the image sample is generated.

[0086] S305, generating a second distribution probability set according to the prediction score set and the hidden state of the LSTM at the current moment.

[0087] In this step, the image sample is first input into an object detector pre-trained based on a preset image data set, and a set of prediction scores corresponding to the prediction scores of the target vocabulary for each entity object contained in the image sample is generated. Further, based on the set of prediction scores and the hidden state of the long short-term memory network model at the current decoding moment, the target vocabulary is determined to be the correct semantic word of the entity object, and a second distribution probability set of the second distribution probability of the target vocabulary at each decoding moment is generated. The calculation formula 2 of the second distribution probability is as follows:

[0088]

[0089] Among them, h t is the output state of LSTM at decoding time t, I c is the output of the object detector, and is the transformation matrix.

[0090] S306, the pointing mechanism dynamically allocates weight values ​​to the first distribution probability set and the second distribution probability set.

[0091] Here, after the first distribution probability set and the second distribution probability set are input into the pointing mechanism, the pointing mechanism dynamically allocates weight values. The specific calculation process is reflected by the following calculation formula 3:

[0092] Pt=σ(G s w t +G h h t +b p ) (3)

[0093] in, and They represent the transformation matrix of text features and the transformation matrix of LSTM model output, respectively. p is the bias vector.

[0094] S307: Determine a third distribution probability set according to the assigned weight values.

[0095] Here, based on the first weight value assigned to the first distribution probability of the target vocabulary and the second weight value assigned to the second distribution probability, a third distribution probability set including the third distribution probability corresponding to the target vocabulary at each decoding time is calculated, wherein the sum of the first weight value and the second weight value is 1. t+1 The final distribution probability is obtained by assigning the weight value p calculated by formula 3 to the above formula 1 and formula 2. t It turns out that:

[0096]

[0097]

[0098] in, and Represent the first weight value and the second weight value respectively, and φ represents a normalization function, such as Softmax.

[0099] S308, determining a language sequence loss function.

[0100] In this step, the language sequence loss function is determined based on the prediction sentence and description sentence of the image sample output by the first prediction model. Since the first prediction model generates a corresponding target word at each decoding moment, the chain rule is used to model the distribution probability of all words. The negative logarithmic probability of the entire output prediction sentence is equal to the sum of the negative logarithmic first distribution probabilities of all target words, which can be expressed as follows:

[0101]

[0102] By minimizing the language sequence loss function shown in Formula 5, the contextual dependencies between the target words in the output prediction sentence can be made more accurate.

[0103] S309, determining the statement entity object coverage loss function.

[0104] Here, based on the second distribution probability of the target vocabulary represented by the aforementioned formula 2 and the weight value determined by formula 3, the coverage of each target vocabulary included in the output prediction sentence is further calculated. The specific calculation formula is as follows:

[0105]

[0106] Among them, ω oi represents the target word to be copied, σ is a nonlinear activation function, such as the Sigmoid function, such as Figure 2 As shown on the right, at each decoding moment, the product of the second distribution probability and the weight value of each target word is obtained, and the sum of the products of each target word in the sentence, that is, the weighted sum, is obtained. The coverage rate of each target word is determined. Further, the sentence entity object coverage loss function is determined according to the following formula:

[0107]

[0108] Formula 7 is the final sentence entity object coverage loss function. By minimizing the sentence entity object coverage loss function, the description sentence is closer to the true meaning of the image sample.

[0109] S310, determining a model loss function.

[0110] Here, the model loss function is determined according to the language sequence loss function determined by Formula 5 and the sentence entity object coverage loss function determined by Formula 7. The specific model loss function is expressed as follows:

[0111]

[0112] Among them, λ is a weight parameter. The purpose of the model loss function is to optimize the image description model so that the description sentence predicted by the image description model finally conforms to the language logic and can accurately describe the input image to be predicted.

[0113] S311, optimize the image description model and generate the model.

[0114] The embodiment of the present application implements a method for generating an image description model based on the above steps. By constructing an image description model based on the LSTM-P structure, a method for image description is implemented. At the same time, a configuration method for vocabulary expansion and a training method for a hybrid network of an image description model are implemented. Among them, based on an available known image data set, an object detector is pre-trained, and then a pointing mechanism is constructed to balance the first distribution probability of vocabulary recognition results generated by a first prediction model composed of CNN and LSTM, and the target vocabulary directly copied and extracted from the data recognized by the object detector. In addition, the coverage of the target vocabulary in the predicted sentence is further explored, thereby improving the accuracy of the image description. Experiments based on the COCO image description dataset and the ImageNet image recognition dataset fully verified the model and analysis results.

[0115] In addition, if Figure 4 A method for obtaining an image description provided in Example 40 of the present application is shown, in which the image to be identified is input into an image description model in each step of the above-mentioned method for generating an image description model to generate a description statement of the image to be identified.

[0116] S41, obtaining an image to be recognized.

[0117] S42, and input the image to be recognized into the image description model to generate a description sentence of the image to be recognized.

[0118] Based on the same inventive concept, embodiment 50 of the present application further provides a device for generating an image description model, wherein: Figure 5 As shown, the generating device comprises:

[0119] An acquisition module 51 is used to acquire an image sample and a description statement corresponding to the image sample;

[0120] A first generating module 52 is used to input the image sample and each target word in the description sentence into a pre-trained first prediction model in sequence, and generate a first distribution probability set of first distribution probabilities corresponding to each target word included in the description sentence at each decoding moment;

[0121] A second generation module 53 is used to obtain a second distribution probability set of second distribution probabilities corresponding to target words corresponding to each entity object in the image sample at each decoding moment;

[0122] A third generating module 54 is used to assign weight values ​​to the first distribution probability corresponding to the target word in the first distribution probability set and the second distribution probability corresponding to the target word in the second distribution probability set, respectively, to obtain a third distribution probability set including the third distribution probability corresponding to the target word at each decoding moment, and output a predicted sentence;

[0123] A first determination module 55, used to determine a sentence entity object coverage loss function and a language sequence loss function respectively based on the first distribution probability set, the second distribution probability set and the weight value;

[0124] The second determination module 56 is used to determine the model loss function according to the sentence entity object coverage loss function and the language sequence loss function, and optimize the image description model according to the model loss function until the image description model is generated.

[0125] In this embodiment, the specific functions and interaction methods of the acquisition module 51, the first generation module 52, the second generation module 53, the third generation module 54, the first determination module 55 and the second determination module 56 can be found in Figure 1 The description of the corresponding embodiments will not be repeated here.

[0126] Optionally, the generating device further comprises:

[0127] An extraction module 57, used to input the image sample into a pre-trained convolutional neural network model and extract a feature vector of the image sample;

[0128] The input module 58 is used to input the feature vector and the target vocabulary corresponding to the description sentence at the last decoding moment into the long short-term memory network model.

[0129] Optionally, the second generating module 53 includes:

[0130] An input unit, used to input the image sample into an object detector pre-trained based on a preset image data set, and generate a set of prediction scores for each entity object contained in the image sample corresponding to the target vocabulary;

[0131] A generating unit is used to determine the target word as the correct semantic word of the entity object based on the prediction score set and the hidden state of the long short-term memory network model at the current decoding moment, and generate a second distribution probability set of the second distribution probability of the target word at each decoding moment.

[0132] Optionally, the third generating module 54 is further used for:

[0133] Based on a first weight value assigned to a first distribution probability of a target word and a second weight value assigned to a second distribution probability, a third distribution probability set including a third distribution probability corresponding to the target word at each decoding moment is calculated, wherein the sum of the first weight value and the second weight value is 1.

[0134] Optionally, the first determining module 55 includes:

[0135] A first determining unit is used to determine a sentence entity object coverage loss function based on the sum of the second distribution probability corresponding to each target word at each decoding moment in the second distribution probability set and the assigned weight value, and the actual distribution probability of the target word in the description sentence;

[0136] The second determination unit is used to determine the language sequence loss function based on the prediction sentence and the description sentence of the image sample output by the first prediction model.

[0137] Based on the same inventive concept, embodiment 60 of the present application further provides a device for obtaining image description, wherein: Figure 6 As shown, the device comprises:

[0138] An acquisition module 61 is used to acquire an image to be recognized;

[0139] The generating module 62 is used to input the image to be identified into the image description model to generate a description sentence of the image to be identified.

[0140] In this embodiment, the specific functions and interaction methods of the acquisition module 61 and the generation module 62 can be found in Figure 4 The description of the corresponding embodiments will not be repeated here.

[0141] like Figure 7 As shown, another embodiment 70 of the present application further provides a terminal device, including a processor 70, wherein the processor 70 is used to execute the steps of the above-mentioned method.

[0142] from Figure 7 It can also be seen that the terminal device provided in the above embodiment also includes a non-transitory computer-readable storage medium 71, on which a computer program is stored. When the computer program is executed by the processor 70, the steps of the above-mentioned method for generating an image description model are executed.

[0143] Specifically, the storage medium can be a general storage medium, such as a mobile disk, a hard disk, and FLASH, etc. When the computer program on the storage medium is run, the above-mentioned method for generating an image description model can be executed.

[0144] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present application, which are used to illustrate the technical solutions of the present application, rather than to limit them. The protection scope of the present application is not limited thereto. Although the present application is described in detail with reference to the above-described embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-described embodiments within the technical scope disclosed in the present application, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims

1. A method for generating an image description model, characterized in that: include: Acquire an image sample and a description statement corresponding to the image sample; Inputting the image sample and each target word in the description sentence into a pre-trained first prediction model in sequence, and generating a first distribution probability set of first distribution probabilities corresponding to each target word included in the description sentence at each decoding moment; Obtain a second distribution probability set of second distribution probabilities corresponding to the target vocabulary corresponding to each entity object in the image sample at each decoding moment; Inputting the first distribution probability set of the second distribution probability corresponding to the target vocabulary at each decoding moment and the second distribution probability set of the second distribution probability corresponding to the target vocabulary at each decoding moment into the created pointing mechanism model, the pointing mechanism model assigns a first weight value to the first distribution probability of the target vocabulary and assigns a second weight value to the second distribution probability, so as to obtain a third distribution probability set including the third distribution probability corresponding to the target vocabulary at each decoding moment, and outputting a prediction sentence; Based on the first distribution probability set, the second distribution probability set and the weight value, respectively determine a semantic consistency loss function and a grammatical coherence loss function; Determining a model loss function according to the semantic consistency loss function and the grammatical coherence loss function, and optimizing the image description model according to the model loss function until the image description model is generated; The steps of respectively determining the semantic consistency loss function and the grammatical coherence loss function include: Determine a semantic consistency loss function based on the sum of the second distribution probability corresponding to each target word at each decoding moment in the second distribution probability set and the assigned weight value, and the actual distribution probability of the target word in the description sentence; A grammatical coherence loss function is determined based on the predicted sentence for the image sample and the description sentence output by the first prediction model.

2. The generation method according to claim 1, characterized in that: The first prediction model is composed of a convolutional neural network model and a long short-term memory network model, and between the step of sequentially inputting the image sample and each target word in the description sentence into the pre-trained first prediction model and the step of generating a first distribution probability set of first distribution probabilities corresponding to each target word included in the description sentence at each decoding moment, the generation method further comprises: Inputting the image sample into the pre-trained convolutional neural network model, and extracting the feature vector of the image sample; The feature vector and the target vocabulary corresponding to the description sentence at the last decoding moment are input into the long short-term memory network model.

3. The generation method according to claim 2, characterized in that: The step of obtaining a second distribution probability set of second distribution probabilities corresponding to the target words corresponding to each entity object in the image sample at each decoding moment comprises: Inputting the image sample into an object detector pre-trained based on a preset image data set, and generating a set of prediction scores for each entity object contained in the image sample corresponding to the target vocabulary; Based on the prediction score set and the hidden state of the long short-term memory network model at the current decoding moment, the target vocabulary is determined to be the correct semantic word of the entity object, and a second distribution probability set of the second distribution probability of the target vocabulary at each decoding moment is generated.

4. The generation method according to claim 1, characterized in that: The step of obtaining a third distribution probability set including the third distribution probability corresponding to the target vocabulary at each decoding moment comprises: Based on a first weight value assigned to the first distribution probability of the target vocabulary and a second weight value assigned to the second distribution probability, a third distribution probability set including a third distribution probability corresponding to the target vocabulary at each decoding moment is calculated, wherein the sum of the first weight value and the second weight value is 1.

5. A method for obtaining an image description, characterized in that: Based on the method according to any one of the steps of claims 1 to 4: Obtain an image to be recognized; The image to be recognized is input into an image description model to generate a description sentence of the image to be recognized.

6. A device for generating an image description model, characterized in that: include: An acquisition module, used for acquiring an image sample and a description statement corresponding to the image sample; A first generating module, configured to sequentially input the image sample and each target word in the description sentence into a pre-trained first prediction model, and generate a first distribution probability set of first distribution probabilities corresponding to each target word included in the description sentence at each decoding moment; A second generation module, used for obtaining a second distribution probability set of second distribution probabilities corresponding to the target words corresponding to each entity object in the image sample at each decoding moment; A third generating module is used to input the first distribution probability set of the second distribution probability corresponding to the target vocabulary at each decoding moment and the second distribution probability set of the second distribution probability corresponding to the target vocabulary at each decoding moment into the created pointing mechanism model, the pointing mechanism model assigns a first weight value to the first distribution probability of the target vocabulary and assigns a second weight value to the second distribution probability, so as to obtain a third distribution probability set including the third distribution probability corresponding to the target vocabulary at each decoding moment, and output a prediction sentence; A first determination module, configured to determine a semantic consistency loss function and a grammatical coherence loss function respectively based on the first distribution probability set, the second distribution probability set and the weight value; A second determination module is used to determine a model loss function according to the semantic consistency loss function and the grammatical coherence loss function, and optimize the image description model according to the model loss function until the image description model is generated; The first determining module comprises: A first determining unit, configured to determine a semantic consistency loss function based on the sum of the second distribution probability corresponding to each target word at each decoding moment in the second distribution probability set and the assigned weight value, and an actual distribution probability of the target word in the description sentence; The second determining unit is used to determine a grammatical coherence loss function based on the prediction sentence and the description sentence for the image sample output by the first prediction model.

7. The generating device according to claim 6, characterized in that The first prediction model is composed of a convolutional neural network model and a long short-term memory network model, and between the step of sequentially inputting the image sample and each target word in the description sentence into the pre-trained first prediction model and the step of generating a first distribution probability set of first distribution probabilities corresponding to each target word included in the description sentence at each decoding moment, the generating device further includes: An extraction module, used for inputting the image sample into the pre-trained convolutional neural network model and extracting a feature vector of the image sample; An input module is used to input the feature vector and the target vocabulary corresponding to the description sentence at the last decoding moment into the long short-term memory network model.

8. The generating device according to claim 7, characterized in that The second generation module comprises: An input unit, used to input the image sample into an object detector pre-trained based on a preset image data set, and generate a set of prediction scores of each entity object contained in the image sample corresponding to the target vocabulary; A generating unit is used to determine the target vocabulary as the correct semantic word of the entity object based on the prediction score set and the hidden state of the long short-term memory network model at the current decoding moment, and generate a second distribution probability set of the second distribution probability of the target vocabulary at each decoding moment.

9. The generating device according to claim 6, characterized in that: The third generation module is also used for: Based on a first weight value assigned to the first distribution probability of the target vocabulary and a second weight value assigned to the second distribution probability, a third distribution probability set including a third distribution probability corresponding to the target vocabulary at each decoding moment is calculated, wherein the sum of the first weight value and the second weight value is 1.

10. A device for obtaining image description, characterized in that: Based on the device according to any one of the steps of claims 6 to 9: An acquisition module, used for acquiring an image to be recognized; The generation module is used to input the image to be recognized into the image description model to generate a description sentence of the image to be recognized.

11. A non-transitory computer-readable storage medium, characterized in that The non-transitory computer-readable storage medium stores instructions, which, when executed by a processor, cause the processor to perform each step in the method for generating an image description model as described in any one of claims 1 to 4.

12. A terminal device, characterized in that: The method comprises a processor, wherein the processor is used to execute each step in the method for generating an image description model as described in any one of claims 1 to 4.

Images