Training methods, image description methods and models

Abstract

The present disclosure relates to a training method, an image description method and model, an electronic device, and a computer storable medium, and relates to the field of computer technology. The training method for an image description model includes: obtaining a training image and its actual image description; generating an actual object feature sequence based on the training image and its actual image description, wherein the actual object feature sequence includes actual object features arranged in the order in which the objects appear in the actual image description; generating a predicted image description based on the training image using the image description model; training the image description model based on the predicted image description, the actual object feature sequence, and the actual image description to obtain a trained image description model, wherein the actual object feature sequence and the actual image description are supervisory signals for training the image description model. According to the present disclosure, the accuracy of image description can be improved.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a training method, an image description method and model, an electronic device, and a computer storable medium. Background Art

[0002] Image description technology is one of the basic topics in the field of computer vision and language. Given a data pair consisting of an image and a sentence describing its content, the goal of image description is to use these data pairs to learn an image description model that can automatically generate a grammatically correct sentence for the image as the image description of the image. The sentence can encompass the semantic content of the image and describe these semantic contents in an appropriate order.

[0003] In the related art, the training image is input into the image description model for encoding and decoding to obtain the predicted image description, and then the image description model is trained according to the actual image description of the training image and the predicted image description to obtain the trained image description model. During the training process, the actual image description of the training image serves as a supervisory signal for the training image description model. Summary of the invention

[0004] In the related art, the trained image description model will introduce object features that are irrelevant to the image description during the encoding and decoding process, thereby introducing a large amount of noise, resulting in low accuracy of the image description of the trained image description model.

[0005] In view of the above technical problems, the present disclosure proposes a solution to improve the accuracy of image description.

[0006] According to a first aspect of the present disclosure, there is provided a training method for an image description model, comprising: acquiring a training image and an actual image description thereof; generating an actual object feature sequence based on the training image and the actual image description thereof, wherein the actual object feature sequence comprises actual object features arranged in the order in which the objects appear in the actual image description; generating a predicted image description based on the training image using the image description model; training the image description model based on the predicted image description, the actual object feature sequence and the actual image description to obtain a trained image description model, wherein the actual object feature sequence and the actual image description serve as supervisory signals for training the image description model.

[0007] In some embodiments, generating a predicted image description based on the training image using the image description model includes: generating a predicted object feature sequence based on the training image using the image description model, the predicted object feature sequence including predicted object features arranged in sequence; generating a predicted image description based on the predicted object feature sequence using the image description model, wherein the order of appearance of objects in the predicted image description is consistent with the order of objects corresponding to the object features in the predicted object feature sequence, and the predicted object feature sequence is used to train the image description model together with the predicted image description, the actual object feature sequence and the actual image description.

[0008] In some embodiments, training the image description model includes: determining a first loss value based on the predicted object feature sequence and the actual object feature sequence; determining a second loss value based on the predicted image description and the actual image description; and training the image description model based on the first loss value and the second loss value.

[0009] In some embodiments, determining a first loss value includes: for each predicted object feature in the predicted object feature sequence, determining from the actual object feature sequence an actual object feature having the same position as each predicted object feature in the predicted object feature sequence as a positive sample object feature; for each predicted object feature, determining other actual object features in the actual object feature sequence except the positive sample feature as negative sample object features; and determining the first loss value based on the positive sample object features and the negative sample object features corresponding to all predicted object features.

[0010] In some embodiments, determining the first loss value based on the positive sample object features and the negative sample object features corresponding to all predicted object features includes: determining the similarity between each predicted object feature and the corresponding positive sample object feature as the first similarity; determining the similarity between each predicted object feature and the corresponding negative sample feature as the second similarity; and determining the first loss value based on the first similarity and the second similarity.

[0011] In some embodiments, the first loss value is negatively correlated with the first similarity, and the first loss value is positively correlated with the second similarity.

[0012] In some embodiments, determining the second loss value includes: determining the second loss value using a cross entropy loss function based on the predicted image description and the actual image description.

[0013] In some embodiments, training the image description model according to the first loss value and the second loss value includes: training the image description model according to the sum of the first loss value and the second loss value.

[0014] In some embodiments, generating a predicted object feature sequence based on the training image and utilizing the image description model includes: encoding the object in the training image to obtain target visual features of the object in the training image, wherein the target visual features include target object features and target image features; generating the predicted object feature sequence based on the target visual features of the object in the training image; wherein generating a predicted image description based on the predicted object feature sequence and utilizing the image description model includes: generating the predicted image description based on the target visual features of the object in the training image and the predicted object feature sequence.

[0015] In some embodiments, the image description model includes a sequence generation model, which includes a first attention model and a time series prediction model. Generating the predicted object feature sequence according to the target visual features of the object in the training image includes: using the first attention model to perform attention calculation on the target visual features of the object in the training image; and using the time series prediction model to generate the predicted object feature sequence according to the attention calculation result of the first attention model.

[0016] In some embodiments, the time series prediction model is a two-layer long short-term memory network LSTM model, and the two-layer LSTM model includes a first LSTM model located in the first layer and a second LSTM model located in the second layer, and the input of the second LSTM model includes the attention calculation result of the first attention model and the output of the first LSTM model.

[0017] In some embodiments, the input of the first LSTM model includes target image features of the training image.

[0018] In some embodiments, the image description model includes a decoding model, and generating the predicted image description according to the target visual features of the object in the training image and the predicted object feature sequence includes: generating the predicted image description according to the target visual features of the object in the training image and the predicted object feature sequence using the decoding model.

[0019] In some embodiments, the decoding model includes a second attention model and a third attention model, and the input of the third attention model includes the output of the second attention model, the target visual features of the objects in the training image, and the predicted object feature sequence.

[0020] In some embodiments, based on the target visual features of the object in the training image and the predicted object feature sequence, the decoding model is used to generate the predicted image description, including: using the second attention model to process its input to obtain the output of the second attention model; using the third attention model to linearly transform the output of the second attention model to obtain the first vector matrix in the third attention model; linearly transforming the target object features of the object in the training image and the predicted object feature sequence respectively to obtain the second vector matrix and the third vector matrix in the third attention model corresponding to the target visual features of the object in the training image and the predicted object feature sequence respectively; generating the predicted image description based on the first vector matrix, the second vector matrix and the third vector matrix in the third attention model.

[0021] In some embodiments, generating the predicted image description according to the first vector matrix, the second vector matrix and the third vector matrix in the third attention model includes: determining a first attention score matrix according to the first vector matrix in the third attention model and the second vector matrix corresponding to the target visual features of the object in the training image; determining a first output of the third attention model according to the first attention score matrix and the third vector matrix corresponding to the target visual features of the object in the training image; determining a second attention score matrix according to the first vector matrix in the third attention model and the second vector matrix corresponding to the predicted object feature sequence; determining a second output of the third attention model according to the first attention score matrix and the third vector matrix corresponding to the predicted object feature sequence; and generating the predicted image description according to the first output and the second output.

[0022] In some embodiments, the image description model includes an encoding model, which includes a target detection model and a feature fusion model. Encoding the objects in the training image to obtain target visual features of the objects in the training image includes: using the target detection model to perform target detection on the training image to obtain initial object features of the objects in the training image; obtaining initial image features of the training image; using the feature fusion model to fuse the initial object features of the objects in the training image with the initial image features to obtain target visual features of the objects in the training image, wherein the target visual features include target object features of the objects and target image features of the training image.

[0023] In some embodiments, the feature fusion model includes an encoder for fusing initial object features of objects in the training image with the initial image features.

[0024] In some embodiments, generating an actual object feature sequence based on the training image and its actual image description includes: obtaining a word belonging to a preset part of speech in the actual image description as a target word; obtaining a target object feature of an object corresponding to the target word in the training image as an actual object feature; and sorting the actual object features in the order in which the target word appears in the actual image description to obtain the actual object feature sequence.

[0025] According to a second aspect of the present disclosure, there is provided an image description method based on a trained image description model obtained by the training method described in any of the above embodiments, comprising: obtaining an image to be described; and generating an image description corresponding to the image to be described using the trained image description model according to the image to be described.

[0026] According to a third aspect of the present disclosure, there is provided an image description model, comprising: an encoding model and a decoding model, wherein the encoding model is configured to obtain a training image and an actual image description thereof, generate an output of the encoding model based on the training image, and adjust training parameters in the encoding model based on the predicted image description, the actual object feature sequence and the actual image description output by the decoding model to train the encoding model; the decoding model is configured to generate a predicted image description based on the output of the encoding model, and adjust training parameters in the decoding model based on the predicted image description, the actual object feature sequence and the actual image description to train the decoding model; wherein the actual object feature sequence is generated based on the training image and the actual image description thereof, the actual object feature sequence comprises actual object features arranged in the order of appearance of objects in the actual image description, and the actual object feature sequence and the actual image description are supervisory signals for training the image description model.

[0027] In some embodiments, the image description model also includes: a sequence generation model, configured to generate a predicted object feature sequence based on the output of the encoding model, the predicted object feature sequence including sequentially arranged predicted object features; wherein the decoding model is further configured to generate the predicted image description based on the predicted object feature sequence, wherein the order of appearance of objects in the predicted image description is consistent with the order of objects corresponding to the object features in the predicted object feature sequence, and the predicted object feature sequence is used to jointly train the encoding model, the sequence generation model and the decoding model with the predicted image description, the actual object feature sequence and the actual image description.

[0028] In some embodiments, the encoding model is also configured to encode the objects in the training image to obtain target visual features of the objects in the training image, wherein the target visual features include target object features of the objects in the training image and target image features of the training image; the sequence generation model includes: a first attention model, configured to perform attention calculation on the target object features of the objects in the training image; and a time series prediction model, configured to generate the predicted object feature sequence based on the attention calculation results of the first attention model.

[0029] In some embodiments, the time series prediction model is a two-layer long short-term memory network LSTM model, and the two-layer LSTM model includes a first LSTM model located in the first layer and a second LSTM model located in the second layer, and the input of the second LSTM model includes the attention calculation result of the first attention model and the output of the first LSTM model.

[0030] In some embodiments, the encoding model includes: a target detection model, configured to perform target detection on the training image to obtain initial object features of the object in the training image; and a feature fusion model, configured to obtain the initial image features of the training image, and fuse the initial object features of the object in the training image with the initial image features to obtain target visual features of the object in the training image, wherein the target visual features include target object features of the object in the training image and target image features of the training image.

[0031] In some embodiments, the encoding model is also configured to encode the objects in the training image to obtain target visual features of the objects in the training image, wherein the target visual features include target object features of the objects in the training image and target image features of the training image; the decoding model is also configured to generate the predicted image description based on the target visual features of the objects in the training image and the predicted object feature sequence.

[0032] In some embodiments, the decoding model includes a second attention model and a third attention model, and the input of the third attention model includes the output of the second attention model, the target visual features of the objects in the training image, and the predicted object feature sequence.

[0033] According to a fourth aspect of the present disclosure, a training device for an image description model is provided, comprising: an acquisition module, configured to acquire a training image and an actual image description thereof; a first generation module, configured to generate an actual object feature sequence based on the training image and the actual image description thereof, wherein the actual object feature sequence includes actual object features arranged in the order of appearance of the objects in the actual image description; a second generation module, configured to generate a predicted image description based on the training image and using the image description model; a training module, configured to train the image description model based on the predicted image description, the actual object feature sequence and the actual image description to obtain a trained image description model, wherein the actual object feature sequence and the actual image description are supervisory signals for training the image description model.

[0034] According to a fifth aspect of the present disclosure, there is provided an image description device, comprising: an acquisition module, configured to acquire an image to be described and to acquire a trained image description model from the training device described in any one of the above embodiments; and a generation module, configured to generate an image description corresponding to the image to be described based on the image to be described and using the trained image description model.

[0035] According to a sixth aspect of the present disclosure, an electronic device is provided, comprising: a memory; and a processor coupled to the memory, wherein the processor is configured to execute the training method or image description method described in any of the above embodiments based on instructions stored in the memory.

[0036] According to a seventh aspect of the present disclosure, a computer storable medium is provided, on which computer program instructions are stored, and when the instructions are executed by a processor, the training method or image description method described in any of the above embodiments is implemented.

[0037] In the above embodiments, the accuracy of image description can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The accompanying drawings, which constitute a part of the specification, illustrate embodiments of the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0039] The present disclosure may be more clearly understood from the following detailed description with reference to the accompanying drawings, in which:

[0040] Figure 1 is a flowchart illustrating a training method for an image description model according to some embodiments of the present disclosure;

[0041] Figure 2 is a flowchart illustrating the generation of a predicted image description according to some embodiments of the present disclosure;

[0042] Figure 3 is a schematic diagram showing the time steps of a sequence generation model according to some embodiments of the present disclosure;

[0043] Figure 4 is a flowchart illustrating a training image description model according to some embodiments of the present disclosure;

[0044] Figure 5 is a flowchart illustrating an image description method according to some embodiments of the present disclosure;

[0045] Figure 6 is a block diagram illustrating an image description model according to some embodiments of the present disclosure;

[0046] Figure 7 is a block diagram illustrating an electronic device according to some embodiments of the present disclosure;

[0047] Figure 8 is a block diagram illustrating a computer system for implementing some embodiments of the present disclosure;

[0048] Fig. 9 is a block diagram illustrating a training apparatus for an image description model according to some embodiments of the present disclosure;

[0049] Fig.10 is a block diagram illustrating an image description apparatus according to some embodiments of the present disclosure. DETAILED DESCRIPTION

[0050] Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. It should be noted that the relative arrangement of components and steps, numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present disclosure unless otherwise specifically stated.

[0051] At the same time, it should be understood that for the convenience of description, the sizes of the various parts shown in the drawings are not drawn according to the actual proportional relationship.

[0052] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the present disclosure, its application, or uses.

[0053] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered as part of the specification.

[0054] In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limiting. Therefore, other examples of the exemplary embodiments may have different values.

[0055] It should be noted that like reference numerals and letters refer to similar items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0056] Figure 1 is a flowchart illustrating a training method for an image description model according to some embodiments of the present disclosure.

[0057] like Figure 1 As shown, the training method for the image description model includes steps S110 to S140.

[0058] In step S110, a training image and its actual image description are obtained. The actual image description is the actual description of the image, which is used as a labeled sentence during the training process. In some examples, the actual image description of the training image I is formally represented as a sentence representation S = {w0, w1, ..., w T-1}, T represents the sentence length, w i Represents the i-th word in the sentence. i is an integer greater than or equal to 0 and less than or equal to T-1.

[0059] In step S120, an actual object feature sequence is generated based on the training image and its actual image description. The actual object feature sequence includes actual object features arranged in the order in which the objects appear in the actual image description. The actual object features are actual features of the objects in the training image. The actual features of the objects in the training image are regional features.

[0060] In some embodiments, generating the actual object feature sequence may be achieved in the following manner.

[0061] First, words belonging to the preset part of speech in the actual image description are obtained as target words. For example, a syntactic analysis tool can be used to tag the part of speech of each word in the actual image description, and then the words with the noun part of speech are used as target words. In this process, the target words are stored in the order in which they appear in the original actual image description.

[0062] Then, the target object features of the object corresponding to the target word in the training image are obtained as actual object features. In some embodiments, the objects in the training image can be encoded to obtain the target visual features of the objects in the training image. The target visual features include target object features and target image features.

[0063] For example, taking the example that the image description model includes a coding model and the coding model includes a target detection model and a feature fusion model, the target detection model is first used to perform target detection on the training image to obtain the initial object features of the object in the training image; then the initial image features of the training image are obtained; and then the feature fusion model is used to fuse the initial object features and initial image features of the object in the training image to obtain the target visual features of the training image, and the target visual features include the target object features of the object in the training image and the target image features of the training image. The feature fusion model includes an encoder, and the encoder is used to fuse the initial object features and initial image features of the object in the training image. The target image features are global features at the image level.

[0064] In some embodiments, the object detection model is a pre-trained model and is not trained again during the training of the image description model. For example, the object detection model is a pre-trained Faster-RCNN (Faster Region-based Convolutional Neural Networks) model.

[0065] In some embodiments, the encoder is an encoder based on an attention mechanism, for example, a Transformer encoder based on a self-attention mechanism.

[0066] In some embodiments, the initial object features of each object in the training image are represented as v i is the initial object feature of the i-th object, N I is the number of objects in the training image. i and N I Is a positive integer.

[0067] In some embodiments, the initial image features are obtained by random initialization. The initial image features can be expressed as v0=[IMG]. The initial image features can also be called visual markers.

[0068] In some embodiments, the initial object features and initial image features of each object in the training image are first concatenated to obtain the expanded visual features. The expanded visual features are input into a Transformer visual encoder with multiple layers of self-attention mechanism to perform mutual fusion operations between features to obtain a fused and improved visual feature. That is, the target visual features.

[0069] Finally, the actual object features are sorted according to the order in which the target words appear in the actual image description to obtain the actual object feature sequence. For example, the actual object feature sequence is expressed as ot is the actual feature of the tth object in the training image. o is the actual number of objects in the training image. t and N o are all positive integers. The actual object feature sequence is the region feature of the object of actual interest. o It also contains two special text tags [BOS] and [EOS], which indicate the beginning and end of the object feature sequence respectively.

[0070] In the above embodiment, in order to obtain a supervisory signal, an actual object feature sequence (also called an object sequence signal of interest) is extracted from the training data. The actual object feature sequence imitates the subjective experience of humans in cross-modal reasoning and can serve as a skeleton to guide descriptive sentences written by humans.

[0071] In step S130, a predicted image description is generated based on the training image using the image description model.

[0072] In some embodiments, the Figure 2 The above step S130 is implemented in the manner shown.

[0073] Figure 2 is a flow chart illustrating the generation of a predicted image description according to some embodiments of the present disclosure.

[0074] like Figure 2 As shown, generating the predicted image description includes steps S131 and S132.

[0075] In step S131, based on the training image, the image description model is used to generate a prediction object feature sequence. The prediction object feature sequence includes sequentially arranged prediction object features. For example, the prediction object feature sequence is represented as in, Represents the predicted object features of the i-th object.

[0076] For example, generating a prediction object feature sequence may be achieved in the following manner.

[0077] First, the objects in the training images are encoded to obtain the target visual features of the objects in the training images. The target visual features include target object features and target image feature object features.

[0078] In some embodiments, taking the case where the image description model includes a coding model and the coding model includes a target detection model and a feature fusion model, the target detection model can be used to perform target detection on the training image to obtain the initial object features of the object in the training image; then the initial image features of the training image can be obtained; and then the feature fusion model can be used to fuse the initial object features and initial image features of the object in the training image to obtain the target visual features of the object in the training image. The target visual features include target object features and target image features of the training image.

[0079] In some embodiments, the feature fusion model includes an encoder, which is used to fuse the initial object features and the initial image features of the object in the training image. For example, the encoding model also includes an embedding layer or a mapping layer connected in series after the feature fusion model, also known as an embedding layer. The embedding layer or the mapping layer is used to perform a dimensionality reduction operation on the output of the feature fusion model to improve the operation efficiency.

[0080] In some embodiments, the encoder is an encoder based on an attention mechanism. For example, the encoder is an encoding layer of a Transformer model, including multiple stacked encoders based on a self-attention mechanism. The specific structure refers to the Transformer model and will not be described in detail here.

[0081] Then, a predicted object feature sequence is generated based on the target visual features of the object in the training image. In this case, a predicted image description can be generated based on the target visual features of the object in the training image and the predicted object feature sequence.

[0082] In the above embodiment, the target visual feature represents the bottom-up signal, and the predicted object feature sequence represents the top-down signal. The combination of the two signals further improves the accuracy of image description. The top-down feature is the whole-to-part feature, while the bottom-up feature is the part-to-whole feature.

[0083] In some embodiments, the image description model includes a sequence generation model. The sequence generation model includes a first attention model and a time series prediction model. In this case, the generation of a predicted object feature sequence based on the target visual features of the object in the training image can be achieved in the following manner.

[0084] First, the first attention model is used to perform attention calculation on the target visual features of objects in the training images.

[0085] Then, according to the attention calculation result of the first attention model, the time series prediction model is used to generate a predicted object feature sequence. By combining the attention model and the time series prediction model, the time series prediction model can be assisted to more accurately generate a predicted object feature sequence, thereby further improving the accuracy of the image description. Taking the example that the encoding model also includes an embedding layer or a mapping layer connected in series after the feature fusion model, the sequence generation model also includes an embedding layer or a mapping layer connected in series after the time series prediction model.

[0086] In some embodiments, the time series prediction model is a two-layer LSTM (Long Short-Term Memory) model. The two-layer LSTM model includes a first LSTM model located in the first layer and a second LSTM model located in the second layer. The input of the second LSTM model includes the attention calculation result of the first attention model and the output of the first LSTM model.

[0087] In some embodiments, the input of the first LSTM model includes target image features of the training image. The target image features can provide contextual relationships of global features of the image for modeling dependencies between salient objects.

[0088] The input of the first LSTM model at each current time step also includes the output of the two-layer LSTM model at the time step before the current time step, that is, the hidden state of the output of the two-layer LSTM model. In the case where the sequence generation model also includes an embedding layer or a mapping layer connected in series after the time series prediction model, the input of the first LSTM model also includes the output of the embedding layer or the mapping layer connected in series after the time series prediction model at the previous time step. At the initial time step, the input of the first LSTM model is the output of the previous time step of the randomly initialized current time step and the start symbol [BOS].

[0089] Figure 3 is a time step schematic diagram showing a sequence generation model according to some embodiments of the present disclosure.

[0090] like Figure 3 As shown in Figure 1, the sequence generation model exists for t time steps. At each time step, the output of the first attention model is input into the second LSTM model for processing. At time step 1, the input of the first LSTM model includes the randomly initialized hidden state Starting symbol [BOS] and target image features Among them, the superscript 2 represents the hidden state output by the second LSTM model, and the subscript represents the time step.

[0091] For example, at time step 1, the hidden state of the output of the second LSTM model is It passes through the embedding layer to obtain the predicted object features of the object "man", which are represented by circles in the figure.

[0092] For example, at time step 2, the input of the first LSTM model includes Predicted object features and target image features for object "man"

[0093] For example, until time step t, the output of the first LSTM model consists of hidden states After the embedding layer, it gets the end symbol [EOS]. The input of the tth time step includes The predicted object features and target image features of the object output at the t-1 time step

[0094] return Figure 2 In step S132, the predicted image description is generated using the image description model according to the predicted object feature sequence. The order of appearance of the objects in the predicted image description is consistent with the order of objects corresponding to the object features in the predicted object feature sequence. The predicted object feature sequence is used to train the image description model together with the predicted image description, the actual object feature sequence and the actual image description.

[0095] In the above embodiment, the image description model first generates a predicted object feature sequence in the process of generating a predicted image description, and then generates a predicted image description based on the predicted object feature sequence. In the process of training the image description model, on the basis of the predicted image description, the actual object feature sequence and the actual image description, the predicted object feature sequence is further introduced as a supervisory signal for training, so that the trained image description model can further accurately learn the objects in the image and the order of the objects in the image description in the image description scene, thereby further improving the accuracy of the image description.

[0096] In some embodiments, taking the generation of a predicted object feature sequence according to target visual features of an object in a training image as an example, a predicted image description may be generated according to the target visual features of the object in the training image and the predicted object feature sequence.

[0097] In some embodiments, the image description model includes a decoding model. In this case, the predicted image description can be generated using the decoding model based on the target visual features of the object in the training image and the predicted object feature sequence. For example, the target visual features of the object in the training image and the predicted object feature sequence are input into the decoding model to obtain the predicted image description.

[0098] In some embodiments, the decoding model includes a second attention model and a third attention model. The input of the third attention model includes the output of the second attention model, the target visual features of the objects in the training image, and the predicted object feature sequence.

[0099] In some embodiments, the generation of a predicted image description may be achieved by using a decoding model based on target visual features of objects in training images and a predicted object feature sequence.

[0100] First, the second attention model is used to process its input to obtain the output of the second attention model.

[0101] In some embodiments, during the training process, the input of the second attention model is the actual image description of the training image and its position code. During the prediction process, the input of the second attention model at each time step is the output of the decoding model of the previous time step and its position code. For example, before the actual image description or the output of the decoding model of the previous time step is input into the second attention model, the actual image description or the output of the decoding model of the previous time step is first subjected to word embedding processing to obtain the corresponding encoding vector, and then the encoding vector is fused with the position code and the result is input into the second attention model.

[0102] Secondly, the output of the second attention model is linearly transformed using the third attention model to obtain the first vector matrix in the third attention model. d For example, the first vector matrix is ​​W q1 H d , W q1 is the weight matrix of the query vector matrix.

[0103] Then, the target visual features and the predicted object feature sequence of the objects in the training image are linearly transformed respectively to obtain the second vector matrix and the third vector matrix in the third attention model corresponding to the target visual features and the predicted object feature sequence of the objects in the training image respectively.

[0104] In some embodiments, the target visual feature The corresponding second vector matrix is ​​expressed as And the feature sequence of the predicted object The corresponding second vector matrix is ​​expressed as W k1 and W k2 are the weight matrices of the key vector matrix respectively.

[0105] In some embodiments, the target visual feature The corresponding third vector matrix is ​​expressed as And the feature sequence of the predicted object The corresponding third vector matrix is ​​expressed as W v1 and W v2 are the weight matrices of the value vector matrices respectively.

[0106] Finally, a predicted image description is generated according to the first vector matrix, the second vector matrix, and the third vector matrix in the third attention model.

[0107] For example, generating a predicted image description based on the first vector matrix, the second vector matrix and the third vector matrix in the third attention model can be achieved in the following manner.

[0108] First, a first attention score matrix is ​​determined based on the first vector matrix in the third attention model and the second vector matrix corresponding to the target visual features of the object in the training image. For example, the first attention score matrix is

[0109] Secondly, the first output of the third attention model is determined based on the first attention score matrix and the third vector matrix corresponding to the target visual features of the objects in the training image. For example, the first output of the third attention model is

[0110] Again, the second attention score matrix is ​​determined based on the first vector matrix in the third attention model and the second vector matrix corresponding to the feature sequence of the predicted object. For example, the second attention score matrix is

[0111] Then, the second output of the third attention model is determined according to the first attention score matrix and the third vector matrix corresponding to the feature sequence of the predicted object. For example, the second output of the third attention model is

[0112] Finally, a predicted image description is generated based on the first output and the second output. For example, the decoding model includes multiple decoders connected in series, each decoder includes a second attention model and a third attention model connected in series after the second attention model. A linear normalization (Linear and Softmax) layer is connected in series after the last decoder. The linear normalization layer can be used to perform a linear normalization operation on the first output and the second output of the third attention model in the last decoder to obtain a predicted image description. The decoding model predicts the probability distribution of the next word through the softmax function to obtain a predicted image description.

[0113] return Figure 1In step S140, the image description model is trained according to the predicted image description, the actual object feature sequence and the actual image description to obtain a trained image description model. The actual object feature sequence and the actual image description are supervisory signals for training the image description model.

[0114] In the above embodiment, during the training process, an actual object feature sequence is extracted from the training image based on the actual image description. The actual object features in the actual object feature sequence are arranged in the order in which the objects appear in the actual image description. During the training process, the actual image description and the actual object feature sequence are introduced together as supervisory signals for training, so that the trained image description model can more accurately learn the objects in the image and the order of the objects in the image description in the image description scene, thereby improving the accuracy of the image description.

[0115] In some embodiments, the Figure 4 The steps shown implement training an image description model.

[0116] Figure 4 is a flowchart illustrating training an image description model according to some embodiments of the present disclosure.

[0117] like Figure 4 As shown, training the image description model includes steps S141 to S143.

[0118] In step S141, a first loss value is determined according to the predicted object feature sequence and the actual object feature sequence.

[0119] For example, determining the first loss value may be achieved through the following steps 1)-3).

[0120] Step 1): For each predicted object feature in the predicted object feature sequence, determine from the actual object feature sequence the actual object feature that has the same position as each predicted object feature in the predicted object feature sequence as the positive sample object feature. For example, the predicted object feature is The positive sample object feature is represented as For S o The actual feature o of the tth object in t , that is The t-th target visual feature in Represents the hidden state of the tth object output by the two-layer LSTM model, that is, the hidden feature.

[0121] Step 2): For each predicted object feature, determine the actual object features other than the positive sample features in the actual object feature sequence as negative sample object features. For example, the set of negative sample object features is represented as That is S o The actual feature o of the tth object is t The actual characteristics of objects other than Except for the t-th target visual feature and target image features Other target visual features other than are marked by j.

[0122] Step 3): Determine a first loss value based on the positive sample object features and the negative sample object features corresponding to all predicted object features.

[0123] In some embodiments, the above step 3) can be implemented as follows.

[0124] First, determine the similarity between each predicted object feature and the corresponding positive sample object feature as the first similarity. For example, the first similarity is

[0125] Then, the similarity between each predicted object feature and the corresponding negative sample feature is determined as the second similarity. For example, the second similarity is

[0126] Finally, a first loss value is determined according to the first similarity and the second similarity. In some embodiments, the first loss value is negatively correlated with the first similarity and positively correlated with the second similarity.

[0127] For example, the first loss value is calculated by formula (1).<a,b> Calculates the cosine similarity between vectors a and b. exp() is an exponential calculation function. o +1 object includes the actual N in the training image o objects and 1 end text tag [EOS]. Formula (1) is the calculation based on the InfoNCE (Information Noise Contrastive Estimation) loss function.

[0128]

[0129] In the above embodiment, the first loss value is determined based on the similarity between the predicted object features and the positive and negative sample object features, so that the image description model continuously learns how to distinguish between object features that have a positive effect on image description and object features that have a negative effect on image description during the training process, thereby further improving the accuracy of image description performed by the image description model.

[0130] In step S142, a second loss value is determined based on the predicted image description and the actual image description. In some embodiments, the second loss value is determined based on the predicted image description and the actual image description using a cross entropy loss function.

[0131] In step S143, the image description model is trained according to the first loss value and the second loss value. In some embodiments, the image description model can be trained according to the sum of the first loss value and the second loss value. For example, by training the image description model, the sum of the first loss value and the second loss value is reduced to meet the preset loss condition. For example, the preset loss condition can be that the sum of the first loss value and the second loss value is less than the loss value threshold or meets the loss value range.

[0132] Figure 5 is a flowchart illustrating an image description method according to some embodiments of the present disclosure.

[0133] like Figure 5 As shown, the image description method includes steps S510 to S520.

[0134] In step S510, an image to be described is obtained.

[0135] In step S520, based on the image to be described, the trained image description model is used to generate an image description corresponding to the image to be described. The trained image description model is obtained by using the training method in any of the embodiments of the present disclosure. The training process of the image description model can be referred to for generating an image description corresponding to the image to be described using the trained image description model, which will not be described in detail here.

[0136] Figure 6 is a block diagram illustrating an image description model according to some embodiments of the present disclosure.

[0137] like Figure 6 As shown, the image description model 6 includes an encoding model 61 and a decoding model 63 .

[0138] The coding model 61 is configured to obtain a training image and its actual image description, generate an output of the coding model based on the training image, and adjust the training parameters in the coding model to train the coding model based on the predicted image description, the actual object feature sequence and the actual image description output by the decoding model.

[0139] The decoding model 63 is configured to generate a predicted image description according to the output of the encoding model, and to adjust the training parameters in the decoding model according to the predicted image description, the actual object feature sequence and the actual image description to train the decoding model.

[0140] The actual object feature sequence is generated according to the training image and its actual image description. The actual object feature sequence includes the actual object features arranged in the order in which the objects appear in the actual image description. The actual object feature sequence and the actual image description are supervisory signals for training the image description model. The actual object features are the actual features of the objects in the training images. The image description model is trained by training the encoding model and the decoding model to obtain the trained image description model.

[0141] In some embodiments, the encoding model 61 includes an object detection model 611 and a feature fusion model 613. The feature fusion model 613 is connected in series after the object detection model 611. For example, the feature fusion model 613 includes one or more encoders 6131. ​​In the case where the feature fusion model 613 includes multiple encoders 6131, the multiple encoders 6131 are connected in series. For example, the structure of the encoder is the encoder structure of the Transformer model.

[0142] The object detection model 611 is configured to perform object detection on the training image to obtain initial object features of the object in the training image.

[0143] In some embodiments, the encoding model 61 further includes a first embedding layer 612 connected in series after the object detection model 611 and before the feature fusion model 613. The first embedding layer 612 is configured to perform dimensionality reduction processing on the output of the object detection model 611.

[0144] The feature fusion model 613 is configured to obtain the initial image features of the training image, and fuse the initial object features and the initial image features of the object in the training image to obtain the target visual features of the object in the training image. The target visual features include the target object features of the object in the training image and the target image features of the training image.

[0145] In some embodiments, the encoding model 61 further includes a second embedding layer 614 connected in series after the feature fusion model 613. The second embedding layer 614 is configured to perform embedding processing or mapping processing on the output of the feature fusion model 613 to achieve dimensionality reduction.

[0146] In some embodiments, the image description model 6 further includes a sequence generation model 62. The sequence generation model 62 is configured to generate a predicted object feature sequence according to the output of the encoding model. The predicted object feature sequence includes predicted object features arranged in sequence. The decoding model is further configured to generate a predicted image description according to the predicted object feature sequence, wherein the order of appearance of objects in the predicted image description is consistent with the order of objects corresponding to the object features in the predicted object feature sequence, and the predicted object feature sequence is used to train the encoding model, the sequence generation model and the decoding model together with the predicted image description, the actual object feature sequence and the actual image description.

[0147] In some embodiments, the encoding model 61 is further configured to encode objects in the training images to obtain target visual features of the objects in the training images.

[0148] The sequence generation model 62 includes a first attention model 621 and a time series prediction model 622. The first attention model 621 is configured to perform attention calculation on the target object features of the object in the training image. The time series prediction model 622 is configured to generate a predicted object feature sequence based on the attention calculation result of the first attention model.

[0149] In some embodiments, the time series prediction model 622 is a two-layer long short-term memory network LSTM model. The two-layer LSTM model includes a first LSTM model 6221 located in the first layer and a second LSTM model 6222 located in the second layer. The input of the second LSTM model 6222 includes the attention calculation result of the first attention model 621 and the output of the first LSTM model 6221.

[0150] In some embodiments, the sequence generation model 62 further includes a third embedding layer 623. The third embedding layer 623 is configured to perform dimensionality reduction processing on the output of the time series prediction model 622.

[0151] In some embodiments, taking the example where the encoding model is configured to encode objects in a training image to obtain target visual features of the objects in the training image, the decoding model 63 is also configured to generate a predicted image description based on the target visual features of the objects in the training image and the predicted object feature sequence.

[0152] In some embodiments, the decoding model 63 includes one or more decoders 631. Each decoder 631 includes a second attention model 6311 and a third attention model 6312. The input of the third attention model 6312 includes the output of the second attention model 6311, the target visual features of the object in the training image, and the predicted object feature sequence. In the case where there are multiple decoders 631, multiple decoders 631 are connected in series. For example, the structure of each decoder 631 is the same as the decoder structure of the Transformer model.

[0153] In some embodiments, the decoding model 63 further includes a linear normalization layer 632. The linear normalization layer 632 is configured to perform linear normalization processing on the output of the last decoder 631 to obtain a predicted image description.

[0154] In some embodiments, the decoding model 63 further includes a fourth embedding layer 630. The fourth embedding layer 630 is configured to embed the input of the decoding model, fuse it with the position code, and input the fusion result into the first decoder 631.

[0155] The various parts of the above image description model are also applicable to the image description process, that is, given an image to be described, the corresponding image description is obtained. The image description process is similar to the processing of the training image in the training process, and will not be repeated here.

[0156] Figure 7 is a block diagram illustrating an electronic device according to some embodiments of the present disclosure.

[0157] like Figure 7 As shown, the electronic device 7 includes a memory 71; and a processor 72 coupled to the memory 71. The memory 71 is used to store instructions for executing the training method or the image description method corresponding to the embodiment. The processor 72 is configured to execute the training method or the image description method in any of the embodiments of the present disclosure based on the instructions stored in the memory 71.

[0158] Figure 8 is a block diagram illustrating a computer system for implementing some embodiments of the present disclosure.

[0159] like Figure 8 As shown, the computer system 80 may be embodied in the form of a general-purpose computing device. The computer system 80 includes a memory 810, a processor 820, and a bus 800 that connects the various system components.

[0160] The memory 810 may include, for example, a system memory, a non-volatile storage medium, etc. The system memory may store, for example, an operating system, an application, a boot loader, and other programs. The system memory may include a volatile storage medium, such as a random access memory (RAM) and / or a cache memory. The non-volatile storage medium may store, for example, instructions for executing a corresponding embodiment of at least one of the training method and the image description method. The non-volatile storage medium includes, but is not limited to, a disk storage, an optical storage, a flash memory, and the like.

[0161] The processor 820 can be implemented by a general processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistors, etc. Discrete hardware components. Accordingly, each module such as the judgment module and the determination module can be implemented by a central processing unit (CPU) running instructions in a memory that execute corresponding steps, or can be implemented by a dedicated circuit that executes corresponding steps.

[0162] The bus 800 may use any of a variety of bus architectures, including, but not limited to, an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MCA) bus, and a Peripheral Component Interconnect (PCI) bus.

[0163] The computer system 80 may also include an input / output interface 830, a network interface 840, a storage interface 850, etc. These interfaces 830, 840, 850, the memory 810, and the processor 820 may be connected via a bus 800. The input / output interface 830 may provide a connection interface for input / output devices such as a display, a mouse, and a keyboard. The network interface 840 may provide a connection interface for various networked devices. The storage interface 850 may provide a connection interface for external storage devices such as a floppy disk, a USB flash drive, and an SD card.

[0164] Here, various aspects of the present disclosure are described with reference to flowcharts and / or block diagrams of methods, devices, and computer program products according to embodiments of the present disclosure. It should be understood that each frame of the flowchart and / or block diagram and the combination of frames can be implemented by computer-readable program instructions.

[0165] These computer-readable program instructions may be provided to a processor of a general-purpose computer, a special-purpose computer or other programmable device to produce a machine, so that the processor executes the instructions to produce means for implementing the functions specified in one or more blocks in the flowchart and / or block diagram.

[0166] These computer-readable program instructions may also be stored in a computer-readable memory, which cause the computer to work in a specific manner to produce an article of manufacture, including instructions for implementing the functions specified in one or more blocks in the flowchart and / or block diagram.

[0167] Fig. 9 is a block diagram illustrating a training apparatus for an image description model according to some embodiments of the present disclosure.

[0168] like Fig. 9 As shown, the training device 9 includes an acquisition module 91 , a first generation module 92 , a second generation module 93 and a training module 94 .

[0169] The acquisition module 91 is configured to acquire the training image and its actual image description, for example, by performing the following steps: Figure 1 Step S110 is shown.

[0170] The first generating module 92 is configured to generate an actual object feature sequence according to the training image and its actual image description, wherein the actual object feature sequence includes actual object features arranged in the order in which the objects appear in the actual image description, for example, by performing the following steps: Figure 1 Step S120 is shown.

[0171] The second generation module 93 is configured to generate a predicted image description based on the training image using the image description model, for example, by performing the following steps: Figure 1 Step S130 is shown.

[0172] The training module 94 is configured to train the image description model according to the predicted image description, the actual object feature sequence and the actual image description to obtain a trained image description model, wherein the actual object feature sequence and the actual image description are supervisory signals for the training image description model, for example, by performing the following steps: Figure 1 Step S140 is shown.

[0173] Fig.10 is a block diagram illustrating an image description apparatus according to some embodiments of the present disclosure.

[0174] like Fig.10 As shown, the image description device 10 includes an acquisition module 101 and a generation module 102 .

[0175] The acquisition module 101 is configured to acquire the image to be described and obtain the trained image description model from the training device, for example, by executing Figure 5 Step S510 is shown. The training device is the training device in any of the embodiments of the present disclosure. The image description model is the image description model in any of the embodiments of the present disclosure.

[0176] The generating module 102 is configured to generate an image description corresponding to the image to be described by using the trained image description model according to the image to be described, for example, Figure 5 Step S520 is shown.

[0177] The present disclosure can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects.

[0178] The accuracy of image description can be improved through the training method and device, image description method, device and model, electronic device, and computer storable medium in the above embodiments.

[0179] So far, the training method and device, image description method, device and model, electronic device, and computer storable medium according to the present disclosure have been described in detail. In order to avoid obscuring the concept of the present disclosure, some details known in the art are not described. Based on the above description, those skilled in the art can fully understand how to implement the technical solution disclosed herein.

Claims

1. A training method for an image description model, comprising: Get training images and their actual image descriptions; Generate an actual object feature sequence according to the training image and its actual image description, wherein the actual object feature sequence includes actual object features arranged in the order in which the objects appear in the actual image description; Generate a predicted image description based on the training image using the image description model; The image description model is trained according to the predicted image description, the actual object feature sequence and the actual image description to obtain a trained image description model, wherein the actual object feature sequence and the actual image description are supervisory signals for training the image description model.

2. The training method according to claim 1, wherein: Generating a predicted image description according to the training image and using the image description model includes: According to the training image, using the image description model, generating a prediction object feature sequence, the prediction object feature sequence comprising sequentially arranged prediction object features; According to the predicted object feature sequence, a predicted image description is generated using the image description model, wherein the order of appearance of objects in the predicted image description is consistent with the order of objects corresponding to the object features in the predicted object feature sequence, and the predicted object feature sequence is used to train the image description model together with the predicted image description, the actual object feature sequence and the actual image description.

3. The training method according to claim 2, wherein: Training the image description model includes: Determining a first loss value according to the predicted object feature sequence and the actual object feature sequence; Determining a second loss value according to the predicted image description and the actual image description; The image description model is trained according to the first loss value and the second loss value.

4. The training method according to claim 3, wherein: Determining the first loss value includes: For each predicted object feature in the predicted object feature sequence, determining, from the actual object feature sequence, an actual object feature having the same position as each predicted object feature in the predicted object feature sequence as a positive sample object feature; For each predicted object feature, determining other actual object features in the actual object feature sequence except the positive sample object feature as negative sample object features; The first loss value is determined according to the positive sample object features and the negative sample object features corresponding to all the predicted object features.

5. The training method according to claim 4, wherein: Determining the first loss value according to the positive sample object features and the negative sample object features corresponding to all the predicted object features includes: Determine the similarity between each predicted object feature and the corresponding positive sample object feature as a first similarity; Determine the similarity between each predicted object feature and the corresponding negative sample feature as a second similarity; The first loss value is determined according to the first similarity and the second similarity.

6. The training method according to claim 5, wherein: The first loss value is negatively correlated with the first similarity, and the first loss value is positively correlated with the second similarity.

7. The training method according to claim 3, wherein: Determining the second loss value includes: The second loss value is determined according to the predicted image description and the actual image description using a cross entropy loss function.

8. The training method according to claim 3, wherein: According to the first loss value and the second loss value, training the image description model includes: The image description model is trained according to the sum of the first loss value and the second loss value.

9. The training method according to claim 2, wherein: Generating a prediction object feature sequence according to the training image and using the image description model comprises: Encoding the object in the training image to obtain target visual features of the object in the training image, wherein the target visual features include target object features and target image features; According to the target visual features of the object in the training image, the predicted object feature sequence is generated; wherein, Generating the predicted image description according to the predicted object feature sequence and using the image description model includes: generating the predicted image description according to the target visual features of the object in the training image and the predicted object feature sequence.

10. The training method according to claim 9, wherein: The image description model includes a sequence generation model, the sequence generation model includes a first attention model and a time series prediction model, and generating the predicted object feature sequence according to the target visual features of the object in the training image includes: Using the first attention model, performing attention calculation on target visual features of the object in the training image; According to the attention calculation result of the first attention model, the time series prediction model is used to generate the predicted object feature sequence.

11. The training method according to claim 10, wherein: The time series prediction model is a two-layer long short-term memory network LSTM model, which includes a first LSTM model located in the first layer and a second LSTM model located in the second layer, and the input of the second LSTM model includes the attention calculation result of the first attention model and the output of the first LSTM model.

12. The training method according to claim 11, wherein: The input of the first LSTM model includes target image features of the training image.

13. The training method according to claim 9, wherein: The image description model includes a decoding model, and generating the predicted image description according to the target visual features of the object in the training image and the predicted object feature sequence includes: The predicted image description is generated using the decoding model according to the target visual features of the object in the training image and the predicted object feature sequence.

14. The training method according to claim 13, wherein: The decoding model includes a second attention model and a third attention model, and the input of the third attention model includes the output of the second attention model, the target visual features of the objects in the training image, and the predicted object feature sequence.

15. The training method according to claim 14, wherein: Generating the predicted image description using the decoding model according to the target visual features of the object in the training image and the predicted object feature sequence includes: Processing the input using the second attention model to obtain an output of the second attention model; Using the third attention model, linearly transform the output of the second attention model to obtain a first vector matrix in the third attention model; Performing linear transformations on the target object features of the object in the training image and the predicted object feature sequence, respectively, to obtain a second vector matrix and a third vector matrix in the third attention model that correspond to the target visual features of the object in the training image and the predicted object feature sequence, respectively; The predicted image description is generated according to the first vector matrix, the second vector matrix and the third vector matrix in the third attention model.

16. The training method according to claim 15, wherein: Generating the predicted image description according to the first vector matrix, the second vector matrix, and the third vector matrix in the third attention model includes: determining a first attention score matrix based on a first vector matrix in the third attention model and a second vector matrix corresponding to a target visual feature of an object in the training image; determining a first output of the third attention model based on the first attention score matrix and a third vector matrix corresponding to target visual features of objects in the training image; Determining a second attention score matrix according to the first vector matrix in the third attention model and the second vector matrix corresponding to the prediction object feature sequence; Determining a second output of the third attention model according to the first attention score matrix and a third vector matrix corresponding to the prediction object feature sequence; The predicted image description is generated based on the first output and the second output.

17. The training method according to claim 9, wherein: The image description model includes a coding model, and the coding model includes a target detection model and a feature fusion model. The object in the training image is encoded to obtain the target visual features of the object in the training image, including: Using the target detection model, performing target detection on the training image to obtain initial object features of the object in the training image; Acquiring initial image features of the training image; The feature fusion model is used to fuse the initial object features of the object in the training image and the initial image features to obtain the target visual features of the object in the training image, wherein the target visual features include the target object features of the object and the target image features of the training image.

18. The training method according to claim 17, wherein: The feature fusion model includes an encoder, and the encoder is used to fuse the initial object features of the object in the training image and the initial image features.

19. The training method according to claim 1, wherein: Generating an actual object feature sequence according to the training image and its actual image description comprises: Obtaining a word belonging to a preset part of speech in the actual image description as a target word; Acquire a target object feature of an object corresponding to the target word in the training image as an actual object feature; The actual object features are sorted according to the order in which the target words appear in the actual image description to obtain the actual object feature sequence.

20. An image description method based on the trained image description model obtained by the training method according to any one of claims 1 to 19, comprising: Obtain the image to be described; According to the image to be described, an image description corresponding to the image to be described is generated using the trained image description model.

21. An image description model, comprising: Encoding model and decoding model, where The encoding model is configured to obtain a training image and its actual image description, generate an output of the encoding model based on the training image, and adjust training parameters in the encoding model based on the predicted image description output by the decoding model, the actual object feature sequence and the actual image description to train the encoding model; The decoding model is configured to generate a predicted image description according to an output of the encoding model, and to adjust a training parameter in the decoding model according to the predicted image description, the actual object feature sequence and the actual image description to train the decoding model; The actual object feature sequence is generated based on the training image and its actual image description, and the actual object feature sequence includes actual object features arranged in the order of appearance of objects in the actual image description. The actual object feature sequence and the actual image description are supervisory signals for training the image description model.

22. The image description model according to claim 21, further comprising: A sequence generation model, configured to generate a prediction object feature sequence according to an output of the encoding model, wherein the prediction object feature sequence includes sequentially arranged prediction object features; In which, the decoding model is also configured to generate the predicted image description based on the predicted object feature sequence, wherein the order of appearance of objects in the predicted image description is consistent with the order of objects corresponding to the object features in the predicted object feature sequence, and the predicted object feature sequence is used to jointly train the encoding model, the sequence generation model and the decoding model with the predicted image description, the actual object feature sequence and the actual image description.

23. The image description model according to claim 22, wherein: The encoding model is further configured to encode the object in the training image to obtain a target visual feature of the object in the training image, wherein the target visual feature includes a target object feature of the object in the training image and a target image feature of the training image; The sequence generation model comprises: a first attention model configured to perform attention calculations on target object features of objects in the training image; and The time series prediction model is configured to generate the predicted object feature sequence according to the attention calculation result of the first attention model.

24. The image description model according to claim 23, wherein: The time series prediction model is a two-layer long short-term memory network LSTM model, which includes a first LSTM model located in the first layer and a second LSTM model located in the second layer, and the input of the second LSTM model includes the attention calculation result of the first attention model and the output of the first LSTM model.

25. The image description model according to claim 21, wherein: The coding model includes: an object detection model configured to perform object detection on the training image to obtain initial object features of the object in the training image; and The feature fusion model is configured to obtain the initial image features of the training image, and fuse the initial object features of the object in the training image with the initial image features to obtain the target visual features of the object in the training image, wherein the target visual features include the target object features of the object in the training image and the target image features of the training image.

26. The image description model according to claim 22, wherein: The encoding model is further configured to encode the object in the training image to obtain a target visual feature of the object in the training image, wherein the target visual feature includes a target object feature of the object in the training image and a target image feature of the training image; The decoding model is also configured to generate the predicted image description based on the target visual features of the object in the training image and the predicted object feature sequence.

27. The image description model according to claim 26, wherein: The decoding model includes a second attention model and a third attention model, and the input of the third attention model includes the output of the second attention model, the target visual features of the objects in the training image, and the predicted object feature sequence.

28. An electronic device comprising: Memory; as well as A processor coupled to the memory, wherein the processor is configured to execute the training method according to any one of claims 1 to 19 or the image description method according to claim 20 based on instructions stored in the memory.

29. A computer storable medium having computer program instructions stored thereon, which, when executed by a processor, implement the training method according to any one of claims 1 to 19 or the image description method according to claim 20.

Images