Image description model training method and image description method based on modality enhancement

Abstract

The present invention provides an image description model training method based on modality enhancement and an image description method, wherein the model training method, when performing a single model training, comprises: obtaining image data in current training data, extracting image region features of the image data; obtaining text data in the current training data; for each word in the text data, sequentially performing a contextual reasoning process based on modality information enhancement to obtain corresponding semantic reasoning information; constructing a cross-entropy loss of the model based on the semantic reasoning information corresponding to each word and the semantic label information of the current image, so as to optimize the image description model based on the cross-entropy loss, and fully exploit the correlation between image visual features and text language features, thereby enhancing the model reasoning ability and improving the accuracy of the model in understanding multimodal semantics.

Description

Technical Field

[0001] The present invention relates to the field of image description technology, and in particular to an image description model training method based on modality enhancement and an image description method. One or more embodiments of the present specification also relate to an image description model training method based on modality enhancement and an image description method, a device and a computer storage medium. Background Art

[0002] Image description is a machine-generated description of the image content for a given image, which is similar to the process of "translating" an image into a language description. Its essence is to segment the image and use data, symbols, and formal language to represent the characteristics of different sub-areas after segmentation. Therefore, the understanding of the image content of each sub-area, that is, expressing the semantic information contained in the visual features in the semantic space of the language modality, plays a vital role in the accuracy of the entire image description task.

[0003] At present, the commonly used image description models are usually based on the encoder-decoder structure; the encoder usually uses a convolutional neural network pre-trained on a large-scale image classification dataset to extract visual features from the image; the decoder usually uses a network based on a recurrent neural network or a self-attention mechanism, such as a long short-term memory network (LSTM) and a transformer, to generate a natural sentence describing the image content, i.e., a sequence of words, based on the extracted image visual features. However, the image description model currently used does not fully consider the correlation between the multiple visual features associated with each word in the text sentence and the semantic information in the semantic space. Therefore, there are problems such as insufficient understanding of visual information and inaccurate description content, which in turn leads to insufficient accuracy of image description. Summary of the invention

[0004] In view of the above shortcomings in the prior art, the purpose of the present invention is to provide an image description model training method based on modality enhancement and an image description method, device and computer storage medium, which are used to solve the problems of insufficient understanding of visual information and inaccurate description content of the existing image description model, so as to improve the accuracy of image description.

[0005] To achieve the above-mentioned purpose and other related purposes, the present invention provides, in a first aspect, a method for training an image description model based on modality enhancement, wherein a model training process is performed on an image description model based on each group of training data and semantic label information corresponding to the training data to obtain a trained image description model; wherein the training data includes image data and text data associated with the image data; the text data includes pre-calibrated words; when performing a single model training process, the method comprises: extracting image area features of the image data based on the image data in the current training data; based on the text data in the current training data; for each word in the text data, sequentially performing a contextual reasoning process based on modality information enhancement to obtain semantic reasoning information corresponding to each word; based on the semantic reasoning information corresponding to each word and the semantic label information of the current image, constructing a cross entropy loss of the current text data to optimize the image description model based on the cross entropy loss; updating the current training data to perform the next model training process based on the updated current training data until exiting.

[0006] In one embodiment of the present invention, the image description model includes a hidden state feature memory pool and a visual attention feature memory pool; when executing a single contextual reasoning process based on modal information enhancement, it includes: obtaining the hidden state feature and visual attention feature of the current word; based on the hidden state feature and the visual attention feature of the current word, correspondingly updating the hidden state feature memory pool and the visual attention feature memory pool to obtain the current hidden state feature sequence and the current visual attention feature sequence; obtaining the sequence feature of the current visual attention feature sequence; based on the sequence feature and the word embedding feature of the current word, constructing a hidden state feature query vector for the current word; based on the hidden state feature query vector and the visual attention feature of the current word Hidden state features, construct a multimodal feature query vector of the current word; perform time and semantic enhancement on the current hidden state feature sequence to obtain an enhanced hidden state feature sequence; and perform time and semantic enhancement on the current visual attention feature sequence to obtain an enhanced visual attention feature sequence; based on the enhanced hidden state feature sequence and the enhanced visual attention feature sequence, obtain the multimodal feature sequence of the current word; based on the multimodal feature sequence of the current word and the multimodal feature query vector of the current word, use the attention mechanism to obtain the contextual reasoning features of the current word; based on the contextual reasoning features of the current word and the visual attention features of the current word, obtain the semantic reasoning information of the current word.

[0007] In one embodiment of the present invention, the method for obtaining the hidden state features and visual attention features of the current word includes: obtaining the hidden state features corresponding to the previous word and obtaining the word embedding features of the current word; obtaining the hidden state features of the current word based on the word embedding features of the current word, the image area features, and the hidden state features corresponding to the previous word; and, based on the hidden state features of the current word and the image area features, using an attention mechanism to obtain the visual attention features of the current word.

[0008] In one embodiment of the present invention, the implementation method of updating the hidden state feature memory pool and the visual attention feature memory pool accordingly based on the hidden state feature and the visual attention feature of the current word includes: inputting the hidden state feature of the current word into the top-level storage of the hidden state feature memory pool, and deleting the hidden state feature stored in the bottom-level storage of the hidden state feature memory pool; and inputting the visual attention feature of the current word into the top-level storage of the visual attention feature memory pool, and deleting the visual attention feature stored in the bottom-level storage of the visual attention feature memory pool.

[0009] In one embodiment of the present invention, the temporal and semantic enhancement of the current hidden state feature sequence includes: based on the sequence position of each hidden state feature in the current hidden state feature sequence, using a first position encoder to obtain the timing information of each hidden state feature; superimposing the timing information of each hidden state feature and the word embedding feature of the current word to the corresponding hidden state feature; and, the temporal and semantic enhancement of the current visual attention feature sequence includes: based on the sequence position of each visual attention feature in the current visual attention feature sequence, using a second position encoder to obtain the timing information of each visual attention feature; superimposing the timing information of each visual attention feature and the word embedding feature of the current word to the corresponding visual attention feature.

[0010] In one embodiment of the present invention, the first position encoder and the second position encoder are the same, and both include:

[0011]

[0012] Among them, p i Indicates the relative position of the feature in the corresponding feature sequence; j is the dimension of the position encoding representation. When j is an odd number, f() is sin(), and when j is an even number, f() is cos().

[0013] In one embodiment of the present invention, the implementation method of using an attention mechanism to obtain contextual reasoning features of the current word based on the multimodal feature sequence of the current word and the multimodal feature query vector of the current word includes: based on the multimodal feature sequence of the current word, using a self-attention mechanism to obtain a new multimodal feature sequence; based on the new multimodal feature sequence and the multimodal feature query vector of the current word, using a traditional attention mechanism to obtain contextual reasoning features of the current word.

[0014] In one embodiment of the present invention, the implementation method of constructing a multimodal feature query vector of the current word includes: using a gated linear unit, taking the hidden state feature query vector of the current word and the hidden state feature of the current word as inputs of the gated linear unit, and constructing the multimodal feature query vector of the current word.

[0015] In one embodiment of the present invention, when executing the model training process once, it also includes: executing each of the contextual reasoning processes based on modal information enhancement in sequence to obtain the semantic alignment loss corresponding to each word; based on the semantic alignment loss corresponding to each word, obtaining the semantic alignment loss of the current text data; based on the cross loss and semantic alignment loss of the current text data, constructing the model total loss of the current text data, so as to optimize the image description model based on the model total loss; wherein, in the contextual reasoning process based on modal information enhancement once, after obtaining the enhanced hidden state feature sequence and obtaining the enhanced visual attention feature sequence, it also includes: obtaining the sequence feature of the current hidden state feature sequence; based on the sequence feature and the word embedding feature of the current word, constructing the visual attention feature query vector of the current word; based on the enhanced hidden state feature sequence and the hidden state feature query vector, using the attention mechanism to obtain the hidden state semantic feature; based on the enhanced visual attention feature and the visual attention feature query vector, using the attention mechanism to obtain the visual semantic feature; and, based on the hidden state semantic feature and the visual semantic feature, using the semantic alignment method to construct the semantic alignment loss of the current word.

[0016] In one embodiment of the present invention, the implementation method of obtaining hidden state semantic features by using an attention mechanism based on the enhanced hidden state feature sequence and the hidden state feature query vector includes: based on the enhanced hidden state feature sequence, using a self-attention mechanism to obtain an interactively enhanced hidden state feature sequence; based on the interactively enhanced hidden state feature sequence and the hidden state feature query vector, using a traditional attention mechanism to obtain the interactively enhanced hidden state semantic features; and, the implementation method of obtaining visual semantic features by using an attention mechanism based on the enhanced visual attention features and the visual attention feature query vector includes: based on the enhanced visual attention feature sequence, using a self-attention mechanism to obtain an interactively enhanced visual attention feature sequence; based on the interactively enhanced visual attention feature sequence and the visual attention feature query vector, using a traditional attention mechanism to obtain the interactively enhanced visual attention features.

[0017] The present invention provides an image description method in a second aspect, characterized in that it includes: constructing each training data set based on sample data of image description; a single group of the training data includes image data, and text data associated with the image data; based on each of the training data sets and the semantic label information corresponding to the training data, using any of the above-described modality-enhanced image description model training methods to train a preset model to obtain a trained image description model; for the image to be described, using the trained image description model to perform contextual semantic information reasoning to obtain the semantic information of the input image.

[0018] The present invention provides an electronic device in a third aspect, comprising: a processor and a memory; the memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory, so that the electronic device executes any of the above-described modality-enhanced image description model training methods or the above-described image description methods.

[0019] In a fourth aspect, the present invention provides a computer storage medium storing a computer program, wherein the computer program is executed by a processor to perform any of the above-described modality-enhanced image description model training methods or the above-described image description methods.

[0020] As described above, the image description model training method based on modal enhancement and the image description method, device and computer storage medium proposed in the present invention; use the memory pool of each modality to store the feature sequence of the corresponding modality, and enhance the modal information of each modal feature respectively, so as to realize multimodal contextual reasoning of each word in the text sentence based on the semantics and time associations in the multimodal time feature sequence by performing intra-modal and inter-modal information interaction. This method can fully explore the correlation between image visual features and text language features, thereby enhancing the model reasoning ability and improving the accuracy of the model in understanding multimodal semantics. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 Shown is a schematic flow chart of executing a single model training process of the modality-enhanced image description model training method provided by the present invention in one implementation;

[0022] Figure 2 It is a schematic diagram of the process of step S200 in one embodiment of the present invention;

[0023] Figure 3 It is a schematic diagram of the process of step S201 in one embodiment of the present invention;

[0024] Figure 4 Shown is a schematic flow chart of executing a single model training process of the modality-enhanced image description model training method provided by the present invention in another embodiment;

[0025] Figure 5 Shown is a schematic flow chart of the image description method provided by the present invention in one embodiment. DETAILED DESCRIPTION

[0026] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.

[0027] It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present invention, and thus the drawings only show components related to the present invention rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed arbitrarily, and the component layout may also be more complicated.

[0028] To facilitate the understanding of the technical solution of this application, the following terms are explained as follows:

[0029] Image description is to represent the visual content of an image in the form of textual language.

[0030] Contextual reasoning is to perform comprehensive semantic information reasoning on the current text based on the semantic information of the current text and the visual feature information of the image, and based on the semantic information of the context.

[0031] In order to solve the problems existing in the prior art, the present invention provides, in a first aspect, a method for training an image description model based on modality enhancement, wherein a model training process is performed on the image description model based on each group of training data and the semantic label information corresponding to the training data to obtain a trained model; that is, after the model training process is performed on the image description model based on a single group of training data, the training data is updated to re-execute the model training process based on the new training data, and the process is repeated until exited, thereby obtaining the final image description model.

[0032] The image description model is used to obtain the semantic information of the text data associated with the image through contextual reasoning as the description information of the image.

[0033] The training data includes image data and text data associated with the image data; the text data includes pre-marked words; and the words are arranged in sequence in the text data.

[0034] The semantic tag information is the semantic information actually corresponding to each word in the text data.

[0035] See also Figure 1 , which is a flow chart of a method for training an image description model based on modality enhancement in one embodiment of the present invention, and a process of executing a single model training process.

[0036] In this embodiment, the image description model training includes a pre-constructed feature memory pool, including a hidden state memory pool and a visual attention feature memory pool; wherein the hidden state memory pool is used to store the hidden state features of words, and the visual attention feature memory pool is used to store the visual attention features of words.

[0037] Among them, the hidden state feature is a word semantic feature combined with an image partition feature; and the visual attention feature is an image feature combined with a word semantic feature.

[0038] like Figure 1 As shown, the image description model training method based on modality enhancement comprises the following steps when executing a single model training process:

[0039] S100, acquiring image data in current training data, and extracting image region features of the image data;

[0040] Specifically, the current image data is partitioned to obtain various regions of the graphic; optionally,

[0041] The target detection model is used to partition the image data and extract the regional feature set of each partition in the image, which is:

[0042]

[0043] Among them, R is the regional feature set of each partition of the image; m is the number of partitions; d v is the characteristic dimension of the regional feature; in this embodiment, d v =2048.

[0044] The regional features of each partition are mapped to new regional features through the fully connected layer. The feature dimension of the new regional features is d h =1024, take the average of each new regional feature to get the image regional feature

[0045] S200, obtaining text data in the current training data; and sequentially performing a contextual reasoning process based on modal information enhancement on each word in the text data to obtain corresponding semantic reasoning information;

[0046] In this embodiment, for a single word, the contextual reasoning process based on modal information enhancement (hereinafter referred to as the "contextual reasoning process") is specifically executed, such as Figure 2 As shown, including:

[0047] S201, obtaining hidden state features and visual attention features of the current word;

[0048] Specifically, Figure 3 As shown, this step includes the following sub-steps:

[0049] S201A, obtaining the hidden state feature corresponding to the previous word;

[0050] The previous word is a word located in the previous order of the current word in the current text data, that is, the word corresponding to the context reasoning process in the previous cycle;

[0051] The hidden state feature of the previous word is the hidden state feature obtained when the context reasoning process is performed on the previous word.

[0052] Specifically, before executing the context reasoning process of the current cycle, the hidden state feature corresponding to the previous word is extracted from the hidden state feature sequence stored in the hidden state memory pool.

[0053] S201B, obtaining a word embedding feature of the current word; obtaining a hidden state feature of the current word based on the word embedding feature of the current word, the image region feature, and a hidden state feature corresponding to a previous word;

[0054] In a specific embodiment, a word embedding function is used to extract the word embedding feature corresponding to the current word, which is:

[0055] e t =embed(w t )

[0056] Among them, w t is the current word, embed() represents the word embedding function, e t is the word embedding feature of the current word, and

[0057] The word embedding feature of the current word and the image region feature are concatenated to obtain the concatenated feature of the current word, which is:

[0058] x t =[e t ;I g ]

[0059] Among them, x t For the current word w t The concatenation feature of the current word and the hidden state feature of the previous word are obtained by using a long short-term memory neural network, which is:

[0060] h t =LSTM(x t ,h t-1 )

[0061] Among them, h t is the hidden state feature of the current word; LSTM() is a long short-term memory neural network.

[0062] S201C, based on the hidden state feature of the current word and the image region feature, using an attention mechanism to obtain a visual attention feature of the current word;

[0063] The traditional attention mechanism is adopted to perform feature interaction on the hidden state features of the current word and the image region features.

[0064] Specifically, the hidden state feature h of the current word t As the query vector, the image region feature R is used as the key vector and the value vector, and is input into the traditional attention mechanism to calculate the visual attention feature related to the hidden state feature; that is, according to the correlation between the query vector and the key vector, a weight is assigned to the value vector, and the weighted value vectors are weighted summed to obtain the visual attention feature of the current word, which is:

[0065] a t =attention(R,h t )

[0066] Among them, attention() is the traditional attention mechanism; a t is the visual attention feature of the current word.

[0067] Optionally, before the traditional attention mechanism is used to perform feature interaction on the hidden state feature of the current word and the image region feature, the step further includes:

[0068] The self-attention mechanism is used to perform internal feature interaction on the image region features to obtain new image region features, so as to perform subsequent steps based on the new image region features, which are:

[0069] R s =self-attention(R)

[0070] Among them, R s Represents new image region features.

[0071] S202, based on the hidden state feature and the visual attention feature of the current word, correspondingly updating the hidden state feature memory pool and the visual attention feature memory pool to correspondingly obtain a current hidden state feature sequence and a current visual attention feature sequence;

[0072] Specifically, the hidden state feature h of the current word t Input to the top layer of the hidden state feature memory pool for storage, and delete the hidden state features stored in the bottom layer of the hidden state feature memory pool to update the hidden state feature memory pool, and use the feature sequence stored in the updated hidden state feature memory pool as the current hidden state feature sequence, that is:

[0073] H t ={h t ,h t-1 ,...,h t-k+1};

[0074] Among them, h t ,ht-1 ……、h t-k+1 are the hidden features in the updated hidden state feature memory pool; k is the number of features stored in the memory pool; optionally, k is 8.

[0075] And, the visual attention feature a of the current word t Input to the top layer storage of the visual attention feature memory pool, and delete the visual attention feature stored in the bottom layer of the visual attention feature memory pool to obtain an updated visual attention feature memory pool, and use the feature sequence stored in the updated visual attention feature memory pool as the current visual attention feature sequence, that is:

[0076] A t ={a t ,a t-1 ,...,a t-k+1}

[0077] Among them, a t 、a t-1 ……、a t-k+1 are the visual attention features in the updated visual attention feature memory pool; k is the number of features stored in the memory pool; optionally, k is 8.

[0078] In this embodiment, the capacity of the hidden state feature memory pool and the visual attention feature memory pool is the same, which is Used to store the corresponding features generated in the contextual reasoning process based on modal information enhancement in the loop iteration k times.

[0079] S203, obtaining a sequence feature of the current visual attention feature sequence; based on the sequence feature and the word embedding feature of the current word, constructing a hidden state feature query vector of the current word;

[0080] Optionally, the sequence feature includes the mean of the current visual attention feature sequence, that is, the mean of each feature in the current visual attention feature sequence.

[0081] In a specific implementation, the mean of the visual attention feature sequence is obtained, which is the mean of each visual attention feature in the current visual attention feature sequence; based on the mean and the word embedding feature of the current word, the first gate vector is constructed as:

[0082] g H =σ(W H (e t +mean(A t ))

[0083] Among them, g His the first gate vector, used to dynamically balance the semantic information obtained from the word embedding feature and the mean of the attention feature sequence; σ is sigmoid, which limits the feature to between 0 and 1, is the Hadamard product; mean(A t ) is the mean of the visual attention feature sequence.

[0084] Based on the first gate vector, the mean of the visual attention feature sequence and the word embedding feature of the previous word, a hidden state feature query vector of the current word is constructed as follows:

[0085]

[0086] in, The hidden state feature query vector for the current word.

[0087] S204, constructing a multimodal feature query vector of the current word based on the hidden state query vector of the current word and the hidden state feature of the current word;

[0088] Specifically, a gated linear unit is used, and the hidden state feature query vector of the current word and the hidden state feature of the current word are used as inputs of the gated linear unit to construct a multimodal feature query vector of the current word, which is:

[0089]

[0090] in, is the multimodal feature query vector of the current word; GLU() is the gated linear unit function; this function divides the input feature into two parts according to the dimension size, one part is used as information, and the other part is processed as a gate using the sigmoid function, and the multiplication result of the two is output; W M is the model parameter.

[0091] In a specific implementation, the GLU function is calculated using the following formula:

[0092]

[0093] Among them, A is taken from the first half dimension of the input feature M as information; B is taken from the second half dimension of the input feature M as a gate after being processed by the sigmoid function.

[0094] S205, performing temporal and semantic enhancement on the current hidden state feature sequence to obtain an enhanced hidden state feature sequence; and performing temporal and semantic enhancement on the current visual attention feature sequence to obtain an enhanced visual attention feature sequence;

[0095] In this embodiment, for the current hidden state feature sequence, based on the sequence position of each hidden state feature in the current hidden state feature sequence, the time sequence information of each hidden state feature is obtained; optionally, a pre-constructed position encoder is used to convert the sequence position of each hidden state feature into time sequence information;

[0096] Based on the temporal information of each hidden state feature and the word embedding feature of the current word, each hidden state feature is temporally and semantically enhanced to obtain each hidden state feature after temporal and semantic enhancement, which is:

[0097]

[0098] in, is the enhanced hidden state feature; p i is the hidden state feature h i In the current hidden state feature sequence H t The relative position in W he is a model parameter; pe() is a relative position encoder used to obtain the timing information of each hidden state feature.

[0099] The above enhancement process is performed on each of the hidden state features in the current hidden state feature memory pool to obtain the enhanced hidden state feature sequence, which is

[0100] For the current visual attention feature sequence, based on the sequence position of each visual attention feature in the current visual attention feature sequence, obtaining the time sequence information of each visual attention feature; optionally, using a pre-built position encoder to convert the sequence position of each visual attention feature into time sequence information;

[0101] Based on the time sequence information of each of the visual attention features and the word embedding feature of the current word, each of the visual attention features is semantically enhanced to obtain each of the visual attention features after semantic enhancement, which is:

[0102]

[0103] in, is the enhanced visual attention feature; p i Represents the visual attention feature a i In the current visual attention feature sequence A t The relative position in W ae is a model parameter; pe() is a relative position encoder used to obtain the timing information of each of the visual attention features.

[0104] In a specific implementation, pe() is calculated using the following formula:

[0105]

[0106] Among them, p i Indicates the relative position of each feature in the corresponding feature sequence; j is the dimension of the position encoding representation. When j is an odd number, f() is sin(), and when it is an even number, it is cos().

[0107] The above enhancement process is performed on each visual attention feature in the current visual attention feature sequence to obtain the enhanced visual attention feature sequence, which is

[0108] S206, obtaining a multimodal feature sequence of the current word based on the enhanced hidden state feature sequence and the enhanced visual attention feature sequence;

[0109] In a specific implementation, the enhanced hidden state feature sequence and the enhanced visual attention feature sequence are concatenated to obtain a multimodal feature sequence of the current word, which is:

[0110]

[0111] in, is the multimodal feature sequence of the current word.

[0112] It should be noted that, in other embodiments, the execution order of steps S206 to S207 may be earlier than the execution order of steps S203 to S205 or may be executed simultaneously, which is not limited here.

[0113] S207, based on the multimodal feature sequence of the current word and the multimodal feature query vector of the current word, using an attention mechanism to obtain a contextual reasoning feature of the current word;

[0114] The contextual inference feature of the current word is contextual comprehensive information associated with the word embedding information of the current word and the current image feature information.

[0115] Based on the multimodal feature sequence of the current word, a self-attention mechanism is used to perform feature interaction to obtain a new multimodal feature sequence; in a specific implementation, the self-attention mechanism is used to perform two feature interactions, which are:

[0116]

[0117] Among them, self-attention 2() indicates that the self-attention layer of the self-attention mechanism has two layers, which is used to realize two feature interactions; Represents a new multimodal feature sequence.

[0118] Based on the new multimodal feature sequence and the multimodal feature query vector of the current word, the traditional attention mechanism is used to obtain the contextual reasoning feature of the current word, which is:

[0119]

[0120] in, Infer features for the context of the current word.

[0121] S208, based on the contextual reasoning features of the current word and the visual attention features of the current word, obtain the semantic reasoning features of the current word; based on the semantic reasoning features of the current word, determine the semantic information with the highest correlation in the semantic table as the semantic reasoning information of the current word.

[0122] In a specific implementation, a gated linear unit is used to calculate the semantic reasoning feature of the current word, which is:

[0123]

[0124] Among them, out t is the semantic inference feature of the current word; W out are model parameters and satisfy

[0125] Based on a pre-constructed word semantic table, a normalized exponential function for solving multi-classification problems is adopted to select semantic information with the highest feature association weight in the semantic table as the semantic reasoning information of the current word.

[0126] y t =softmax(W v out t )

[0127] in, d w is the vocabulary size, softmax() is the normalized exponential function used to select the semantic information with the highest weight in the semantic table; t It is the semantic reasoning information of the current word.

[0128] For each word in the current text data, the above steps S201 to S208 are executed in sequence according to the arrangement order in the text data until exiting, thereby obtaining semantic reasoning information corresponding to each word.

[0129] S300, constructing a cross entropy loss of the current text data based on the semantic reasoning information of each word and the semantic label information of the current image, so as to optimize the image description model based on the cross entropy loss;

[0130] Specifically, based on the semantic reasoning information of each word and the corresponding semantic label information, the cross entropy loss corresponding to each word is constructed; and the cross entropy loss corresponding to each word is accumulated to obtain the cross entropy loss of the current text data, which is:

[0131]

[0132] Among them, L xe is the cross entropy loss of the current text data, which is used to maximize the probability of the model outputting the semantic label information of the current image, that is, constraining the distribution probability between the semantic reasoning information and the semantic label information to be similar. p() represents the probability distribution when the model outputs the semantic reasoning information, I is the current image, l is the total number of words in the current text data, Represents a word sequence consisting of the first word to the t-1th word; is the tth word in the text data.

[0133] S400, updating the current training data to perform the next model training process based on the updated current training data until exiting.

[0134] Specifically, new training data is obtained and the new training data is used to replace the current training data to update the current training data, and the next model training process is performed based on the updated current training data until exiting, thereby obtaining the trained image description model.

[0135] Optionally, the model training process, before executing step S200, further includes:

[0136] The feature vectors stored in the model are initialized to zero, including the hidden state features, the hidden state feature memory pool, and the feature sequences stored in the visual attention feature memory pool.

[0137] See also Figure 4 , showing a flow chart of a single model training process of the modality-enhanced image description model training method provided by the present invention in another embodiment.

[0138] like Figure 4 As shown, the image description model training method based on modality enhancement is used with Figure 1The methods shown are basically the same, except that, in this embodiment, the image description model training method based on modality enhancement, when performing a single model training process, when executing step S200, after obtaining the enhanced hidden state feature sequence and obtaining the enhanced visual attention feature sequence, further includes:

[0139] S209, obtaining a sequence feature of the current hidden state feature sequence; based on the sequence feature and the word embedding feature of the current word, constructing a visual attention feature query vector of the current word;

[0140] Optionally, the sequence feature includes the mean of the current hidden state feature sequence, that is, the mean of each feature in the current hidden state feature sequence.

[0141] In a specific implementation, the mean of each previous hidden state feature in the current hidden state feature sequence is obtained as the mean of the hidden state feature sequence; based on the mean and the word embedding feature of the current word, the second gate vector is constructed as:

[0142] g A =σ(W A (e t +mean(H t ))

[0143] In, g A is the second gate vector, which is used to dynamically balance the semantic information obtained from the word embedding feature and the mean of the hidden state feature sequence; σ is sigmoid, which limits the feature to between 0 and 1. is the Hadamard product; mean(H t ) is the mean of the hidden state feature sequence.

[0144] Based on the second gate vector, the mean of the hidden state feature sequence and the word embedding feature of the previous word, a visual attention feature query vector of the current word is constructed as follows:

[0145]

[0146] in, Query the visual attention feature vector for the current word.

[0147] S210, based on the enhanced hidden state feature sequence and the hidden state feature query vector, using an attention mechanism to obtain hidden state semantic features; and based on the enhanced visual attention feature and the visual attention feature query vector, using an attention mechanism to obtain visual semantic features;

[0148] In this embodiment, the enhanced hidden state feature sequence is input into the self-attention mechanism, and the internal features of the sequence are interacted for each of the enhanced hidden state features to obtain an interactively enhanced hidden state feature sequence, which is

[0149]

[0150] in, is a sequence of hidden state features enhanced for interaction.

[0151] Based on the interactively enhanced hidden state feature sequence and the hidden state feature query vector, the traditional attention mechanism is used to aggregate features to obtain the interactively enhanced hidden state semantic features, which are:

[0152]

[0153] Among them, the is the hidden state semantic feature.

[0154] And, the enhanced visual attention feature sequence is input into the self-attention mechanism, and the internal features of the sequence are interacted for each of the enhanced visual attention features to obtain an interactively enhanced visual attention feature sequence, which is

[0155]

[0156] in, Feature sequences for interactive enhancement of visual attention.

[0157] Based on the interactively enhanced visual attention feature sequence and the visual attention feature query vector, the traditional attention mechanism is used to aggregate features to obtain the interactively enhanced visual semantic features, which are:

[0158]

[0159] Among them, the is the visual semantic feature.

[0160] S211, based on the hidden state semantic features and the visual semantic features, a semantic alignment method is used to construct a semantic alignment loss of the current word.

[0161] Specifically, the hidden state semantic features after interactive enhancement and the visual semantic features after interactive enhancement are used to construct the semantic alignment loss of the current word using a semantic alignment method, so as to make the two features as close as possible in the semantic space, which is:

[0162]

[0163] Among them, Lalign is the current word semantic alignment loss. The Smooth-L1 function is composed of the L1 loss and the L2 loss, which can solve the non-smooth problem of L1 loss in the range of -1 to 1.

[0164] In this embodiment, for each word in the current text data, the above steps S201 to S211 are executed in sequence according to the arrangement order in the text data until exiting, thereby obtaining the semantic reasoning information corresponding to each word and the corresponding semantic alignment loss.

[0165] In this embodiment, the image description model training method based on modality enhancement, when executing a single model training process, when executing step S300, is:

[0166] S300, based on the semantic reasoning information of each word and the semantic label information of the current image, construct the cross entropy loss of the current text data; and based on the semantic alignment loss of each word, obtain the semantic alignment loss of the current text data; based on the cross entropy loss and semantic alignment loss of the current text data, construct the total model loss of the current text data, so as to optimize the image description model based on the total model loss.

[0167] Specifically, the total model loss of the current text data consists of two parts: the cross entropy loss and the semantic alignment loss, and the calculation formula is:

[0168] L=L xe +λL ALN

[0169] Among them, L is the total loss of the model; L xe is the cross entropy loss of the current text data; L ALN is the semantic alignment loss of the current text data; λ is used to balance the proportion of the cross entropy loss and the semantic alignment loss in the total loss of the model, and is optionally set to 1.

[0170] In this embodiment, the semantic alignment loss of the current text data is:

[0171]

[0172] Among them, L aln is the semantic alignment loss of each single time in the current text data.

[0173] In order to solve the problems existing in the prior art, the present invention provides an image description method in a second aspect, which is used to implement an image description process for an image to be described to obtain description information of the image.

[0174] See also Figure 5 , showing a flow chart of the image description method provided by the present invention in one implementation.

[0175] like Figure 5 As shown, the image description method comprises the following steps:

[0176] S10, constructing groups of training data based on sample data of image description, wherein a single group of training data includes image data and text data associated with the image data;

[0177] The text data includes pre-marked words; the words are arranged in sequence in the text data.

[0178] S20, training a preset image description model based on each group of training data and semantic label information corresponding to each group of training data to obtain a trained image description model;

[0179] The semantic tag information is the semantic information actually corresponding to each word in the text data.

[0180] Specifically, the preset image description model is trained using the above-mentioned modality-enhanced image description model training method to obtain a trained image description model.

[0181] Among them, the image description model includes a pre-constructed feature memory pool, including a hidden state memory pool and a visual attention feature memory pool; the hidden state memory pool is used to store the hidden state features of words; the visual attention feature memory pool is used to store the visual attention features of words.

[0182] The hidden state feature is a word semantic feature combined with an image partition feature; and the visual attention feature is an image feature combined with a word semantic feature.

[0183] S30, for the image to be described, using the trained image description model to perform contextual semantic information reasoning on the image to obtain semantic information of the input image.

[0184] In order to solve the problems existing in the prior art, the present invention also provides an electronic device in a third aspect, which includes: a processor, a memory, a transceiver, a communication interface and a system bus; the memory and the communication interface are connected to the processor and the transceiver through the system bus and complete communication with each other, the memory is used to store computer programs, the communication interface is used to communicate with other devices, and the processor and the transceiver are used to run computer programs so that the processing device executes the above-mentioned image description model training method based on modality enhancement or each step in the above-mentioned image description method.

[0185] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0186] In addition, the present invention also provides a computer-readable storage medium in a fourth aspect, on which a computer program is stored, and when the program is called by a processor, it implements the above-mentioned image description model training method based on modality enhancement or each step in the above-mentioned image description method.

[0187] Among them, the computer-readable storage medium can be a tangible device that can hold and store instructions used by the instruction execution device. The computer-readable storage medium can be, for example, (but not limited to) an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the above. More specific examples of computer-readable storage media (a non-exhaustive list) include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, and a mechanical encoding device.

[0188] The computer-readable program described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. A network adapter card or a network interface in each computing / processing device receives computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in a computer-readable storage medium in each computing / processing device.

[0189] In summary, the image description model training method based on modality enhancement and the image description method, device and computer storage medium proposed in the present invention; use the memory pool of each modality to store the feature sequence of the corresponding modality, and enhance the time and semantic information of each modal feature respectively, so as to realize multimodal contextual reasoning of each word in the text sentence based on the semantic and time association in the multimodal time feature sequence. This method can fully explore the correlation between the image visual features and the text language features, thereby enhancing the model reasoning ability and improving the accuracy of the model in understanding multimodal semantics; in addition, through multimodal semantic alignment, the model explicitly aligns the semantic information extracted from the visual and language time feature sequences at each moment, constrains the visual semantics and language semantics in the same semantic space, further enhances the model's ability to understand and capture visual semantic information, and can significantly improve the model's ability to understand multimodal semantics and perform effective reasoning, alleviating the semantic gap problem of multimodal tasks.

[0190] The above embodiments are merely illustrative of the principles and effects of the present invention, and are not intended to limit the present invention. Anyone familiar with the art may modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by a person of ordinary skill in the art without departing from the spirit and technical concept disclosed by the present invention shall still be covered by the claims of the present invention.

Claims

1. A method for training an image description model based on modality enhancement, characterized in that: Based on the training data and the semantic label information corresponding to the training data, repeatedly performing the model training process on the image description model to obtain a trained image description model; wherein the training data includes image data and text data associated with the image data; and the text data includes pre-marked words; When executing the model training process once, it includes: Based on the image data in the current training data, extracting image region features of the image data; Based on the text data in the current training data; for each word in the text data, sequentially perform a contextual reasoning process based on modal information enhancement to obtain semantic reasoning information corresponding to each word; Based on the semantic reasoning information corresponding to each word and the semantic label information of the current image, construct a cross entropy loss of the current text data, so as to optimize the image description model based on the cross entropy loss; The current training data is updated to perform the next model training process based on the updated current training data until exiting; wherein, The image description model includes a hidden state feature memory pool and a visual attention feature memory pool; when performing a single contextual reasoning process based on modal information enhancement, it includes: Acquire the hidden state feature and the visual attention feature of the current word; based on the hidden state feature and the visual attention feature of the current word, correspondingly update the hidden state feature memory pool and the visual attention feature memory pool to obtain a current hidden state feature sequence and a current visual attention feature sequence; Acquire the sequence feature of the current visual attention feature sequence; construct a hidden state feature query vector of the current word based on the sequence feature and the word embedding feature of the current word; Constructing a multimodal feature query vector of the current word based on the hidden state feature query vector of the current word and the hidden state feature; Performing temporal and semantic enhancement on the current hidden state feature sequence to obtain an enhanced hidden state feature sequence; and performing temporal and semantic enhancement on the current visual attention feature sequence to obtain an enhanced visual attention feature sequence; Based on the enhanced hidden state feature sequence and the enhanced visual attention feature sequence, a multimodal feature sequence of the current word is obtained; based on the multimodal feature sequence of the current word and the multimodal feature query vector of the current word, an attention mechanism is used to obtain contextual reasoning features of the current word; Based on the contextual reasoning features of the current word and the visual attention features of the current word, semantic reasoning information of the current word is obtained.

2. The method for training an image description model based on modality enhancement according to claim 1, characterized in that: The implementation method of obtaining the hidden state feature and visual attention feature of the current word includes: Get the hidden state features corresponding to the previous word and get the word embedding features of the current word; Based on the word embedding feature of the current word, the image region feature, and the hidden state feature corresponding to the previous word, obtaining the hidden state feature of the current word; and, Based on the hidden state features of the current word and the image region features, an attention mechanism is used to obtain the visual attention features of the current word.

3. The method for training an image description model based on modality enhancement according to claim 1, characterized in that: The implementation method of correspondingly updating the hidden state feature memory pool and the visual attention feature memory pool based on the hidden state feature and the visual attention feature of the current word includes: Input the hidden state feature of the current word into the top storage of the hidden state feature memory pool, and delete the hidden state feature stored in the bottom storage of the hidden state feature memory pool; and, The visual attention feature of the current word is input into the top-level storage of the visual attention feature memory pool, and the visual attention feature stored in the bottom-level storage of the visual attention feature memory pool is deleted.

4. The method for training an image description model based on modality enhancement according to claim 1, characterized in that: The performing time and semantic enhancement on the current hidden state feature sequence comprises: Based on the sequence position of each hidden state feature in the current hidden state feature sequence, using a first position encoder, obtaining the time sequence information of each hidden state feature; superimposing the time sequence information of each hidden state feature and the word embedding feature of the current word into the corresponding hidden state feature; and, The performing temporal and semantic enhancement on the current visual attention feature sequence comprises: Based on the sequence position of each of the visual attention features in the current visual attention feature sequence, a second position encoder is used to obtain the timing information of each of the visual attention features; the timing information of each of the visual attention features and the word embedding feature of the current word are superimposed on the corresponding visual attention feature.

5. The method for training an image description model based on modality enhancement according to claim 4, characterized in that: The first position encoder and the second position encoder are the same, and both include: Among them, p i Indicates the relative position of the feature in the corresponding feature sequence; j is the dimension of the position encoding representation. When j is an odd number, f() is sin(); when j is an even number, f() is cos(). h is the characteristic dimension of regional characteristics.

6. The method for training an image description model based on modality enhancement according to claim 1, characterized in that: The implementation method of acquiring the contextual reasoning feature of the current word by using the attention mechanism based on the multimodal feature sequence of the current word and the multimodal feature query vector of the current word includes: Based on the multimodal feature sequence of the current word, a new multimodal feature sequence is obtained by using a self-attention mechanism; Based on the new multimodal feature sequence and the multimodal feature query vector of the current word, a traditional attention mechanism is used to obtain contextual reasoning features of the current word.

7. The method for training an image description model based on modality enhancement according to claim 1, characterized in that: The implementation method of constructing the multimodal feature query vector of the current word includes: A gated linear unit is adopted, and the hidden state feature query vector of the current word and the hidden state feature of the current word are used as inputs of the gated linear unit to construct a multimodal feature query vector of the current word.

8. The method for training an image description model based on modality enhancement according to claim 1, characterized in that: When executing the model training process once, it also includes: Executing each of the contextual reasoning processes based on modal information enhancement in sequence to obtain the semantic alignment loss corresponding to each word; Based on the semantic alignment loss corresponding to each word, the semantic alignment loss of the current text data is obtained; based on the cross loss and the semantic alignment loss of the current text data, the total model loss of the current text data is constructed, so as to optimize the image description model based on the total model loss; Wherein, in the single execution of the contextual reasoning process based on modal information enhancement, after obtaining the enhanced hidden state feature sequence and obtaining the enhanced visual attention feature sequence, it also includes: Acquire the sequence feature of the current hidden state feature sequence; construct a visual attention feature query vector of the current word based on the sequence feature and the word embedding feature of the current word; Based on the enhanced hidden state feature sequence and the hidden state feature query vector, an attention mechanism is used to obtain hidden state semantic features; Based on the enhanced visual attention feature and the visual attention feature query vector, an attention mechanism is used to obtain visual semantic features; and, Based on the hidden state semantic features and the visual semantic features, a semantic alignment method is adopted to construct a semantic alignment loss of the current word.

9. The method for training an image description model based on modality enhancement according to claim 8, characterized in that: The implementation method of acquiring hidden state semantic features by using an attention mechanism based on the enhanced hidden state feature sequence and the hidden state feature query vector includes: Based on the enhanced hidden state feature sequence, a self-attention mechanism is used to obtain an interactively enhanced hidden state feature sequence; based on the interactively enhanced hidden state feature sequence and the hidden state feature query vector, a traditional attention mechanism is used to obtain an interactively enhanced hidden state semantic feature; and, The method for obtaining visual semantic features by using an attention mechanism based on the enhanced visual attention features and the visual attention feature query vector includes: Based on the enhanced visual attention feature sequence, a self-attention mechanism is used to obtain an interactively enhanced visual attention feature sequence; based on the interactively enhanced visual attention feature sequence and the visual attention feature query vector, a traditional attention mechanism is used to obtain the interactively enhanced visual attention feature.

10. An image description method, characterized in that: include: Construct each training data set based on sample data of image description; A single set of said training data includes image data, and text data associated with the image data; Based on each of the training data sets and the semantic label information corresponding to the training data, the preset model is trained using the modality-enhanced image description model training method according to any one of claims 1 to 9 to obtain a trained image description model; For the image to be described, the trained image description model is used to perform contextual semantic information reasoning to obtain semantic information of the input image.

11. An electronic device, characterized in that: include: Processor and memory; The memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory, so that the electronic device performs the modality-enhanced image description model training method as described in any one of claims 1 to 9 or the image description method as described in claim 10.

12. A computer storage medium storing a computer program, characterized in that: The computer program is executed by a processor to implement the image description model training method based on modality enhancement as described in any one of claims 1 to 9 or the image description method as described in claim 10.

Images