Image description model training method and system, and image description generation method

Abstract

The invention relates to the field of image processing, in particular to an image description model training method and system and an image description generation method, and the training method comprises the steps: inputting a target image into an image feature extractor to extract grid features and region features; in each layer of the encoder, the grid features and the regional features are fused and then encoded, and final codes are obtained; inputting the final code into a decoder for decoding to obtain the image description of the target image; and training the image description model according to a predefined loss function until a predetermined convergence standard is reached. According to the method, fine-grained details and coarse-grained information can be effectively integrated, and the difference between the grid features and the regional features is made up, so that the accuracy of generated image description is improved.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to a training method and system for an image description model, and a method for generating image description. Background Art

[0002] Image description, or image annotation, is a technology that converts image content into natural language descriptions. Image description is currently widely used in various scenarios, such as visual search, automatic image annotation, content recommendation, and assistance for the visually impaired. The accuracy of image description is directly related to the effectiveness of these applications and user experience, and has important practical significance.

[0003] Most existing image description methods are implemented using deep learning models. The most common method is to combine convolutional neural networks with Transformers, using convolutional neural networks to extract visual features of images, and then using Transformers to generate descriptions. However, the grid features extracted by convolutional neural networks usually focus on the global features and fine-grained details of the image, while ignoring coarse-grained information such as object information; and because the grid features are usually segmented and flattened before being input into the Transformer, this may lead to the loss of high-level semantic information centered on the object, resulting in inaccurate or incomplete image descriptions generated by existing image description methods. Summary of the invention

[0004] In order to solve the above problems, the present invention provides a method and system for training an image description model, and a method for generating image description.

[0005] The first aspect of the present invention discloses a method for training an image description model, comprising:

[0006] Inputting a sample into an image feature extractor to extract grid features and region features, the sample including a target image and its corresponding image description label;

[0007] At each layer of the encoder, the grid features and the regional features are fused and encoded to obtain the final code;

[0008] The final code is input into the decoder for decoding to obtain the image description of the target image;

[0009] The image description model is trained according to a predefined loss function until a predetermined convergence criterion is reached.

[0010] Furthermore, at each layer of the encoder, the grid features and the regional features are fused and encoded to obtain the final encoding, including:

[0011] For the first layer of the encoder, the regional features are injected into the grid features to obtain the first layer of enhanced grid features, the first layer of enhanced grid features are input into the first layer encoder to obtain the fused grid features corresponding to the first layer, and the regional features are used as the enhanced regional features corresponding to the first layer of the encoder;

[0012] For the i+1th layer of the encoder, extract global semantic information and fine-grained semantic information from the fused grid features output by the oth layer of the encoder and fuse them into the enhanced regional features corresponding to the i-th layer of the encoder to obtain the enhanced regional features of the i+1th layer, and inject the enhanced regional features of the i+1th layer into the fused grid features output by the i-th layer of the encoder to obtain the i+1th layer enhanced grid features, and input the i+1th layer enhanced grid features into the i+1th layer encoder to obtain the fused grid features corresponding to the i+1th layer;

[0013] The fused grid features output by the last layer of the encoder are used as the final encoding.

[0014] Furthermore, the i+1th layer enhanced grid features are input into the i+1th layer encoder to obtain the i+1th layer corresponding fused grid features, including:

[0015] Enhance the grid features of the i+1th layer Input the i+1th layer encoder to get the fused grid features corresponding to the i+1th layer

[0016]

[0017] Among them, FFN represents feedforward network, LayerNorm represents layer normalization operation, MMHLA represents multi-head multi-level attention, and the i+1th layer enhances grid features. for:

[0018]

[0019] in, represents the enhanced regional features corresponding to the i+1th layer of the encoder, α is the preset weight parameter, G represents the grid features, and R represents the regional features.

[0020] Furthermore, the global semantic information and the fine-grained semantic information extracted from the fused grid features outputted by the i-th layer of the encoder are fused into the enhanced regional features corresponding to the i-th layer of the encoder to obtain the enhanced regional features of the i+1th layer, including:

[0021] Enhanced regional features corresponding to the i+1th layer of the encoder for:

[0022]

[0023]

[0024] in, is the self-attention region feature corresponding to the i+1th layer encoder, is the normalized self-attention region feature corresponding to the i+1th layer encoder, is the normalized fused grid feature corresponding to the i+1th layer encoder, is the cross-attention region feature corresponding to the i+1th layer encoder, FFN represents the feedforward network, LayerNorm represents the layer normalization operation, and MMHLA stands for multi-head multi-level attention.

[0025] Furthermore, the multi-head multi-level attention MMHLA is:

[0026] MHMLA(Q,K,V)=(Concat(Head1,Head2,…,Head H ))W O ;

[0027] Head H =MultiLevelAttention(Q,K,V);

[0028] Among them, Concat represents the vector concatenation operation, H represents the number of attention heads in the multi-head attention, and Head H represents the output of the Hth attention head, W O is a trainable projection matrix, Q, K, V are the query matrix, key matrix and value matrix of the attention mechanism respectively, and MultiLevelAttention is multi-level attention:

[0029] MultiLevelAttention(Q,K,V)=FFN(LayerNorm(Q+(S+λC)));

[0030] Among them, LayerNorm represents the layer normalization operation, λ represents the weight factor, S is the spatial attention weight, and C is the channel attention weight:

[0031]

[0032] Among them, SpatialAtention represents spatial attention, d k is the scaling factor, Represents the matrix multiplication symbol, and ChannelAtention represents channel attention.

[0033] Furthermore, the final code is input into the decoder for decoding to obtain the image description of the target image, including:

[0034] The word vector w for the tth word in the image description to be generated t , the word vectors of the generated words are used to form the input matrix w in the order of the words ＜t :

[0035] w ＜t =[w0,…,w t-1 ];

[0036] Among them, w0 represents the predefined sentence start marker;

[0037] The input matrix is ​​fed into the decoder, and the output of the l+1th layer encoder for the tth word for:

[0038]

[0039] Among them, CrossAttention represents the multi-head cross attention module. represents the cross-attention word vector, represents the normalized cross-attention word vector, represents the normalized word vector, MMHSA represents the multi-head self-attention module with mask, represents a masked multi-head self-attention word vector, Represents the word vector matrix of the first t-1 words generated by the l-th layer decoder, and LayerNorm represents the layer normalization operation;

[0040] The output of the last layer decoder for the tth word As the tth word w in the image description t ;

[0041] After T time steps of word autoregression generation, a description sentence sequence of the target image is obtained.

[0042] Furthermore, the loss function is:

[0043]

[0044] Among them, θ represents the parameters of the image description model, p θ Represents the model at time t under the condition Generate words based on The conditional probability of , T represents the number of time steps required to generate the image description, is the image description label corresponding to the target image.

[0045] Furthermore, the generation method further includes:

[0046] The reinforcement learning model is used to fine-tune the image description model, using the CIDEr score as the reward function r, and performing self-criticism sequence training. The loss function of the reinforcement learning model is:

[0047]

[0048] Where E represents the expected value, u 1:T p θ Represents the probability of generating the corresponding image description from the 1st to the Tth time step;

[0049] The model parameter θ is adjusted based on the gradient descent method, and the sample gradient is calculated using the following gradient expression:

[0050]

[0051] Where b is the average reward obtained by the sampled sequence, K is the number of sampled sequences, and u k represents the kth sampling sequence, Represents the gradient of the current sample with respect to the image description model parameter θ.

[0052] A second aspect of the present invention discloses a method for generating an image description, comprising:

[0053] Obtaining an image description model according to any one of the image description model training methods disclosed in the first aspect of the present invention;

[0054] The image to be processed is input into the image description model to obtain the image description corresponding to the image to be processed.

[0055] The third aspect of the present invention discloses a training system for an image description model, comprising:

[0056] An extraction module, used for inputting a sample into an image feature extractor to extract grid features and region features, wherein the sample includes a target image and its corresponding image description label;

[0057] The encoding module is used to fuse the grid features and the regional features at each layer of the encoder and encode them to obtain the final code;

[0058] A decoding module, used for inputting the final code into a decoder for decoding to obtain an image description of a target image;

[0059] The training module is used to train the image description model according to a predefined loss function until a predetermined convergence criterion is reached.

[0060] The present invention can effectively integrate fine-grained details and coarse-grained information by injecting object-level information in regional features into grid features and extracting global information and fine-grained semantics from grid features into regional features, thus bridging the difference between grid features and regional features and improving the accuracy of the generated image description. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0062] Figure 1 is a flow chart of a training method for an image description model disclosed in an embodiment of the present invention;

[0063] Figure 2 It is a module diagram of a training system for an image description model disclosed in an embodiment of the present invention. DETAILED DESCRIPTION

[0064] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0065] The terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, device, or product end including a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units that are not listed, or may optionally include other steps or units inherent to these processes, methods, products, or ends.

[0066] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present invention. The appearance of the phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0067] The training method of the image description model implemented by the image description model of the present invention is as follows Figure 1 The following is a detailed explanation of the specific steps of the training method for implementing the image description model in conjunction with a specific embodiment:

[0068] 1. Data preparation

[0069] A data set including several samples is prepared, each sample including one target image and several description texts of the target image. In this embodiment, each sample target image has five description texts of the target image.

[0070] 2. Obtaining feature representation of samples

[0071] A pre-trained deep learning model is selected as the image feature extractor, such as residual neural network (ResNet50), improved residual neural network (ResNeXt-101), etc.

[0072] The target image is input into the image feature extractor, and the target image is evenly divided into n image blocks (i.e., grids) and features are extracted separately to obtain the grid features G = {g1, g2, …, g n}, g n Represents the features of the nth image block, with a dimension of d.

[0073] On the other hand, the image feature extractor also identifies the salient regions in the target image and extracts features for each salient region to obtain the regional features R = {r1, r2, …, r m}, where m represents the number of salient regions identified by the image feature extractor in the target image, and r m Represents the features of the mth salient region, with a dimension of d.

[0074] 3. Enhanced feature representation based on multi-scale feature fusion method

[0075] The present invention designs two multi-scale feature interaction operations to effectively make up for the differences between features and complement each other, supplementing the grid features with object-centric high-level semantic information lost during image segmentation. The multi-scale feature fusion method enhances the semantic understanding ability of the visual coding features ultimately used for decoding through multiple interactions of two different granularity image features, grid features and regional features.

[0076] The regional feature R is injected into the grid feature to obtain the enhanced grid feature, and the enhanced grid feature is input into the encoder to obtain the fused grid feature rich in salient object information and context information.

[0077] For the first layer of the encoder, the regional features are injected into the grid features to obtain the first layer enhanced grid features, and the first layer enhanced grid features are input into the first layer encoder to obtain the fused grid features corresponding to the first layer, and the regional features are used as the enhanced regional features corresponding to the first layer of the encoder.

[0078] For the i+1th layer of the encoder, the global semantic information and fine-grained semantic information are extracted from the fused grid features output by the i-th layer of the encoder and fused into the enhanced regional features corresponding to the i-th layer of the encoder to obtain the enhanced regional features of the i+1th layer, and the enhanced regional features of the i+1th layer are injected into the fused grid features output by the i-th layer of the encoder to obtain the i+1th layer enhanced grid features, and the i+1th layer enhanced grid features are input into the i+1th layer encoder to obtain the fused grid features corresponding to the i+1th layer.

[0079] The fused grid features obtained by the i+1th layer encoder for:

[0080]

[0081] Among them, FFN represents feedforward network, LayerNorm represents layer normalization operation, and MHMLA stands for multi-head multi-level attention. is the enhanced grid feature corresponding to the i+1th layer encoder:

[0082]

[0083] in, represents the enhanced regional features corresponding to the i+1th layer of the encoder, and α is the preset weight parameter. In order to accelerate the convergence of the model and improve the generalization ability of the model, G, R, After layer normalization, it can be input into multi-head multi-level attention for processing.

[0084] Enhanced regional features corresponding to the i+1th layer of the encoder for:

[0085]

[0086]

[0087] in, is the self-attention region feature corresponding to the i+1th layer encoder, is the normalized self-attention region feature corresponding to the i+1th layer encoder, is the normalized fused grid feature corresponding to the i+1th layer encoder, is the cross-attention region feature corresponding to the i+1th layer encoder.

[0088] MHMLA stands for Multi-head Multi-level Attention:

[0089] MHMLA(Q,K,V)=(Concat(Head1,Head2,…,Head H ))W O ;

[0090] Head H =MultiLevelAttention(Q,K,V);

[0091] Among them, Concat represents the vector concatenation operation, H represents the number of attention heads in the multi-head attention, and Head H represents the output of the Hth attention head, W O is a trainable projection matrix, Q, K, V are the query matrix, key matrix and value matrix of the attention mechanism respectively, and MultiLevelAttention is multi-level attention:

[0092] MultiLevelAttention(Q,K,V)=FFN(LayerNorm(Q+(S+λC)));

[0093] Among them, LayerNorm represents the layer normalization operation, λ represents the weight factor, S is the spatial attention weight, and C is the channel attention weight.

[0094] The present invention inputs visual features into spatial attention and channel attention respectively to simultaneously capture spatial dependency and channel dependency, and weights the obtained spatial attention weight S and channel attention weight C to enhance the semantic understanding ability of the final weighted attention weight.

[0095]

[0096] Among them, SpatialAtention represents spatial attention, d k is the scaling factor, Represents the matrix multiplication symbol, and ChannelAtention represents channel attention.

[0097] The fused grid features obtained by the encoder of the last layer are used as the final encoding output: the final feature M.

[0098] 4. Generate image description

[0099] The final feature M output by the encoder is input into the decoder for decoding to generate a description of the input image. Each decoder layer consists of a multi-head self-attention module with mask, a multi-head cross-attention module, and a feed-forward layer.

[0100] The word vector for the tth word in the image description to be generated The word vectors of the generated words are used to form the input matrix w in the order of the words. ＜t :

[0101] w ＜t =[w0,…,w t-1 ];

[0102] Wherein, w0 represents a predefined sentence start marker, which is defined as “ <bos>”.

[0103] The input matrix is ​​fed into the decoder, and the output of the l+1th layer encoder for the tth word for:

[0104]

[0105] Among them, CrossAttention represents the multi-head cross attention module. represents the cross-attention word vector, represents the normalized cross-attention word vector, represents the normalized word vector, MMHSA represents the multi-head self-attention module with mask, represents a masked multi-head self-attention word vector, Represents the word embedding matrix of the first t-1 words generated by the l-th layer decoder.

[0106] The output of the last layer of decoder for the tth word Will be used to predict the tth word w t :

[0107]

[0108] Where L represents the number of decoder layers.

[0109] After T time steps of word autoregression generation, the description sentence sequence W = {w1,w2,…,w t ,…,w T }.

[0110] 5. Model Training

[0111] The model training process is divided into two stages. First, using the cross entropy loss To optimize the image description model:

[0112]

[0113] Among them, θ represents the parameters of the image description model, p θ Represents the model at time t under the condition Generate words based on The conditional probability of , T represents the number of time steps required to generate the description, is the image description label corresponding to the target image, that is, the real word sequence annotated by humans.

[0114] Secondly, reinforcement learning is used to fine-tune the image description model, using CIDEr (Consensus-based Image Description Evaluation) score as the reward function r() and performing self-criticism sequence training.

[0115]

[0116] Where E represents the expected value, u 1:T p α Represents the probability of generating the corresponding image description from the 1st to the Tth time step;

[0117] The model parameter θ is adjusted based on the gradient descent method, and the sample gradient is calculated using the following gradient expression:

[0118]

[0119] Where b is the average reward obtained by the sampled sequence, K is the number of sampled sequences, and u k represents the kth sampling sequence, Represents the gradient of the current sample with respect to the image description model parameter θ.

[0120] Based on the above two stages, the training of the image description model is completed to obtain a trained image description model.

[0121] like Figure 2 As shown, the present invention also provides a training system for an image description model, comprising:

[0122] An extraction module, used for inputting a sample into an image feature extractor to extract grid features and region features, wherein the sample includes a target image and its corresponding image description label;

[0123] The encoding module is used to fuse the grid features and the regional features at each layer of the encoder and encode them to obtain the final code;

[0124] A decoding module, used for inputting the final code into a decoder for decoding to obtain an image description of a target image;

[0125] The training module is used to train the image description model according to a predefined loss function until a predetermined convergence criterion is reached.

[0126] The present invention provides a method for generating image description, which inputs an image to be described into a trained image description model to obtain a description corresponding to the image to be described.

[0127] In summary, the training method and system of an image description model and the method for generating image description disclosed in the present invention can effectively integrate fine-grained details and coarse-grained information, make up for the difference between grid features and regional features, and thus improve the accuracy of the generated image description. Therefore, the present invention effectively overcomes various shortcomings in the prior art and has a high industrial utilization value.

[0128] The above embodiments are merely illustrative of the principles and effects of the present invention, and are not intended to limit the present invention. Anyone familiar with the art may modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by a person of ordinary skill in the art without departing from the spirit and technical concept disclosed by the present invention shall still be covered by the claims of the present invention.< / bos>

Claims

1. A training method for an image description model, characterized in that: The image description model is used to generate an image description for an image, and the image description model includes an image feature extractor, an encoder and a decoder. The training method includes: Inputting a sample into an image feature extractor to extract grid features and region features, the sample including a target image and its corresponding image description label; At each layer of the encoder, the grid features and the regional features are fused and encoded to obtain the final code; The final code is input into the decoder for decoding to obtain the image description of the target image; The image description model is trained according to a predefined loss function until a predetermined convergence criterion is reached.

2. The method for training an image description model according to claim 1, characterized in that: At each layer of the encoder, the grid features and the regional features are fused and encoded to obtain the final encoding, including: For the first layer of the encoder, the regional features are injected into the grid features to obtain the first layer of enhanced grid features, the first layer of enhanced grid features are input into the first layer encoder to obtain the fused grid features corresponding to the first layer, and the regional features are used as the enhanced regional features corresponding to the first layer of the encoder; For the i+1th layer of the encoder, extract global semantic information and fine-grained semantic information from the fused grid features output by the i-th layer of the encoder and fuse them into the enhanced regional features corresponding to the i-th layer of the encoder to obtain the enhanced regional features of the i+1th layer, and inject the enhanced regional features of the i+1th layer into the fused grid features output by the i-th layer of the encoder to obtain the i+1th layer enhanced grid features, and input the i+1th layer enhanced grid features into the i+1th layer encoder to obtain the fused grid features corresponding to the i+1th layer; The fused grid features output by the last layer of the encoder are used as the final encoding.

3. The method for training an image description model according to claim 2, characterized in that: Input the i+1th layer enhanced grid features into the i+1th layer encoder to obtain the i+1th layer corresponding fused grid features, including: Enhance the grid features of the i+1th layer Input the i+1th layer encoder to get the fused grid features corresponding to the i+1th layer Among them, FFN represents feedforward network, LayerNorm represents layer normalization operation, MMHLA represents multi-head multi-level attention, and the i+1th layer enhances grid features. for: in, represents the enhanced regional features corresponding to the i+1th layer of the encoder, α is the preset weight parameter, G represents the grid features, and R represents the regional features.

4. The method for training an image description model according to claim 2, characterized in that: The global semantic information and fine-grained semantic information extracted from the fused grid features output by the i-th layer of the encoder are fused into the enhanced regional features corresponding to the i-th layer of the encoder to obtain the enhanced regional features of the i+1th layer, including: Enhanced regional features corresponding to the i+1th layer of the encoder for: in, is the self-attention region feature corresponding to the i+1th layer encoder, is the normalized self-attention region feature corresponding to the i+1th layer encoder, is the normalized fused grid feature corresponding to the i+1th layer encoder, is the cross-attention region feature corresponding to the i+1th layer encoder, FFN represents the feedforward network, LayerNorm represents the layer normalization operation, and MMHLA stands for multi-head multi-level attention.

5. The method for training an image description model according to any one of claims 3 or 4, characterized in that: The multi-head multi-level attention MHMLA is: MHMLA(Q,K,V)=(Concat(Head1,Head2,…,Head H ))W O ; Head H =MultiLevelAttention(Q,K,V); Among them, Concat represents the vector concatenation operation, H represents the number of attention heads in the multi-head attention, and Head H represents the output of the Hth attention head, W O is a trainable projection matrix, Q, K, and V are the query matrix, key matrix, and value matrix of the attention mechanism, respectively, and MultiLevelAttention is multi-level attention: MultiLevelAttention(Q,K,V)=FFN(LayerNorm(Q+(S+λC))); Among them, LayerNorm represents the layer normalization operation, λ represents the weight factor, S is the spatial attention weight, and C is the channel attention weight: Among them, SpatialAtention represents spatial attention, d k is the scaling factor, Represents the matrix multiplication symbol, and ChannelAtention represents channel attention.

6. The method for training an image description model according to claim 1, characterized in that: The final code is input into the decoder for decoding to obtain the image description of the target image, including: The word vector w for the tth word in the image description to be generated t , the word vectors of the generated words are used to form the input matrix w in the order of the words <t : In ＜t =[w0,…,w t-1 ]; Among them, w0 represents the predefined sentence start marker; The input matrix is ​​fed into the decoder, and the output of the l+1th layer encoder for the tth word for: Among them, CrossAttention represents the multi-head cross attention module. represents the cross-attention word vector, represents the normalized cross-attention word vector, represents the normalized word vector, MMHSA represents the multi-head self-attention module with mask, represents a masked multi-head self-attention word vector, Represents the word vector matrix of the first tv1 words generated by the l-th layer decoder, and LayerNorm represents the layer normalization operation; The output of the last layer decoder for the tth word As the tth word w in the image description t ; After T time steps of word autoregression generation, a description sentence sequence of the target image is obtained.

7. The method for training an image description model according to claim 1, characterized in that: The loss function is: Among them, θ represents the parameters of the image description model, and pθ represents the model under the condition at time t. Generate words based on The conditional probability of , T represents the number of time steps required to generate the image description, is the image description label corresponding to the target image.

8. The method for training an image description model according to claim 1, characterized in that: The generating method further comprises: The reinforcement learning model is used to fine-tune the image description model, using the CIDEr score as the reward function r, and performing self-criticism sequence training. The loss function of the reinforcement learning model is: Where E represents the expected value, u 1：T p θ Represents the probability of generating the corresponding image description from the 1st to the Tth time step; The model parameter θ is adjusted based on the gradient descent method, and the sample gradient is calculated using the following gradient expression: Where b is the average reward obtained from the sampled sequence, K is the number of sampled sequences, and u k represents the kth sampling sequence, Represents the gradient of the current sample with respect to the image description model parameter θ.

9. A method for generating an image description, characterized in that: Obtaining an image description model according to the training method of an image description model as described in any one of claims 1 to 8; The image to be processed is input into the image description model to obtain the image description corresponding to the image to be processed.

10. A training system for an image description model, characterized in that: The image description model is used to generate an image description for an image, and the training system includes: An extraction module, used for inputting a sample into an image feature extractor to extract grid features and region features, wherein the sample includes a target image and its corresponding image description label; The encoding module is used to fuse the grid features and the regional features at each layer of the encoder and encode them to obtain the final code; A decoding module, used for inputting the final code into a decoder for decoding to obtain an image description of a target image; The training module is used to train the image description model according to a predefined loss function until a predetermined convergence criterion is reached.

Images