Image description generation method based on semantic content guidance and tensor product unbinding

Abstract

The application discloses a kind of based on semantic content guide and tensor product unbinding image description generation method, comprising:1) extracting global feature and local feature set to image data, extracting global feature to text data;2) based on the global feature of image data and text data, obtain image data semantic information as semantic content by embedding space alignment mode;3) based on semantic content, the local feature set of image data is enhanced by multi-head attention mechanism, and enhanced local feature set is obtained;4) based on enhanced local feature set fusion, obtain the tensor product representation of image data;5) based on tensor product unbinding from tensor product representation extract the content component of predicted vocabulary, and calculate vocabulary probability distribution, obtain predicted vocabulary, and all predicted vocabulary is spliced to obtain the complete description of image data.The application compared with other methods, performance is improved obviously, generation description semantic accuracy is obviously improved, and interpretability is more optimal.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing and natural language generation, and particularly to an image description generation method based on semantic content guidance and tensor product unbinding. BACKGROUND

[0002] With the further development of artificial intelligence technology, the interdisciplinary research of exploring multi-modal intelligence has attracted widespread attention. As a classic cross-modal research, image description generation aims to imitate the process of human looking at pictures and speaking, and realizes the conversion of semantic information from the image modality to the text modality. In order to generate a complete description that conforms to the main visual content of the image, the computer needs to fully learn the visual understanding ability of human beings and the fluent language expression ability of human beings, which makes the image description generation task extremely challenging. As a cross-modal bridge between computer vision and natural language processing, image description generation research not only promotes the progress of multi-modal learning, but also constantly provides new ideas for visual intelligence and language intelligence research, which has important academic value. In addition, the image description generation task itself has broad prospects in applications that need to balance visual understanding and language expression, such as voice assistants to help visually impaired people avoid surrounding dangers, intelligent hardware to assist early childhood education, and multimedia data managers in mobile terminals.

[0003] The current mainstream image description generation method only obtains local visual information and target interaction information in the local field of view from the local area. This local sensitivity limits the global understanding of the model, resulting in the inability to implement the logical thinking of first overviewing based on the global perspective and then targeted deep understanding in the process of human looking at pictures and speaking, and finally easily generating simple descriptions that are pieced together from visual vocabulary. On the other hand, the language model relied on by many current methods is affected by dataset bias and tends to use high-frequency phrases or high-frequency sentences, resulting in the generated description being easily repetitive and not consistent with the true content of the image. These problems all reflect that there is still room for improvement in the semantic accuracy of the current image description generation model. SUMMARY

[0004] The purpose of the present application is to overcome the shortcomings and deficiencies of the prior art, and to provide an image description generation method based on semantic content guidance and tensor product unbinding. The method obtains image semantic content through embedding space alignment to provide a global perspective for model understanding of the image, guides the model to conduct deep understanding based on the structural information of the visual target, and introduces a tensor product unbinding operation to design a word prediction process with explainability, thereby improving the problem that the image description generation model easily generates descriptions that are pieced together from simple visual vocabulary and the decoding process relies on high-frequency expression templates, and improving the semantic accuracy of the generated description.

[0005] To achieve the above objectives, the technical solution provided by this invention is: an image description generation method based on semantic content guidance and tensor product unbinding, comprising the following steps:

[0006] 1) Extract global and local feature sets from image data, and extract global features from text data;

[0007] 2) Based on the global features of image and text data, semantic information of image data is obtained as semantic content through embedding spatial alignment;

[0008] 3) Based on semantic content, feature enhancement is performed on the local feature set of image data through a multi-head attention mechanism to obtain an enhanced local feature set;

[0009] 4) Based on the enhanced local feature set, the tensor product representation of the image data is obtained by fusion;

[0010] 5) Extract the content components of the predicted words from the tensor product representation based on tensor product unbinding, calculate the word probability distribution, obtain the predicted words, and concatenate all the predicted words to obtain a complete description of the image data.

[0011] Further, in step 1), the global and local feature sets of the image data are extracted using the ResNet101 convolutional neural network and the Faster R-CNN object detection model, respectively, and the global features of the text data are extracted using the Long Short-Term Memory (LSTM) network. Specifically, given an image dataset, each image in the dataset is scaled to a size of 224×224, and then input into the ResNet101 convolutional neural network and the Faster R-CNN object detection model, respectively. The output of the ResNet101 average pooling layer is taken as the global feature v. g Take the set of n region features extracted by Faster R-CNN As a local feature set, where v i Let represent the i-th local feature of the image data; for each text in the text dataset, encode its words into word embedding vectors using a word embedding matrix, thus encoding each text into a sequence of word embedding vectors. Where x j Let represent the embedding vector of the j-th word, and m represent the number of words in the text. Then, the sequence of word embedding vectors of the text is sequentially input into the LSTM to obtain the updated hidden state vector as the global feature t of the text data. g .

[0012] Furthermore, in step 2), through the mapping matrix P s global features v of image data g Global features t embedded in text data gthe semantic space where the image data embedding feature v s = P s v g , based on the triplet loss L triplet pulls the image data embedding feature v s closer to the global feature t g of the matching text data, and pushes the image data embedding feature v s farther away from the global feature t of the non-matching text data, and the calculation is as follows:

[0013]

[0014] In the formula, max(·) represents the maximum value function, cos(·) represents the cosine similarity function, a is a hyperparameter, and the obtained image embedding feature v s is the semantic content of the image.

[0015] Further, in step 3), first, the semantic content v s is fused with the local feature set V of the image data to obtain a semantic initialized local feature set V where the i-th semantic initialized local feature is calculated as follows:

[0016]

[0017] In the formula, W fuse is a fully connected layer for dimension reduction, and [·,·] is a concatenation operation; then the local feature set V of the image data and the semantic initialized local feature set V s are respectively initialized to obtain the query set Q, the key set K, and the value set The calculation is as follows:

[0018]

[0019] In the formula, W Q ,W K , are parameter matrices; then based on the multi-head attention mechanism, the above three sets are embedded into h different subspaces for related relationship calculation and mutual fusion of structural information, and the i'-th subspace query set Q i′ , the key set K i′ , and the value set The calculation process is as follows:

[0020]

[0021] In the formula, is the mapping matrix of the i'th subspace, i.e., the i'th head of the relation R i′ is calculated as where d is a scale factor to avoid numerical overflow, and the attention of the i'th head is i′ is calculated as The attention of all heads is concatenated to obtain the enhanced local feature set

[0022]

[0023] where MultiHead(·) represents the multi-head attention mechanism, and head h is the attention of the h-th head, and W o is a parameter matrix, and the enhanced local feature set The i-th enhanced local feature in the set is

[0024] Further, in step 4), the attention LSTM is denoted as LSTM att At the prediction time t, based on the word embedding vector x t-1 , the context vector c t-1 and the average of the enhanced local features obtain the word context information at the current prediction time t The calculation process is as follows:

[0025]

[0026] where and are the memory cells of the attention LSTM at the prediction times t and t-1, respectively, is the word context information at the time t-1; then the semantic content v s and the word context information are fused to obtain the context query vector q t :

[0027]

[0028] where W q is a fully connected layer for dimension reduction; then the enhanced local feature set is queried using the context query vector q t to obtain the intermediate representation of the image modality to the text modality at the prediction time t:

[0029]

[0030] where s tIt contains visual content and visual structure information and is defined as the tensor product representation of image data.

[0031] Furthermore, step 5) includes the following steps:

[0032] 5.1) Extracting the content components of predicted words from tensor product representations based on tensor product unbinding: First, extract the content components of predicted words from tensor product representations in vector form s t Dimensionality recovery is achieved by projecting the matrix onto a second-order tensor space, denoted as S. t =diag(s t )P e Where diag(·) is the operation to convert to a diagonal matrix, P e The mapping matrix is ​​used; then a structure matrix U is learned, based on the context query vector q. t and dimensionality recovery of the tensor product representation s t Let S be the tensor product representation. t Calculate the probability distribution a of its structural components. t , represented as a t =softmax(P u [q t ,s t ]), where P u The mapping matrix is ​​used to obtain the unbinding vector u. t , represented as u t =a t U; meanwhile, to ensure the orthogonality of the structure matrix and the sparsity of the probability distribution of the structure components, an unbinding loss L is defined. u for:

[0033]

[0034] In the formula, ||·|| F Let λ denote the F-norm, ||·||1 denote the 1-norm, and λ u and λ a Here, I is the hyperparameter and the identity matrix; the unbinding operation based on tensor product uses the unbinding vector u. t The tensor product representation S after dimension recovery t Extract the content vector f of the predicted words t , represented as f t =S t u t ;

[0035] 5.2) Content vector f based on predicted vocabulary t Perform context enhancement to obtain the context vector c at prediction time t. t :

[0036] c t =W i[q t ,f t ]⊙σ(W g [q t ,f t ])

[0037] where σ(·) is a sigmoid activation function, is a dot product operation, W i and W g are fully connected layers for dimension reduction;

[0038] 5.3) Calculate the probability distribution p(y t |y t , θ) of the predicted vocabulary at the prediction time t based on the context vector c 1:t-1 :

[0039] p(y t |y 1:t-1 , θ) = softmax(W Σ c t )

[0040] where y t represents the predicted vocabulary obtained at the prediction time t, y 1:t-1 represents all predicted vocabularies obtained from 1 to t-1 at the prediction time, θ is a model parameter, and W ∑ is a fully connected layer; the probability distribution of the predicted vocabulary is constrained based on cross-entropy L xe :

[0041]

[0042] where v and v are the vocabulary of the training set text data, and log(·) is a logarithmic function;

[0043] 5.4) Select the vocabulary with the maximum probability at each prediction time to splice into a complete description.

[0044] Compared with the prior art, the present application has the following advantages and beneficial effects:

[0045] 1. The present application proposes to use multi-modal data based on embedding space alignment to obtain the semantic content of image data, providing an effective global perspective for the image description generation process, and simply and effectively avoiding the semantic deviation problem in the image understanding and vocabulary prediction process.

[0046] 2. The present application proposes an intermediate representation generation method that combines visual content and description structure information, which benefits from the guidance of semantic content to enhance visual feature structure information, which makes the obtained intermediate representation can be modeled through tensor product, and also makes the intermediate representation comply with the global semantics and embodies the image structure, showing deep semantic understanding.

[0047] 3. The present application proposes a vocabulary prediction process with explainability, based on the tensor product unbinding operation, which simulates the process of language organization based on vocabulary structure information in the process of human looking at pictures and speaking, effectively improving the problem of dependence on high-frequency expression templates in the vocabulary prediction process of existing models due to data set bias.

[0048] 4. Compared with other image description generation methods, the performance of the method of the present application is significantly improved, the semantic accuracy of the generated description is significantly improved, and the overall method is more explainable. BRIEF DESCRIPTION OF DRAWINGS

[0049] Figure 1 is the logical flowchart of the method of the present application.

[0050] Figure 2 is the logical flowchart of the vocabulary prediction process based on tensor product unbinding proposed by the method of the present application. DETAILED DESCRIPTION

[0051] The present application will be further described in detail below in conjunction with the embodiments and drawings, but the embodiments of the present application are not limited thereto.

[0052] As shown in Figure 1 , the present embodiment provides an image description generation method based on semantic content guidance and tensor product unbinding, comprising the following steps:

[0053] 1) Given an image description data set, features are extracted from image data and text data respectively. For the image data set, each image therein is scaled to a size of 224x224, and then input into a convolutional neural network ResNet101 and a target detection model Faster R-CNN respectively, taking the output of the average pooling layer of ResNet101 as the global feature v g , and taking the set composed of n regional features extracted by Faster R-CNN as the local feature set, wherein v i represents the i-th local feature of the image data; as for the text data set, each text in the text data set is taken, the vocabulary therein is encoded into a word embedding vector through a word embedding matrix, so as to encode each text into a word embedding vector sequence , wherein x j represents the j-th word embedding vector, and m represents the number of words in the text, and then the word embedding vector sequence of the text is input into an LSTM model in turn, and the updated hidden state vector is taken as the global feature t g .

[0054] ​2) Based on the global features of the image data and the text data constructed in the previous step, the semantic information of the image data is obtained as semantic content through embedding space alignment. First, the global feature v of the image data is embedded into the semantic space where the global feature t of the text data is located, to obtain the image data embedding feature v s g g s =P s v g , and then based on the triple loss L triplet , the distance between the image data embedding feature v and the global feature t of the matching text data is narrowed, and the distance between the image data embedding feature v and the global feature t of the non-matching text data is widened, which is calculated as follows: s g s

[0055]

[0056] where max(·) represents the maximum value function, cos(·) represents the cosine similarity function, and a is a hyperparameter. The obtained image embedding feature v s is the semantic content of the image.

[0057] 3) Based on the semantic content obtained in the previous step, the local feature set of the image data is enhanced through the multi-head attention mechanism to obtain the enhanced local feature set. First, the semantic content v is fused with the local feature set V of the image data to obtain the semantic initialized local feature set s where the i-th semantic initialized local feature is calculated as follows:

[0058]

[0059] where W fuse is a fully connected layer for dimension reduction, and [·,·] is a concatenation operation; then the local feature set V of the image data and the semantic initialized local feature set V s are used to initialize the query set Q, the key set K and the value set V respectively, which are calculated as follows:

[0060]

[0061] where W Q ,W K , ​​​​​​​​The parameter matrix is ​​used; then, based on the multi-head attention mechanism, the above three sets are embedded into h different subspaces for correlation calculation, and the structural information of visual features is supplemented by feature fusion. The query set Q of the i′-th subspace is... i′ Key set K i′ and value set The calculation process is as follows:

[0062]

[0063] In the formula, Let R be the mapping matrix of the i′-th subspace; the i′-th subspace is the correlation R of the i′-th head. i′ Calculated as Where d is a scaling factor to avoid excessively large values, followed by the attention head of the i′th head. i′ The calculation is expressed as The attention calculation results of all heads are concatenated to obtain the enhanced local feature set.

[0064]

[0065] Here, MultiHead(·) represents the multi-head attention mechanism, where head h For the attention of the h-th head, W o For the parameter matrix, enhance the local feature set The i-th enhanced local feature is

[0066] 4) Fuse the enhanced local feature sets obtained in the previous step to obtain the tensor product representation of the image data. First, construct an attention LSTM, denoted as LSTM. att At prediction time t, based on the word embedding vector x at time t-1 t-1 and context vector c t-1 and enhance the mean of local features Obtain the lexical context information at the current prediction time t The calculation process is as follows:

[0067]

[0068] In the formula, and These are the memory units of the attention LSTM at prediction times t and t-1, respectively. The lexical context information at time t-1; then the semantic content v s and vocabulary context information The context query vector q is obtained by fusion. t :

[0069]

[0070] where W q is the reduced dimension full connection layer; then the enhanced local feature set is processed based on the multi-head attention mechanism The context query vector q t is used to query to obtain the intermediate representation of the image modality to the text modality at the prediction time t:

[0071]

[0072] Benefiting from the completion of the local feature structure information in the previous step, the intermediate representation s t at this time contains both visual content and visual structure information. Combined with the definition of the tensor product, which is a symbolic structure representation of the binding of content and structure, the obtained intermediate representation is defined as the tensor product representation of the image data.

[0073] 5) The unbinding operation of the tensor product refers to that, for the tensor product representation composed of binding of content vectors and structure vectors, if the unbinding vector corresponding to the structure vector is obtained, the corresponding content vector of the structure vector can be extracted from the tensor product through multiplication operation, and when the structure vectors are mutually orthogonal, the structure vector itself is equivalent to the corresponding unbinding vector. Therefore, the content vector of the predicted word can be extracted from the tensor product representation obtained in the previous step based on the tensor product unbinding, and the word probability distribution is calculated to obtain the predicted word, and finally all the predicted words are spliced to obtain the complete description of the image data. The specific logic flow is shown in Figure 2 , including the following steps:

[0074] 5.1) Extract the content component of the predicted word from the tensor product representation based on the tensor product unbinding. First, project the vector form of the tensor product representation s t to the second-order tensor space through matrix mapping to realize dimension recovery, represented as S t = diag(s t )P e , where diag(·) is the operation of converting to a diagonal matrix, P e is the mapping matrix; then learn a structure matrix U based on the context query vector q t and the tensor product representation s t before dimension recovery, calculate the probability distribution a t of the structure component of the tensor product representation S t , represented as a t = softmax(P u [q t ,s t ]), where P u is the mapping matrix, and then obtain the unbinding vector ut , denoted as u t t U; meanwhile, to ensure the orthogonality of the structure matrix and the sparsity of the probability distribution of the structure component, a disentanglement loss L u is designed

[0075]

[0076] In the formula, ||·||F represents the F-norm, ||·||1 represents the 1-norm, λ F and λ u are hyperparameters, and I is the unit matrix; based on the disentanglement operation of the tensor product, a disentanglement vector u t is used to recover the dimension of the tensor product represented as S t , and the content vector f t of the predicted vocabulary is extracted t , denoted as f t = S t u t ;

[0077] 5.2) Context enhancement is performed based on the content vector f t of the predicted vocabulary, and the context vector c t of the prediction time t is obtained:

[0078] c i = W t [q t ,f g ]⊙σ(W t [q t ,f i ])

[0079] where σ(·) is a Sigmoid activation function, ⊙ is a dot product operation, W g and W t are fully connected layers for dimension reduction;

[0080] 5.3) The probability distribution p(y t |y 1:t-1 , θ) of the predicted vocabulary at the prediction time t is calculated based on the context vector c t :

[0081] p(y 1:t-1 |y ∑ , θ) = softmax(W t c t )

[0082] where y 1:t-1 represents the predicted vocabulary obtained at the prediction time t, y ∑ represents all predicted vocabularies obtained from 1 to t-1 at the prediction time, θ is a model parameter, and W​∑ is a full connection layer; L is a cross-entropy loss xe The predicted word probability distribution is constrained as follows:

[0083]

[0084] wherein, and is a word in the training set text data, and log(·) is a logarithm function;

[0085] 5.4) The word with the maximum probability at each prediction time is selected to be spliced into a complete description.

[0086] The above embodiments are the preferred embodiments of the present application, but the embodiments of the present application are not limited to the above embodiments, and any changes, modifications, substitutions, combinations, simplifications made without departing from the spirit and principles of the present application should be equivalent replacement methods, and are all included in the protection scope of the present application.

Claims

1. A method for guiding image description generation based on semantic content and unbinding with tensor product, characterized in that, The method comprises the following steps: 1) extracting a global feature set and a local feature set from image data, and extracting a global feature from text data; 2) obtaining semantic content of the image data as semantic content by an embedding space alignment method based on the global features of the image data and the text data; 3) performing feature enhancement on the local feature set of the image data based on the semantic content by a multi-head attention mechanism to obtain an enhanced local feature set, and the specific steps are as follows: First, the semantic content v as image embedding features s is fused with the local feature set V of the image data to obtain a semantic initialized local feature set where the i-th semantic initialized local feature is calculated as follows: In the formula, W fuse is a fully connected layer for realizing dimension reduction, v i represents the i-th local feature of the image data, and [·, ·] is a splicing operation; then, the local feature set V of the image data and the semantic are used to initialize the local feature set V s respectively, to obtain the query set Q, the key set K and the value set The calculation is as follows: In the formula, is a parameter matrix; then the above three sets are embedded into h different subspaces based on the multi-head attention mechanism to calculate the correlation and integrate the structural information. The query set Q i' , the key set K i' , and the value set The calculation process is as follows: In the formula, is the mapping matrix of the i'th subspace; the i'th subspace is the correlation R of the i'th head i' The calculation is Where d is a scale factor to avoid numerical overflow, and then the attention head of the i'th head i' The calculation is represented as The attention calculation results of all heads are spliced to obtain the enhanced local feature set where MultiHead(·) represents a multi-head attention mechanism, head h is the attention for the h-th head, W o is a parameter matrix, enhancing the set of local features is the i-th enhanced local feature in the set 4) performing fusion based on the enhanced local feature set to obtain a tensor product representation of the image data; 5) extracting a content component of a predicted vocabulary from the tensor product representation based on tensor product unbinding, calculating a vocabulary probability distribution, obtaining the predicted vocabulary, and splicing all the predicted vocabularies to obtain a complete description of the image data.

2. The method of claim 1, wherein, In step 1), the ResNet101 convolutional neural network and the Faster R-CNN object detection model are used to extract global and local feature sets of the image data, respectively. The Long Short-Term Memory (LSTM) network is used to extract global features of the text data. Specifically, given an image dataset, each image in the dataset is scaled to a size of 224×224, and then input into the ResNet101 convolutional neural network and the Faster R-CNN object detection model, respectively. The output of the ResNet101 average pooling layer is taken as the global feature v. g Take the set of n region features extracted by Faster R-CNN As a local feature set; each text in the text dataset is taken, and its words are encoded into word embedding vectors through a word embedding matrix, thus encoding each text into a sequence of word embedding vectors. Where x j Let represent the embedding vector of the j-th word, and m represent the number of words in the text. Then, the sequence of word embedding vectors of the text is sequentially input into the LSTM to obtain the updated hidden state vector as the global feature t of the text data. g .

3. The method of claim 2, wherein, In step 2), through the mapping matrix P s global features v of image data g Global features t embedded in text data g The semantic space in which the image data is embedded yields the image data embedding feature v. s =P s v g Based on triplet loss L triplet Zoom in on image data embedding features v s and the global features t of the matching text data g The distance, and the alienation of image data embedding features v s Global features of mismatched text data The distance is calculated as follows: In the formula, max(·) represents a maximum value function, cos(·) represents a cosine similarity function, a is a hyperparameter, and the obtained image embedding feature v s is the semantic content of the image.

4. The method of claim 3, wherein, In step 4), the attention LSTM is denoted as LSTM att At the prediction time t, the word embedding vector x t-1 , the context vector c t-1 and the enhanced local feature mean The word context information at the current prediction time t is obtained The calculation process is: In the formula, and are memory cells of the attention LSTM at prediction time t and t-1, respectively, is the word context information at time t-1; then the semantic content v s and the word context information are fused to obtain the context query vector q t : In the formula, W q To achieve the dimension reduction full connection layer; then based on the multi-head attention mechanism to enhance the local feature set Using the context query vector q t Query to obtain the intermediate representation of the image modality to the text modality at the prediction time t: In the formula, s t Comprising visual content and visual structure information, defined as a tensor product representation of image data.

5. The method of claim 4, wherein, The step 5) comprises the following steps: 5.1) Extracting the content components of predicted words from tensor product representations based on tensor product unbinding: First, extract the content components of predicted words from tensor product representations in vector form s t Dimensionality recovery is achieved by projecting the matrix onto a second-order tensor space, denoted as S. t =diag(s t )P e Where diag(·) is the operation to convert to a diagonal matrix, P e The mapping matrix is ​​used; then a structure matrix U is learned, based on the context query vector q. t and dimensionality recovery of the tensor product representation s t S is represented by the tensor product. t Calculate the probability distribution a of its structural components. t , represented as a t =softmax(P u [q t ,s t ]), where P u The mapping matrix is ​​used to obtain the unbinding vector u. t , represented as u t =a t U; meanwhile, to ensure the orthogonality of the structure matrix and the sparsity of the probability distribution of the structure components, an unbinding loss L is defined. u for: In the formula, ||·|| F Let λ denote the F-norm, ||·||1 denote the 1-norm, and λ u and λ a Here, I is the hyperparameter and the identity matrix; the unbinding operation based on tensor product uses the unbinding vector u. t The tensor product representation S after dimension recovery t Extract the content vector f of the predicted words t , represented as f t =S t u t ; 5.2) Content vector f based on the predicted vocabulary t Contextual enhancement is performed to obtain a context vector c at the prediction time t t : c t = W i [ q t , f t ] σ( W g [ q t , f t ]) In the formula, σ(·) is a Sigmoid activation function, is a dot multiplication operation, W i and W g is a full connection layer for realizing dimension reduction. 5.3) Context-based vector c t Compute the probability distribution p(y t |y 1:t-1 ,θ) over the vocabulary for the prediction time t. p(y t | x, θ) = softmax(W 1:t-1 x + b ∑ c t ) where y t denotes the predicted word at time t, y 1:t-1 denotes all predicted words from 1 to t-1, and 0 is the model parameter, W ∑ is the fully connected layer; the cross-entropy L xe is used to constrain the probability distribution of the predicted words: In the formula, and is the vocabulary of the training set text data, and log(·) is a logarithmic function. 5.4) selecting a vocabulary with the maximum probability at each prediction moment to splice into a complete description.

Images