An entity-guided image text description generation method

Abstract

The application discloses an entity-guided image text description generation method, which comprises the following steps: using a sequence generation paradigm to realize the generation of an ordered sequence containing only entity words, so as to identify key entities in an image; assigning corresponding weights to entities according to the cosine similarity of visual features and entity features, realizing the adaptive fusion of entities and visual features, and obtaining multi-modal representations based on context-dependent relationship fusion; and using an encoder-decoder to extract multi-scale features from the fused multi-modal representations and perform adaptive aggregation, which is used for generating accurate description sentences. The application aims to guide the model to focus on significant visual areas by using explicit entity information, realize efficient fusion of entity information and visual features, enhance the understanding ability of the model to image semantics by using multi-scale feature aggregation technology, and thus generate more targeted and expressive image descriptions.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image description, and in particular to an entity-guided image text description generation method. BACKGROUND

[0002] Image description generation technology has been gaining increasing research attention in recent years by translating the semantics and spatial content of images into intuitive textual descriptions. This technology can fully leverage the information advantage of images and has important application value in disaster monitoring, urban planning, and military decision-making.

[0003] In recent years, the rapid development of deep learning technology has significantly promoted the research progress of image interpretation tasks. Since the salient entities in images often carry core semantic information, more and more research works have introduced entity retrieval methods based on visual language models (Contrastive Language-Image PreTraining, CLIP), combined with entity hierarchical structure information, to assist in generating more accurate and semantically rich image descriptions. However, when facing complex structural or semantically dense image scenes, existing methods still face many challenges: on the one hand, the pre-training data of the CLIP model mainly comes from general open fields, and the fine-grained entity recognition ability in specific scenes is limited, such as small targets, special object shapes, or semantically ambiguous regions; on the other hand, the fusion method of entity information and image global features is relatively simple, which can easily lead to unclear entity recognition, missing details, or weak semantic connection, thereby affecting the description quality and consistency. SUMMARY

[0004] The present application proposes an entity-guided image text description generation method, aiming to solve the technical problem that existing CLIP-based entity retrieval methods have insufficient fine-grained entity feature capturing ability in complex image scenes, and the fusion method is single, which further affects the image description generation effect. The method includes:

[0005] Obtaining an input image, extracting the global visual representation of the input image using a visual language model .

[0006] Inputting the global visual representation into a serialized multi-entity generator module to generate an ordered entity sequence containing only entity words, realizing key entity recognition in the input image.

[0007] Inputting the ordered entity sequence into a word embedding encoder to obtain an entity feature vector E , calculating the global visual representation and the entity feature vector Ecosine similarity between the two vectors, the similarity attention gate mechanism adaptively adjusts the entity feature vector E the contribution weight of each entity in the similarity weight calculation, and the entity feature vector after similarity weight calculation .

[0008] the global visual representation is subjected to prefix expansion processing to obtain a visual condition prefix , the entity feature vector after similarity weight calculation is subjected to average pooling operation and the visual condition prefix is subjected to element-level addition to obtain a multi-modal fusion feature .

[0009] The words in the predicted description sentence are subjected to word vector encoding, and the obtained predicted word embedding encoding vector is injected into the mask self-attention module to obtain a context-attended text representation , the multi-modal fusion feature and the context-attended text representation are jointly input into a decoder based on a hierarchical Transformer architecture, adaptively aggregated, processed by a feedforward neural network and an activation function module, and gradually obtained The text description of the input image.

[0010] Further, the serialized multi-entity generator module comprises:

[0011] the global visual representation is mapped to a location-specific visual representation through linear transformation ; the embedding vector of the predicted entity is obtained by performing word embedding encoding on the predicted entity word ; the embedding vector of the predicted entity is transmitted into the multi-head mask self-attention module to obtain a context-attended entity representation ; the context-attended entity representation and the location-specific visual representation are input into the cross-attention module, and the output of the cross-attention module is processed by the feedforward neural network and the activation function module to predict the probability distribution of the next entity token, and gradually generate an ordered entity sequence containing only entity words.

[0012] Further, the similarity attention gate mechanism function expression is:

[0013] ,

[0014] wherein, denotes the length of the entity sequence, and Sim denotes the cosine similarity, Represents an entity, Represents a learnable linear mapping matrix. It is a learnable threshold initialized to 0.1, used to adaptively adjust the contribution weight of each entity. This represents the sigmoid gate function. This is a temperature parameter that adjusts the gating sharpness; the lower the value, the steeper the response curve. This represents the similarity weight value assigned to each entity in the entity sequence. It is the entity feature calculated based on similarity weight.

[0015] Furthermore, regarding the global visual representation... Prefix expansion is performed to obtain the visual conditional prefix. The entity feature vector after calculating the similarity weights After performing average pooling operation and then combining with the visual conditional prefix Element-wise addition is performed as follows:

[0016] Global visual representation The visual symbols are expanded into visual tags by a linear layer and concatenated with learnable prefix codes. They are then processed by a Transformer, with the output at the corresponding prefix position serving as the visual conditional prefix. ;

[0017] The entity feature vector after calculating the similarity weight After average pooling, a global entity representation is obtained. This global entity representation is then processed... Secondary broadcast, and with visual condition prefix Element-level addition is performed to obtain multimodal fusion features. The function expression is:

[0018] ,

[0019] in, express The next broadcast, Indicates the length of the entity sequence. This represents the entity feature vector calculated based on similarity weights. Individuals.

[0020] Furthermore, each layer of the decoder based on the hierarchical Transformer architecture refines visual features using a self-attention mechanism and combines them with the context-sensitive text representation using cross-attention. The function expression for fusion is:

[0021] ,

[0022] in, From the first A refined visual feature of the self-attention layer, This represents the multi-scale features extracted through the corresponding cross-attention layers; MHA stands for multi-head attention mechanism. This indicates the self-attention mechanism. This represents the cross-attention mechanism.

[0023] Text representation based on the aforementioned contextual attention Guided, using adaptive integration modules to fuse multi-scale features , By utilizing a learnable weighting mechanism, the relative importance of each scale feature is dynamically estimated.

[0024] Preferably, the text representation based on the contextual attention Guided, using adaptive integration modules to fuse multi-scale features , The learningable weighting mechanism is utilized, specifically by introducing two multilayer perceptrons. First, the connectivity features are interactively encoded into intermediate representations, and then normalized fusion weights are generated. The function expression is as follows:

[0025] ,

[0026] In this model, MLP1 is a multilayer perceptron 1, and MLP2 is a multilayer perceptron 2. Both MLP1 and MLP2 contain two linear layers connected by a Gaussian error linear unit activation function. This indicates a feature concatenation operation. This represents the weighted fusion feature representation.

[0027] Then, a gating network is used for further balancing. Modules and The contribution of [the entity / entity] is expressed as a function:

[0028] ,

[0029] in, It consists of a linear layer, a normalized layer, a GELU activation function, and another linear layer in that order. Presentation layer normalization operation, It is the sigmoid activation function. This is the gated vector.

[0030] Furthermore, during the training phase, the loss function expression used is:

[0031] ,

[0032] in, Cosine annealing is used for scheduling. and These represent the lengths of the entity sequence and the descriptive text sequence, respectively. and This indicates the size of the vocabulary corresponding to the entity and its description text. and These represent the time steps. At that time, select the first for the entity sequence and the description text sequence. k The predicted probability of the nth word, based on the actual annotation, when the nth word... The word is in position If the location is correctly marked, then or Otherwise, set it to 0.

[0033] Furthermore, after completing the training process using the loss function, the model is fine-tuned using a self-critical sequence training method. The loss formula for this fine-tuning is as follows:

[0034] ,

[0035] in, N Indicates batch size, Indicates the first The reward value generated by sampling corresponding to each image is described. Represents the baseline reward, which is calculated as the reward for the first... A beam search is performed on the images, and the average reward of the results is... This represents the logarithmic probability described by the sample.

[0036] The proposed entity-guided image description generation method constructs a two-branch sequence prediction framework and designs an entity weight allocation mechanism based on semantic similarity to extract structurally clear and semantically reliable entity sequences from images. Simultaneously, it utilizes the CLIP model to extract image visual features and adaptively fuses them with entity information to enhance semantic expressiveness. Furthermore, it introduces a Transformer architecture combined with a multilayer perceptron to achieve cascaded aggregation of multi-scale features and semantic extraction, thereby significantly improving the accuracy, detail completeness, and semantic relevance of the generated descriptions. This method is suitable for automatic content understanding and language generation tasks for various image types.

[0037] Compared with the prior art, the present application mainly has the following advantages: first, by converting entity recognition into a sequence generation task, fine-grained entities and co-occurring entities in complex scenes can be accurately modeled, and the limitations of existing CLIP-based entity retrieval methods in capturing fine-grained entity features can be effectively compensated; second, with the help of entity perception gating mechanism, entity-vision adaptive fusion can be realized, which can greatly improve the feature fusion effect of entities and visual representations and enhance the relevance of entity information and image features; third, a hierarchical Transformer decoder is used to capture multi-scale context dependencies and adaptively aggregate high-level semantic features, which can realize multi-scale cascading aggregation and further improve the semantic richness and accuracy of the description. BRIEF DESCRIPTION OF DRAWINGS

[0038] Figure 1 The basic flowchart of the embodiment of the application method is shown.

[0039] Figure 2 The overall framework structure diagram of the embodiment of the application method is shown.

[0040] Figure 3 The principle diagram of the adaptive aggregation mechanism in the embodiment of the application is shown. DETAILED DESCRIPTION

[0041] In order to better understand the above technical solutions, the exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0042] As an implementation scheme, Figure 1 The flowchart of the entity-guided image text description generation method involved in the embodiment of the application is shown. Figure 2 The overall framework structure diagram of the entity-guided image text description generation method involved in the embodiment of the application is shown. As Figures 1-2 shown, the method comprises:

[0043] S101: Obtain an input image, and extract a global visual representation of the input image using a visual language model .

[0044] Using a natural language processing tool, extract nouns from sentences in a training data set as candidate entities, and then use a morphological restoration technique to unify the entity form for constructing supervised signals for ordered entity sequence generation.

[0045] S102: Convert the global visual representation An input serialized multi-entity generator module generates an ordered entity sequence containing only entity words, achieving key entity recognition in the input image.

[0046] The serialized multi-entity generator module includes:

[0047] S1021: The global visual representation is mapped to a location-specific visual representation .

[0048] S1022: The predicted entity word is encoded by performing word embedding, obtaining an embedding vector of the predicted entity , , m represents the number of predicted entities for each image sample.

[0049] S1023: The embedding vector of the predicted entity is input into a multi-head mask self-attention module to obtain a context-focused entity representation , the function expression is:

[0050] ,

[0051] In the above formula, represents the context encoding of the predicted entity token, represents a causal mask when modeling the entity sequence, which is used to avoid leaking future predicted entity information, represents a multi-head self-attention mechanism, represents a multi-head attention mechanism.

[0052] The input features are divided into multiple heads, and the function expression is:

[0053] ,

[0054] In the above formula, is a learnable mapping matrix, is an optional additional term for adjusting attention visibility, represents the number of attention heads. The calculation expression of each attention head is:

[0055] ,

[0056] In the above formula, Q, K, and V represent query, key, and value vectors, respectively, is the mapping matrix of the th attention head, represents the dimension of each attention head.

[0057] S1024: The context-focused entity representation and the position-specific visual representation The input cross-attention module, the output of the cross-attention module is processed through the feedforward neural network and the activation function module, to predict the probability distribution of the next entity label, and to generate an ordered entity sequence containing only entity words step by step. The function expression is:

[0058] ,

[0059] In the above formula, CA represents cross-attention calculation, entity representation , position-specific visual representation as the key vector and the value vector.

[0060] S103: input the ordered entity sequence into the word embedding encoder to obtain the entity feature vector , , calculate the cosine similarity between the global visual representation and the entity feature vector E , and adaptively adjust the contribution weight of each entity in the entity feature vector E through the gating mechanism of similarity attention to obtain the entity feature vector after similarity weight calculation . The function expression is:

[0061] ,

[0062] In the above formula, denotes the length of the entity sequence, and Sim denotes the cosine similarity between each entity and the corresponding image, is a learnable threshold initialized to 0.1, used to adaptively adjust the contribution weight of each entity, denotes the sigmoid gating function, is a temperature parameter that adjusts the sharpness of the gating, and the lower the value, the steeper the response curve produced, denotes the similarity weight value assigned to each entity in the entity sequence, is the entity feature calculated according to the similarity weight.

[0063] S104: perform prefix expansion processing on the global visual representation to obtain the visual condition prefix , perform average pooling operation on the entity feature vector after similarity weight calculation , and perform element-level addition with the visual condition prefix to obtain the multi-modal fusion feature .

[0064] This step is implemented as follows:

[0065] S1041: The global visual representation is obtained by linear layer expansion of the visual mark, and is connected with the learnable prefix code, and then is processed by the Transformer, and the output part of the prefix position is used as the visual condition prefix . .

[0066] S1042: The entity feature vector after similarity weight calculation is obtained by linear layer expansion of the visual mark, and is connected with the learnable prefix code, and then is processed by the Transformer, and the output part of the prefix position is used as the visual condition prefix . . . , the function expression is:

[0067] ,

[0068] wherein, represents the broadcasting, represents the length of the entity sequence, represents the individual of the entity feature after similarity weight calculation.

[0069] S105: The words in the predicted description sentence are word vector encoded, and the obtained predicted word embedding code vector is injected into the mask self-attention module to obtain the context attention text representation . , the multi-modal fusion feature and the context attention text representation are jointly input into the decoder based on the hierarchical Transformer architecture, and are adaptively aggregated, and are processed by the feedforward neural network and the activation function module, and are gradually obtained. The text description of the input image.

[0070] The function expression of the context attention text representation is:

[0071] ,

[0072] In the above formula, MSA represents multi-head self-attention, is a causal mask, which is used to avoid leaking the information of the predicted description word in the future in the decoding process.

[0073] Each layer of the decoder based on the hierarchical Transformer architecture refines the visual feature by using the self-attention mechanism, and fuses it with the text representation by using the cross-attention. The function expression is:

[0074]

[0075] In the above formula, is the visual feature refined from the th self-attention layer, represents the multi-scale feature extracted through the corresponding cross-attention layer.

[0076] Text representation based on context attention Guiding, fusing multi-scale features with adaptive integration module , dynamically estimating the relative importance of each scale feature using a learnable weight mechanism. The principle diagram of the adaptive aggregation mechanism is shown in Figure 3 , specifically, by introducing two multi-layer perceptrons, first interactively encode the connection features into intermediate representations, and then generate normalized fusion weights, the function expression is:

[0077] ,

[0078] In the above formula, multi-layer perceptron 1 (MLP1) and multi-layer perceptron 2 (MLP2) adopt similar architecture, both containing two linear layers, connected by Gaussian error linear unit (GELU) activation function, .

[0079] Thereafter, the gating network is used to further balance the contribution degree of module and . The function expression is:

[0080] ,

[0081] In the above formula, consists of a linear layer, a normalization layer, a GELU activation function, and another linear layer, which will be applied in turn in sequence. The sigmoid activation function generates a gating vector , which operates at the word level, dynamically balancing the fusion features and context features.

[0082] The loss function used in model training is cross-entropy loss, which is mathematically expressed as follows:

[0083] ,

[0084] In the above formula, adopting cosine annealing strategy for scheduling, and represent the lengths of the entity sequence and the description text sequence respectively, and represent the vocabulary size corresponding to the entity and the description text, and These represent the time steps. At that time, select the first for the entity sequence and the description text sequence. The predicted probability of the i-th word, based on the actual annotation, if the i-th word... The word is in position If the location is correctly marked, then (or Otherwise, set it to 0.

[0085] After completing the training process using cross-entropy loss, the model is fine-tuned using a self-critical sequence training method. This method directly uses policy gradients to optimize non-differentiable metrics. Specifically, the loss formula for this fine-tuning is:

[0086]

[0087] In the above formula, N represents the batch size. Indicates the first The reward value generated by sampling corresponding to each image is described. Represents the baseline reward, which is calculated as the reward for the first... A beam search is performed on the images, and the average reward of the results is... Then it represents the log probability described by the sample.

[0088] This embodiment presents an entity-guided image description generation method. On one hand, it utilizes a fine-tuned CLIP model to extract global visual features of the image. On the other hand, it introduces an entity weight allocation mechanism based on semantic similarity to extract semantically reliable entity sequences from the image and adaptively fuse them with the image's visual features. Furthermore, it employs a multi-layer Transformer structure integrating a multi-layer perceptron to achieve cascaded aggregation and semantic enhancement of multi-scale features, thereby fully mining explicit entity information and improving the accuracy and expressive richness of the generated descriptions. This method is suitable for automatic content understanding and language generation tasks for various image types.

Claims

1. An entity-guided image text description generation method, characterized in that, The method comprises: obtaining an input image, extracting a global visual representation of the input image using a visual language model ; the global visual representation an input serialized multi-entity generator module that generates an ordered entity sequence containing only entity words, enabling key entity recognition in the input image; inputting the ordered entity sequence into a word embedding encoder to obtain an entity feature vector E , calculating a cosine similarity between the global visual representation and the entity feature vector E , adaptively adjusting a contribution weight of each entity in the entity feature vector E by a gating mechanism of similarity attention to obtain an entity feature vector after similarity weight calculation ; The global visual representation is obtained by performing prefix expansion processing on the global visual representation The visual condition prefix is obtained by performing prefix expansion processing on the global visual representation The similarity weight is calculated according to the similarity between the entity feature vector and the global visual representation The similarity weight is calculated according to the similarity between the entity feature vector and the global visual representation The similarity weight is calculated according to the similarity between the entity feature vector and the global visual representation The similarity weight is calculated according to the similarity between the entity feature vector and the global visual representation word vector encoding is performed on the predicted words in the description sentence, and the obtained predicted word embedding coding vector is obtained An injection mask self-attention module is used to obtain a context-focused text representation The multi-modal fusion feature And the context-focused text representation The joint input is input into a decoder based on a hierarchical Transformer architecture, adaptive aggregation is performed, a feedforward neural network and an activation function module are used for processing, and the text description of the input image is gradually obtained. The serialized multi-entity generator module comprises: mapping the global visual representation to a location-specific visual representation ; by performing word embedding encoding on the predicted entity word, obtaining an embedding vector of the predicted entity ; embedding vectors of the predicted entities incoming multi-head mask self-attention module, obtaining entity representations with contextual attention ; characterizing the entity of interest of the context and the location-specific visual representation an input cross-attention module, the output of the cross-attention module being processed through a feed-forward neural network and an activation function module, predicting a probability distribution of the next entity token, progressively generating an ordered entity sequence containing only entity words.

2. The method of claim 1, wherein, The similarity attention gate mechanism function expression is: ， wherein, denotes the length of the entity sequence, Sim denotes the cosine similarity, denotes the entity, denotes the learnable linear mapping matrix, is a learnable threshold initialized to 0.1, used to adaptively adjust the contribution weight of each entity, denotes the sigmoid gating function, is a temperature parameter that regulates the sharpness of the gating, the lower the value, the steeper the response curve produced, denotes the similarity weight value assigned to each entity in the entity sequence, is the entity feature vector after similarity weight calculation.

3. The method of claim 1, wherein, The global visual representation is obtained by performing prefix expansion processing on the global visual feature vector to obtain a visual condition prefix The global visual representation is obtained by performing prefix expansion processing on the global visual feature vector to obtain a visual condition prefix The global visual representation is obtained by performing prefix expansion processing on the global visual feature vector to obtain a visual condition prefix The global visual representation is obtained by performing prefix expansion processing on the global visual feature vector to obtain a visual condition prefix The global visual representation is obtained by performing prefix expansion processing on the global visual feature vector to obtain a visual condition prefix global visual representation extended by a linear layer, concatenated with a learnable prefix encoding, followed by a Transformer processing, with the output part corresponding to the prefix position as the visual conditioning prefix ; The similarity weight calculated entity feature vector After the average pooling operation, the global entity representation is obtained, and then the global entity representation is The secondary broadcast is added element by element with the visual condition prefix to obtain a multi-modal fusion feature The function expression is: ， wherein, denotes sub-broadcast, denotes the length of the entity sequence, denotes the individual of the entity feature vector after similarity weight calculation.

4. The method of claim 1, wherein, Each layer of the decoder based on the hierarchical Transformer architecture refines visual features with self-attention mechanisms and contextualizes them with cross-attention to text representations fusion, the functional expression of which is: ， wherein, is a visual feature refined from the first self-attention layer, denotes a multi-scale feature extracted through a corresponding cross-attention layer, and MHA denotes a multi-head attention mechanism, denotes a self-attention mechanism, denotes a cross-attention mechanism. Text representation based on the context attention Guiding, fusing multi-scale features with adaptive integration module Dynamically estimating the relative importance of each scale feature with a learnable weight mechanism.

5. The method of claim 4, wherein, Text representation based on the context attention Guiding, fusing multi-scale features with adaptive integration module With a learnable weight mechanism, specifically by introducing two multi-layer perceptrons, first interactively encode the connection features into an intermediate representation, and then generate normalized fusion weights, the function expression is: ， wherein MLP1 is a multi-layer perceptron 1, MLP2 is a multi-layer perceptron 2, both MLP1 and MLP2 comprise two linear layers connected by a Gaussian error linear unit activation function, denotes a feature concatenation operation, denotes a weighted fused feature representation; Then the gating network is used to further balance the contribution of the modules and their function expressions are: ， wherein, is sequentially composed of a linear layer, a normalization layer, a GELU activation function, and a linear layer, is a layer normalization operation, is a sigmoid activation function, is a gating vector.

6. The method of claim 1, wherein, In the training phase, the loss function expression used is: ， in, Cosine annealing is used for scheduling. and These represent the lengths of the entity sequence and the descriptive text sequence, respectively. and This indicates the size of the vocabulary corresponding to the entity and its description text. and These represent the time steps. At that time, select the first for the entity sequence and the description text sequence. k The predicted probability of the nth word, based on the actual annotation, when the nth word... The word is in position If the location is correctly marked, then or Otherwise, set it to 0.

7. The method of claim 6, wherein, After the training process is completed using the loss function, the model is fine-tuned using a self-critical sequence training method, and the loss formula of the fine-tuning is: ， where, N denotes the batch size, denotes the i denotes the reward value of the sampled description corresponding to the image, denotes the baseline reward, which is computed as the average reward of the results of beam search on the denotes the reward of the sampled description. denotes the log probability of the sampled description.

Images