Picture description generation method based on multi-modal entity alignment

Abstract

The invention relates to a picture description generation method based on multi-modal entity alignment, belongs to the technical field of image processing, and solves the problem of low accuracy of image entity description in the prior art. The method comprises the following specific steps: training to obtain a multi-modal entity alignment model based on an image containing marked entities and a first sample set of related articles; on the basis of a second sample set containing the images subjected to annotation description and related articles, candidate entity words subjected to entity alignment are obtained through a multi-modal entity alignment model; based on the second sample set and the candidate entity words, training to obtain an image description generation model based on multi-modal entity alignment; according to the method, the corresponding candidate entity words are obtained by utilizing the multi-modal entity alignment model based on the image and the article which are not labeled and described, and the image description result is obtained by combining the image and the article which are not labeled and described and utilizing the image description generation model, so that the recall rate and the accuracy rate of image entity selection are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a method for generating picture descriptions based on multimodal entity alignment. Background Art

[0002] Image description generation is the process of automatically generating natural language descriptions for images using computer vision and natural language processing techniques. This task aims to generate accurate and coherent sentences based on the image content to describe the main elements, scenes and their relationships in the image by comprehensively analyzing the multimodal information of images and texts. It is widely used in intelligent assistance in medical and educational fields, automated news reporting, human-computer interaction and other fields. Current research methods mainly focus on image perception and understanding and cross-modal sequence generation, and are committed to improving the effect of description generation by deepening the expression of image features and optimizing the encoder-decoder attention mechanism.

[0003] Existing image description generation methods mainly focus on extracting common conceptual elements from images to generate descriptions. However, this method ignores the alignment and association of multimodal image and text entity information (such as names of people and places) in the fusion and transformation stages, which limits the accuracy of entity expression in descriptive sentences. The description sentence generation of news image description generation methods can generate description sentences containing entity information with the help of image background information articles. However, since the input article is long and contains entity information of multiple images, there is a large amount of noise data, which seriously affects the alignment of image and text features and the capture of feature associations between images and entity content. Based on image and text fusion coding, researchers further explored how to improve the correlation between image and text entities. For example, the Bi-LSTM network is used to train the mutual connection between image features and entity words, and entity features related to the image are selected as prompts for description generation; or through the face named entity module, the self-attention network is used to calculate the attention weights between the facial features in the image and the name entity sequence to enhance the association between the image content and the name entity words; or the image entity selection task and the entity-related sentence selection task are introduced to improve the model's understanding of the entity content in the article by training the association between image information and related entities.

[0004] Although these methods have improved the entity generation performance of the model to a certain extent, there are still some problems in the entity alignment process, such as (1) the non-visual entity association problem. Among all the entities associated with the image, there are both visual entities such as "names" and "place names" and non-visual words such as "time" and "commodity". Existing methods find it difficult to learn the alignment relationship between images and non-visual entity words, which limits the performance of entity alignment. (2) The problem of too many negative samples. This means that there is a lot of noise in the image entity description generation task when training entity associations, which seriously interferes with its association learning process and leads to low matching efficiency. Summary of the invention

[0005] In view of the above analysis, an embodiment of the present invention aims to provide a method for generating image description based on multimodal entity alignment, so as to solve the problem of low accuracy of image entity description in the prior art.

[0006] The purpose of the present invention is mainly achieved through the following technical solutions: The embodiment of the present invention provides a method for generating image description based on multimodal entity alignment, comprising the following steps: Based on the first sample set containing the images with annotated entities and the articles related thereto, a multimodal entity alignment model is trained; Based on a second sample set including the annotated images and articles related thereto, using the multimodal entity alignment model, obtaining entity aligned candidate words; Based on the second sample set and the candidate entity words, training an image description generation model based on multimodal entity alignment; Based on the images and articles without labeled descriptions, the multimodal entity alignment model is used to obtain corresponding candidate entity words, and combined with the images and articles without labeled descriptions, the image description generation model is used to obtain image description results.

[0007] Furthermore, the multimodal entity alignment model includes a feature extraction module, a multiple decision feature fusion module and an entity output network, wherein: A feature extraction module is used to extract features from the sample set of images containing annotated entities and articles related thereto, respectively, to obtain image features and article context features, wherein the article context features include multiple article subword features; A multiple decision feature fusion module is used to perform cross-modal feature fusion on the image features and the article context features to obtain multiple fused sub-word features and corresponding scores; The entity output network is used to obtain multiple entity words related to the image based on the fused sub-word features and corresponding scores.

[0008] Furthermore, obtaining the fused subword features and corresponding scores includes: Using the spacy tool to respectively annotate entity word features and a set of non-entity word features in the context features of the article; Based on the image features, the query layer mapping of the self-attention mechanism is used to obtain the image hidden features Based on the context features of the article, the three mapping layers of query, key and value of the self-attention mechanism are used to obtain the hidden features of the article, which can be expressed as ; and obtaining entity context features based on the set of entity word features; Based on the image hidden features And the hidden features of the article , and mask the non-entity word features in the article context features, calculate the similarity between the image features and the entity word features, and obtain the image attention weight ; Based on the hidden features of the article , calculate the similarity between the context features of the article and get the article attention weight ; Mask the non-entity word features in the article context features, calculate the similarity between entity context features, and obtain the entity attention weight ; Based on three attention weights, the latent features of the article are Perform feature fusion to obtain multiple fused sub-word features; The fused sub-word features are passed through two fully connected networks to obtain the corresponding scores of the fused sub-words.

[0009] Furthermore, the plurality of entity words related to the image include: Based on the average of the corresponding scores of each fused subword, the corresponding entity word score is obtained; Based on the entity word score, using The function masks non-entity words in the article and performs entity selection on the words in the article; Based on the scores of multiple entity words after entity selection, multiple entity words related to the image are obtained by using TopK sorting and threshold filtering.

[0010] Furthermore, the feature extraction module uses the CLIP model to extract features from the input image samples to obtain image features; and uses the T5 model to extract text features from the input article samples to obtain article context features.

[0011] Furthermore, the multi-task constrained training strategy is used to train the multimodal entity alignment model, including: Re-label the fused sub-word features, including labeling entity word features related to the image content Labeled as positive samples, entity word features that are irrelevant to the image content Labeled as negative samples; Based on the entity word features and , calculate the cross entropy loss function; Construct constraints to reduce the distance between features of similar samples, expand the distance between features of heterogeneous samples, and expand the distance between the probability of score outputs of positive and negative samples; An objective function is established based on the cross entropy loss function and the constraint conditions, and the multimodal entity alignment model is obtained by multi-task optimization training.

[0012] Furthermore, the expressions of the constraint conditions are: , , , Among them, L1 represents the constraint of reducing the distance between features of similar samples; , Represent the entity word features of positive and negative samples respectively; They represent the positive and negative sample entity word features in a shuffled order; n represents the feature dimension; L represents the number of positive samples; L2 represents the constraint of expanding the distance between the features of heterogeneous samples; is a hyperparameter, which indicates the optimized distance between the features of positive and negative samples; L3 indicates the constraint of expanding the probability distance of the output scores of positive and negative samples; Represents the parameters of the multimodal entity alignment model; ne represents the number of negative samples; po represents the number of positive samples.

[0013] Furthermore, the objective function is expressed as: , , Where y represents the entity word output by the multimodal entity alignment model; Lc is the cross entropy loss function; L is the objective function; are hyper parameters respectively.

[0014] Furthermore, the image description generation model is trained, including: Use the visual encoder and Q-former of the BLIP-2 model to encode the input image samples and obtain image features ; The candidate entity words and the article words obtained by the word segmenter of the BLIP-2 model are encoded using the word embedding layer to obtain the article features ; The image features and the article features The concatenated input is then fed into the BLIP-2 language model for description prediction; The image description generation model is trained based on maximum likelihood estimation loss and an efficient fine-tuning method.

[0015] Furthermore, the efficient fine-tuning method utilizes low-rank re-parameterization fine-tuning technology to fine-tune specific layer parameters of the image description generation model, and the specific layer parameters are feature parameters in the self-attention mechanism of the image description generation model.

[0016] Compared with the prior art, the present invention can achieve at least one of the following beneficial effects: 1. The present invention proposes a news image description generation model based on multimodal entity alignment. The associated entities selected by the multimodal entity alignment model are used as prompt words and input into the image description generation model, thereby improving the association between the generated sentences and non-visual entities; by integrating and understanding the information of the article context, image and entity content, the recall rate and precision of image entity selection are improved.

[0017] 2. In terms of entity prompts, entity words are directly used to guide the large model for sequence generation, and the language encoding of the large model is used to fully understand the input entity prompts, thereby reducing the complexity of the task. At the same time, a multimodal entity alignment model is used for entity screening, which reduces some fitting calculation time during description generation training and improves training efficiency.

[0018] 3. By utilizing the image description generation model based on multimodal entity alignment and combining it with an efficient fine-tuning method, the image-text entity information interaction process in the large visual language model can be efficiently fine-tuned. With a small amount of data, it can achieve performance comparable to the existing highest method, improve matching efficiency, and obtain the best performance in terms of entity recall and accuracy. It can be widely used in intelligent assistance in medical and educational fields, automated news reporting, human-computer interaction, and other fields.

[0019] In the present invention, the above-mentioned technical solutions can also be combined with each other to achieve more preferred combination solutions. Other features and advantages of the present invention will be described in the subsequent description, and some advantages can become obvious from the description, or can be understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained through the contents particularly pointed out in the description and the drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The accompanying drawings are only used for the purpose of illustrating specific embodiments and are not to be considered as limiting the present invention. In the entire drawings, the same reference symbols represent the same components; Figure 1 A schematic diagram of a flow chart of a method for generating a multimodal entity aligned picture description according to an embodiment of the present invention; Figure 2 A framework diagram of an image entity description generation model based on entity alignment learning in an embodiment of the present invention; Figure 3 A network architecture diagram of feature fusion of multiple decisions according to an embodiment of the present invention; Figure 4 This is a network architecture diagram for efficient fine-tuning of BLIP-2 image entity description based on an embodiment of the present invention; Figure 5 A comparison chart of recall rates of different entity types in an embodiment of the present invention; Figure 6 This is a comparison chart of the accuracy of different entity types in the embodiment of the present invention; Figure 7 This is a comparison chart of F1 scores of different entity types in an embodiment of the present invention; Figure 8 This is a comparison chart of visual entity recall, precision and F1 score of an embodiment of the present invention; Fig. 9 This is a comparison chart of the recall rate, precision rate and F1 score of non-visual entities in an embodiment of the present invention. DETAILED DESCRIPTION

[0021] The preferred embodiments of the present invention are described in detail below in conjunction with the accompanying drawings, wherein the accompanying drawings constitute a part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not used to limit the scope of the present invention.

[0022] A specific embodiment of the present invention discloses a method for generating image description based on multimodal entity alignment, such as Figure 1 As shown, the following steps are included: Step S1: training a multimodal entity alignment model based on a first sample set of images containing annotated entities and articles related thereto; Step S2: based on the second sample set including the annotated images and the articles related thereto, using the multimodal entity alignment model, obtaining the candidate entity words after entity alignment; Step S3: Based on the second sample set and the candidate entity words, training an image description generation model based on multimodal entity alignment; Step S4: Based on the unlabeled images and articles, the multimodal entity alignment model is used to obtain corresponding candidate entity words, and the image description generation model is used in combination with the unlabeled images and articles to obtain image description results.

[0023] Through the above method, the multimodal entity alignment model is used to pre-screen candidate entity words associated with the image. Based on the candidate entity words, the input image and the input article, the image description generation model based on multimodal entity alignment is used to describe the picture and obtain the description result. By integrating and understanding the information of the article context, image and entity content, the accuracy of the image entity description is improved.

[0024] In the existing methods that mainly rely on image-text features for description generation, due to the problem that article information and image content do not fully match, the article contains entity information of multiple images, which makes it difficult to associate the image information with the input article information, which in turn leads to the problem that the generated sentence contains incorrect entity description content. The embodiment of the present invention proposes an image entity description generation model based on entity alignment learning, such as Figure 2As shown in the figure, it mainly includes two parts: entity selection and description generation. In the entity selection part, a multimodal large model is used to extract features from images and articles. By integrating image content and long text information into a unified expression space, the expression ability of multimodal entity information is effectively improved. In the encoding stage, in order to further study the interactive relationship between image and text entities, an entity alignment model based on multiple decision fusion (EAM-MDF, referred to as multimodal entity alignment model) is proposed to screen related entities, select entities related to image content, and use them as prompt inputs for the encoder of the multimodal large model. Combined with the description generation part, the decoder is used to strengthen the entity content association between images and articles during modal conversion, thereby achieving more targeted entity description generation. At the same time, a training strategy based on multi-task constraints is introduced to alleviate the impact of imbalanced entity samples.

[0025] Specifically, in step S1, the multimodal entity alignment model includes a feature extraction module, a multiple decision feature fusion module and an entity output network. Among them, the feature extraction module is used to extract features from images and articles, and the expression ability of multimodal entity information is effectively improved by fusing image content and long text information into a unified expression space; a cross-modal entity alignment task is constructed through the multiple decision feature fusion module, while considering the output article context, the relationship between articles and images, and entity words, deepening the intrinsic connection between images and articles, and enhancing the connection between images and non-visual entities; the entity output network module is used to screen and obtain entities related to image content. The steps of obtaining entities related to image content using the multimodal entity alignment model include: S11, based on the annotated image and the data sample set of related articles, using a feature extraction module to extract corresponding features to obtain image features and article context features; wherein the annotated content is entity information; For example, given a data sample ,in, and Represent the i-th input article and related image respectively. The multimodal pre-trained neural network model CLIP (Contrastive Language-Image Pre-training) is used to train the input image Perform feature extraction, expressed as: , (1) in, Based on the input image The extracted features, and , is the dimension of image features.

[0026] The T5 (Text-to-Text Transfer Transformer) model is used to extract text features from the input article A, and the article context features containing multiple article subword features are obtained, which are expressed as: , (2) in, is the context feature extracted based on the input article A, and , is the length of the subwords after the article is segmented. Indicates the feature dimension of each subword. When segmenting words, identifiers are used to encode subwords from the same word.

[0027] S12, inputting the extracted image and text features into a multiple decision feature fusion module to perform cross-modal feature fusion to obtain multiple fused sub-word features and corresponding scores; Specifically, unlike existing methods that only rely on image information and entity association learning, the multi-decision feature fusion module is used to associate images with entities, article contexts, and entity contexts. By mining the entity connections in the article content, the alignment relationship between images and non-visual entities is achieved. The network architecture of the multi-decision feature fusion module is shown in the figure. Figure 3 As shown, obtaining the fused subword features and corresponding scores specifically includes: S121, using the spacy tool to respectively annotate entity word features and a set of non-entity word features in the article context features; S122. Based on article context features and image features , using the self-attention mechanism to obtain the latent features of the article and the image respectively; Specifically, image features After the query layer and mapping layer, the image hidden features are obtained ; Given the input article context features , which is transformed into three kinds of article hidden features through the three linear mapping layers of query, key and value of the self-attention mechanism, namely the latent space state ; and obtaining entity context features based on the set of entity word features.

[0028] S123, based on the image hidden features and the hidden features of the article , and mask the non-entity word features in the article context features, calculate the similarity between the image features and the entity features, and obtain the image attention weight ; S124, based on the hidden features of the article , calculate the similarity between the context features of the article and get the article attention weight ; S125. Mask the non-entity word features in the article context features, calculate the similarity between entity context features, and obtain the entity attention weight ; For example, the association between article contexts is obtained by calculating the similarity between article context features, which is expressed as the article attention weight ; The association between the image and the entity is obtained by calculating the similarity between the image features and the entity features, which is expressed as the image attention weight ; The association between entity contexts is obtained by calculating the similarity between entity context features, which is expressed as entity attention weight ; The calculation formulas are shown in (3) to (5): , (3) , (4) , (5) Where d represents the dimension of the feature; The function is used to mask non-entity words in the article to obtain the characteristics of entity words.

[0029] S125. Based on three weight decisions, the hidden features of the article Perform feature fusion to obtain multiple fused sub-word features; For example, the feature of each subword in the article is obtained by comprehensive calculation by integrating these three weight decisions, and the calculation method is shown in formula (6): , (6) in, is a fused subword feature; Represents feature concatenation.

[0030] S126, fusion sub-word features are input into two fully connected networks, the features are mapped to output scores, and the scores corresponding to the sub-words are obtained; For example, for any fused subword feature , the corresponding score is obtained based on the following formula: , (7) in, is the score corresponding to the fused subword feature; and For a fully connected network; Represents the ReLU activation function.

[0031] S13, the entity output network determines the output of multiple entity words related to the image based on the fused sub-word features and corresponding scores (comprehensive decision scores) output by the multiple decision feature fusion module. Specifically, it includes: S131. Since the features in the article are decomposed into subword units for processing during the encoding process, the entity output first needs to summarize the scores of each subword and combine the subword encoding when extracting the article context features to form the score of the entire entity word; For example, the average of the subword scores is used as the final relevance score of the word. The process is shown in formula (8): , (8) in, Represents an entity word; is the score of the ith subword; n represents the number of subwords of the entity word.

[0032] S132, based on the entity word score, the words of the article are selected for entity selection, also using The function masks non-entity words in the article to avoid interference from non-entity words. The process is shown in formula (9): , (9) S133. According to the scores of the entity words, TopK sorting and threshold filtering are used to output entity words above the threshold, so as to obtain multiple entity words related to the image. The threshold is a hyperparameter, and the specific value of the threshold is determined after parameter tuning during training.

[0033] It should be noted that the entity alignment model based on multiple decision fusion is a binary classification task that selects entities by judging whether the article entity is associated with the image. Sample imbalance will also make it difficult for the model to learn the characteristics of positive samples during training, thereby affecting the performance of the alignment model and increasing the missed detection rate. As shown in Table 1, the entity statistics of the commonly used data sets for news image description generation, in the NYTimes800k and GoodNews data sets, the length of the input articles is about 1100 and 490 words respectively, of which the proportion of entities is about 10%, and there are only 5 to 6 entities related to a single image. The number of entities related to images in the article is much lower than the total number of entities, with a gap of about 10 to 20 times. Existing image entity description generation algorithms usually ignore this problem because they do not analyze and study the entity alignment part separately, which in turn affects the effect of entity generation in the description sentence.

[0034] Table 1

[0035] Methods to alleviate the sample imbalance problem include resampling, weight adjustment, and evaluation indicators. Among them, the resampling method is the most direct solution, which achieves sample balance by increasing the number of positive samples in each training batch. However, in the image entity description generation task, each input sample includes an image and an article containing a large number of negative samples, so the commonly used resampling method is difficult to effectively solve this problem.

[0036] Specifically, a training strategy based on multi-task constraints is proposed to simulate resampling training. By introducing constraints on the expression of positive sample features, the model's utilization of positive samples is improved, and the model parameters are further optimized, thereby reducing the impact of sample imbalance. The specific steps include: S141, re-labeling the fused sub-word features, including labeling entity word features related to the image content Labeled as positive samples, entity word features that are irrelevant to the image content Labeled as negative samples; For example, given the image features , input article context features The fused sub-word features output by the multiple decision feature fusion module have three types, including entity features related to image content , entity features that are independent of image content and other non-entity word features During training, are marked as positive samples, and and Labeled as negative samples. Since the multiple decision feature fusion module is The function masks non-entity words in the article. Therefore, in the entity output network, the negative samples only have entity features that are irrelevant to the image content. .

[0037] S142, based on entity word features and , the training objective function of EAM-MDF adopts the cross entropy loss function, and its optimization objective is as follows: , (10) Where Lc is the cross entropy loss function; y represents the entity word output by the multimodal entity alignment model; Represents the model parameters of the multimodal entity alignment model EAM-MDF.

[0038] S143, respectively constructing constraint conditions for reducing the distance between features of similar samples, expanding the distance between features of heterogeneous samples, and expanding the probability distance of score outputs of positive and negative samples; It should be noted that the training goal of the cross entropy loss function is to make the entity features of the positive samples closer to the image features, while making the features of the negative samples away from the image features. However, due to the problem of sample imbalance, negative samples tend to adjust their parameters in the direction away from the image features after a lot of training. In contrast, due to the limited number of positive samples, a small amount of training data is not enough to make their features sufficiently close to the image features, resulting in the distance between the positive samples and the image features not being close enough. This imbalance makes it difficult for the model to accurately judge the label type when facing new samples. In order to improve this situation, the following three additional constraint tasks are introduced to optimize the training of model parameters: (1) Reduce the distance between similar sample features The regularization function is used to reduce the distance between the positive and negative sample features. By aggregating the feature distances between samples, the goal is to make the sample features of the same label have a higher similarity. The calculation is shown in the following formula: , (11) Among them, L1 represents the constraint of reducing the distance between features of similar samples; , Represent the entity word features of positive and negative samples respectively; They represent the positive and negative sample entity word features in shuffled order respectively; n represents the feature dimension, and L represents the number of positive samples.

[0039] (2) Expanding the distance between features of heterogeneous samples The contrast loss constraint is used to expand the distance between positive and negative sample features to achieve effective separation of positive and negative sample features, as shown in the following formula: , (12) Among them, L2 represents the constraint of expanding the distance between the features of heterogeneous samples; is a hyperparameter, which represents the optimized distance between positive and negative sample features. In the experiment, this value is set to 1.

[0040] (3) Expand the probability distance of score output of positive and negative samples The ranking loss is used to further expand the score distance between positive and negative samples in the output probability, which is calculated as follows: , (13) Among them, L3 represents the constraint of expanding the probability distance of positive and negative sample score output; ne represents the number of negative samples; po represents the number of positive samples.

[0041] S144, establishing an objective function based on the cross entropy loss function and the constraint condition, and obtaining the multimodal entity alignment model by multi-task optimization training, which is calculated as shown in the following formula: , (14) Where L is the objective function; are hyper parameters respectively.

[0042] Specifically, in step S2, the multimodal entity alignment model obtained after training is used to implement an entity alignment method of multiple decision fusion based on a second sample set different from the first sample set before description generation. The annotated images in the second sample set are images annotated with description sentences, and candidate entity words (referred to as entities) related to the image content are extracted, which are expressed as: , (15) Where W represents the extracted entity word set, and , Represents the nth subword of an entity word; n represents the total number of subwords of multiple entity words.

[0043] Specifically, in the description generation phase of step S3, the entities extracted by the entity alignment method of multiple decision fusion serve as a bridge between the image and the article, and the image content is associated with the relevant content of the article. The image description generation model based on entity alignment learning (Entity Alignment Learning based Image Captioning Model, EAL-ICM) is used to generate image descriptions.

[0044] Specifically, the goal of the news image description generation task is to learn to generate a description sentence that describes the entity information in the image. Using images, entity words and articles as prompt information generated by the multimodal large model, the image entity description generation process is carried out to obtain the description words describing the entity information in the image. The calculation formula is expressed as: , (16) in, represents the parameters of the multimodal large model, The sentence generated at the past moment is a sentence composed of multiple entity words predicted at the past moment; Indicates the entity word predicted at the current moment. The specific process of training the image description generation model includes: S31. Use the visual encoder and Q-former of the BLIP-2 model to encode the input image samples and obtain image features. ; For example, given an input image and article word collection , and a collection of entities related to the image content , using the pre-trained BLIP-2 model to complete the image description generation task. For image information, the BLIP-2 visual encoder and Q-former are used to input the image Encode features and convert them into features that can be understood by the language model .

[0045] S32, the candidate entity words and the article words obtained by the word segmenter of the BLIP-2 model are encoded using the word embedding layer to obtain article features ; For example, the word set obtained by the word segmenter of the BLIP-2 model and the entity set obtained by the multimodal entity alignment model are concatenated and input into the word embedding layer of the large model for language encoding. .

[0046] S33, the image features and the article features The concatenated input is then fed into the BLIP-2 language model for description prediction. The language model learns the contextual association information between these sequences through the self-attention mechanism, thereby realizing the association between image, article, and entity word features. For example, entity word features, article word features, and image features are concatenated into an input sequence , and input into the language model of BLIP-2 for image description prediction.

[0047] S34. The image description generation model is obtained by training based on maximum likelihood estimation loss and efficient fine-tuning method.

[0048] It should be noted that in terms of parameter fine-tuning, previous methods usually fine-tune all parameters of the entire model or a specific module. For example, NewsMEP and Qu et al. used the GoodNews and NYTimes800k datasets to fine-tune the 400 million parameters in the BART model, and Zhang fine-tuned about 200 million parameters in the Q-former or MLP layer in InstructBLIP or LLaVa- v1.5. However, the method of fine-tuning all parameters for the entire model is inefficient and has limited scalability. Once a larger-scale model is introduced, the existing data is often difficult to meet its fine-tuning requirements. As for the fine-tuning of specific modules, such as the Q-former or MLP layer in InstructBLIP or LLaVa-v1.5, its core role is to convert image features into a form that the language model can understand. However, fine-tuning these parameters often makes it difficult to efficiently capture the interactive information between image and text entities.

[0049] This embodiment is based on the large-model efficient fine-tuning method for entity word prompts, and adjusts parameters from two aspects: first, fine-tuning the parameters of the specific layer of the interaction between the image and text entity, that is, the interaction part in the self-attention mechanism of the image description generation model, accurately adjusting the model parameters to adapt to the image entity description generation task. Second, the fine-tuning parameters are fine-tuned with low-rank re-parameterization to further reduce the amount of fine-tuning parameters and compress the capacity of trainable parameters. Its model network structure is as follows: Figure 4 shown.

[0050] Specifically, when selecting fine-tuning parameters, focus on selecting the feature matrix in the self-attention mechanism for parameter fine-tuning, as shown in the following formula: , (17) in, and Respectively expressed as The query and value mapping matrix parameters of the layer self-attention mechanism, L represents the number of layers.

[0051] The training is based on the loss function of maximum likelihood estimation and the efficient fine-tuning method, which is expressed as: , (18) in, Represents the conditional probability formula, the input includes image content , entity information W, article information A and the given output sentence Y, represents all parameters of BLIP-2, Represents the fine-tuning parameters in BLIP-2.

[0052] In terms of parameter fine-tuning technology, low-rank re-parameter fine-tuning technology (Low-Rank Adaptation, LoRA) is used. Parameters can be fine-tuned efficiently. In BLIP-2, The parameters are mainly fully connected matrices, and the trainable parameter matrix is ​​expressed as , d represents the dimension of the feature. When fine-tuning the full-rank matrix, the number of trainable parameters in the matrix W is , and LoRA provides a low-rank decomposition solution, that is, Decompose into , Two matrices are used to represent this method. For a parameter matrix W, the parameter quantity is Reduce to , when r is much smaller than d, the trainable parameters will be significantly reduced. renew: , (19) in, represents the original weight of the large model, Represents the gradient update matrix. LoRA fine-tunes the original weight matrix by learning the gradient update matrix and Decompose into two matrices B and A for low-rank processing. Finally, through parameter selection and efficient fine-tuning, the trainable parameters account for only 0.6% of the total parameters (18m / 3B).

[0053] Specifically, in step S4, the multimodal entity alignment model is used to obtain the corresponding candidate entity words, and the image description result is obtained by combining the unlabeled image (i.e., the image without labeled description sentence) and the article using the image description generation model. In order to comprehensively evaluate the effectiveness of EAL-ICM, it is compared with seven best performing image entity description generation methods. As shown in Table 2, the performance results of the image entity description generation task on the NYTimes800k dataset are shown. These methods include Tell, VNC, JoGANIC, NewsMEP, UVLP and VAC, among which, Tell: It uses a self-attention network to fuse multiple image features (such as facial features, salient area features, and global features) and article subword features to generate entity descriptions.

[0054] VNC: Introducing an entity encoding module in the process of image and text feature fusion to enhance entity content generation.

[0055] JoGANIC: It uses five news report elements, namely time, people, place, context and other types, to mark entities in articles, and adds five entity types during the decoder to generate templates.

[0056] NewsMEP: The model uses a large visual language model to encode images and texts, and uses relevant entity features as prompts in the decoding stage.

[0057] UVLP: Improves the encoding capabilities of large visual language models and encodes and decodes graphic entities by leveraging pre-trained models trained on more data and tasks.

[0058] VAC: A face naming module is added when encoding image content. By training an interaction module between facial features and entity words, the entity perception ability of the image is improved.

[0059] BLIP-2: Uses the Q-Former structure to convert image input into a large language model input.

[0060] EAL-ICM: This example proposes using entity alignment learning to generate visual and non-visual entities related to the image content, and uses them as prompts for a large multimodal model to improve the accuracy of entity description.

[0061] Table 2

[0062] As shown in Table 2, when using 5% (30,000) data, EAL-ICM, through efficient fine-tuning, makes the large model with 18M parameters reach a level comparable to most current methods. In terms of description generation evaluation indicators (BLEU-4 / Rouge-L / CIDEr), EAL-ICM is basically on par with the other best method VAC. This shows that the model has reached the current highest level in the generation of entity sentences through learning only 30,000 data samples, which fully proves the feasibility of the method proposed in this embodiment in entity description. At the same time, in terms of the generation performance of entity words, EAL-ICM shows the best effect, and the entity recall rate and precision rate are improved by 6.5% and 11.6% respectively compared with other best methods. This result shows that EAL-ICM can generate more and more accurate entity descriptions than other methods. Compared with BLIP-2, EAL-ICM has advantages in all performance indicators in the comparison of multimodal large models of the same level, and the use of entity alignment model and efficient fine-tuning scheme makes EAL-ICM have significant effects.

[0063] Table 3

[0064] As shown in Table 3, the performance comparison of EAL-ICM and existing entity image description generation methods on the GoodNews dataset is shown. In the experiment, EAL-ICM used 10% (30,000) of the training data for training. The results show that although the performance of EAL-ICM in the GoodNews dataset did not show the significant advantages shown on the NYTimes800k dataset, overall, EAL-ICM is close to the best performance in all indicators, especially in terms of entity recall rate, achieving the best results. The reason for the analysis may be related to the characteristics of the GoodNews dataset. The articles in this dataset are short and contain fewer entities, while other methods can also achieve good performance through training with a large number of samples. In this case, EAL-ICM only has an advantage in entity recall rate, but fails to significantly surpass other methods in terms of precision.

[0065] Figure 5-7The comparison results of recall, precision and F1 score of different entity types are shown respectively. In the comparison of model performance, the similar NewsMEP and the classic image entity description generation method Tell are selected as references. In the figure, the square dotted line represents the EAL-ICM model, the line dotted line represents NewsMEP, and the circle dotted line represents the benchmark model Tell. From the overall performance point of view, the EAL-ICM model shows a trend of being better than the NewsMEP and Tell models in all three indicators. This shows that the EAL-ICM model can show excellent performance on different entity types. Especially for the "PERSON" type entity, the EAL-ICM model shows significant advantages in recall and precision, which shows that the model is more accurate in describing the name of the person and has achieved significant improvements in the description of visual entities mainly based on people. However, on some specific entity types, the performance of EAL-ICM is slightly worse than NewsMEP. For example, on the "PERCENT" and "QUANTITY" type entities involving numerical values, EAL-ICM is slightly lower than NewsMEP in terms of recall and F1 score. This result shows that EAL-ICM still has room for improvement in understanding numerical entities.

[0066] As shown in Table 4, the classification of visual and non-visual entity categories in the dataset is shown. According to the classification method of Zhou et al., entity tags representing people ("WHO") and places ("WHERE") are classified as visual entities, that is, these entities can be found in the image. In contrast, entities representing time ("WHEN") and other tags ("MISC") are classified as non-visual entities, which cannot be directly obtained from the image but are associated with visual entities.

[0067] Table 4

[0068] like Figure 8 and Fig. 9 As shown in the figure, the EAL-ICM model outperforms the NewsMEP model and the Tell model in terms of recall, precision and F1 score, whether in the visual entity category or the non-visual entity category. In particular, the accuracy is improved by 19% and 28% respectively compared with the NewsMEP model. This shows that the EAL-ICM model can show significant performance advantages in processing both visual and non-visual entities, effectively constructing the alignment relationship between graphic and text entities in different aspects, reflecting its comprehensive model performance.

[0069] Compared with the prior art, the present embodiment provides a method for generating image descriptions based on multimodal entity alignment. The input image and article are preprocessed through a multimodal entity alignment model, and candidate entity information related to the image is screened out. The screened entity information, image and article are input into the multimodal large model together to generate image entity descriptions, which ultimately improves the recall rate and precision of image entity selection. Combined with the efficient fine-tuning method, the specific layers of the image-text entity information interaction process in the visual language large model are efficiently fine-tuned, which can achieve performance comparable to the existing highest method with a small amount of data, thereby improving training efficiency.

[0070] Those skilled in the art will appreciate that all or part of the processes of the above-mentioned embodiments can be implemented by instructing related hardware through a computer program, and the program can be stored in a computer-readable storage medium, wherein the computer-readable storage medium is a disk, an optical disk, a read-only storage memory, or a random access memory, etc.

[0071] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by any technician familiar with the technical field within the technical scope disclosed by the present invention should be covered within the protection scope of the present invention.

Claims

1. A method for generating image description based on multimodal entity alignment, characterized in that: The steps include: Based on the first sample set containing the images with annotated entities and the articles related thereto, a multimodal entity alignment model is trained; Based on a second sample set including the annotated images and articles related thereto, using the multimodal entity alignment model, obtaining entity aligned candidate words; Based on the second sample set and the candidate entity words, training an image description generation model based on multimodal entity alignment; Based on the images and articles without labeled descriptions, the multimodal entity alignment model is used to obtain corresponding candidate entity words, and combined with the images and articles without labeled descriptions, the image description generation model is used to obtain image description results.

2. The method for generating image description based on multimodal entity alignment according to claim 1, characterized in that: The multimodal entity alignment model includes a feature extraction module, a multiple decision feature fusion module and an entity output network, wherein: A feature extraction module is used to extract features from the sample set of images containing annotated entities and articles related thereto, respectively, to obtain image features and article context features, wherein the article context features include multiple article subword features; A multiple decision feature fusion module is used to perform cross-modal feature fusion on the image features and the article context features to obtain multiple fused sub-word features and corresponding scores; The entity output network is used to obtain multiple entity words related to the image based on the fused sub-word features and corresponding scores.

3. The method for generating image description based on multimodal entity alignment according to claim 2, characterized in that: Obtaining the fused subword features and corresponding scores includes: Using the spacy tool to respectively annotate entity word features and a set of non-entity word features in the context features of the article; Based on the image features, the query layer mapping of the self-attention mechanism is used to obtain the image hidden features Based on the context features of the article, the three mapping layers of query, key and value of the self-attention mechanism are used to obtain the hidden features of the article, which can be expressed as ; and obtaining entity context features based on the set of entity word features; Based on the image hidden features and the hidden features of the article , and mask the non-entity word features in the article context features, calculate the similarity between the image features and the entity word features, and obtain the image attention weight ; Based on the hidden features of the article , calculate the similarity between the context features of the article and get the article attention weight ; Mask the non-entity word features in the article context features, calculate the similarity between entity context features, and obtain the entity attention weight ; Based on three attention weights, the latent features of the article are Perform feature fusion to obtain multiple fused sub-word features; The fused sub-word features are passed through two fully connected networks to obtain the corresponding scores of the fused sub-words.

4. The method for generating image description based on multimodal entity alignment according to claim 2, characterized in that: The multiple entity words related to the image include: Based on the average of the corresponding scores of each fused subword, the corresponding entity word score is obtained; Based on the entity word score, using The function masks non-entity words in the article and performs entity selection on the words in the article; Based on the scores of multiple entity words after entity selection, multiple entity words related to the image are obtained by using TopK sorting and threshold filtering.

5. The method for generating image description based on multimodal entity alignment according to claim 2, characterized in that: The feature extraction module uses the CLIP model to extract features from the input image samples to obtain image features; and uses the T5 model to extract text features from the input article samples to obtain article context features.

6. The method for generating image description based on multimodal entity alignment according to claim 2, characterized in that: The multi-modal entity alignment model is trained using a multi-task constrained training strategy, including: Re-label the fused sub-word features, including labeling entity word features related to the image content Labeled as positive samples, entity word features that are irrelevant to the image content Labeled as negative samples; Based on the entity word features and , calculate the cross entropy loss function; Construct constraints to reduce the distance between features of similar samples, expand the distance between features of heterogeneous samples, and expand the distance between the probability of score outputs of positive and negative samples; An objective function is established based on the cross entropy loss function and the constraint conditions, and the multimodal entity alignment model is obtained by multi-task optimization training.

7. The method for generating image description based on multimodal entity alignment according to claim 6, characterized in that: The expressions of the constraints are: ， ， ， Among them, L1 represents the constraint of reducing the distance between features of similar samples; , Represent the entity word features of positive and negative samples respectively; They represent the positive and negative sample entity word features in a shuffled order; n represents the feature dimension; L represents the number of positive samples; L2 represents the constraint of expanding the distance between the features of heterogeneous samples; is a hyperparameter, which indicates the optimized distance between the features of positive and negative samples; L3 indicates the constraint of expanding the probability distance of the output scores of positive and negative samples; Represents the parameters of the multimodal entity alignment model; ne represents the number of negative samples; po represents the number of positive samples.

8. The method for generating image description based on multimodal entity alignment according to claim 7, characterized in that: The expression of the objective function is: ， ， Where y represents the entity word output by the multimodal entity alignment model; Lc is the cross entropy loss function; L is the objective function; are hyper parameters respectively.

9. A method for generating image description based on multimodal entity alignment according to any one of claims 1 to 8, characterized in that: The image description generation model is obtained by training, including: Use the visual encoder and Q-former of the BLIP-2 model to encode the input image samples and obtain image features ; The candidate entity words and the article words obtained by the word segmenter of the BLIP-2 model are encoded using the word embedding layer to obtain the article features ; The image features and the article features The concatenated input is then fed into the BLIP-2 language model for description prediction; The image description generation model is trained based on maximum likelihood estimation loss and an efficient fine-tuning method.

10. The method for generating image description based on multimodal entity alignment according to claim 9, characterized in that: The efficient fine-tuning method is to fine-tune the specific layer parameters of the image description generation model using low-rank re-parameter fine-tuning technology, and the specific layer parameters are feature parameters in the self-attention mechanism of the image description generation model.

Images