Lightweight image description system based on heterogeneous mapping network and multi-level feature enhancement

Abstract

The invention discloses a lightweight image description system based on a heterogeneous mapping network and multi-level feature enhancement, which comprises the following steps of: constructing the system by adopting a basic framework of an encoder-decoder, and respectively adopting a maturely trained multi-modal large model CLIP and a language large model GPT-2 as an encoder and a decoder of the system. A parallel heterogeneous mapping network channel is arranged between an encoder and a decoder so as to realize more comprehensive feature extraction and feature mapping of image features. The mapping output of each mapping channel is connected with a decoder by using a cross attention mechanism to form a first-layer feature enhancement mechanism, and the integrated output formed by splicing the mapping outputs of the three channels is connected with the decoder to form a second-layer feature enhancement mechanism. And carrying out overall design on the implementation process by utilizing the designed lightweight framework, the heterogeneous mapping network and the multi-level feature enhancement mechanism to form an overall model to realize automatic description of the image.

Description

Technical Field

[0001] The present invention relates to the field of multimodality, and in particular to a lightweight image description system based on a heterogeneous mapping network and multi-level feature enhancement. Background Art

[0002] The image description task is one of the important tasks in the cross-modal field. That is, given a picture, automatically generate a text description for the picture to summarize the content of the picture. In real applications, the rapid development of this task can effectively help the lives and travel of visually impaired people.

[0003] Unlike single image and text tasks such as single object detection and semantic understanding, the image description task cannot rely on feature learning of a single modality such as image or text to complete the task. It is a multimodal task. This requires that the model that performs the image description task not only needs to learn the data features of the image and text modalities at the same time, but also needs to be able to establish a specific mapping relationship between the two modalities, so that when the model detects a certain type of object in a given picture, it can generate correct text to describe the object and describe the relationship between the object and other objects in the picture. This obviously places higher requirements on the model that performs the image description task.

[0004] The current mainstream image description model adopts the encoder-decoder method, that is, a visual semantic understanding model is used as the encoder of the model, and an autoregressive language model that can generate clear semantics and grammar is used as the decoder; excellent image description models are mostly from visual language models pre-trained on large-scale image-text pairs. Such models have good generalization for multimodal features, and then fine-tuned to downstream tasks, such as image description, visual question answering, etc. However, such models need to rely on a large amount of computing resources and training data, which is not very friendly to fine-tuning of downstream tasks.

[0005] In response to the problem of increasing model size, Mokady et al. proposed a new lightweight model idea by using the pre-trained visual encoder CLIP and language decoder GPT-2, and freezing most of the parameters during training. However, freezing the encoder and decoder at the same time will cause the problem of feature offset between the two. Therefore, this method adds a mapping network between the encoder and decoder to generate a fixed-length prefix for each description to eliminate the feature offset between the two modules. This method effectively alleviates the feature offset between image and text features, but its limitation is that fixed-length vectors cannot capture subtle visual semantic information. Ramos et al. continued the idea of ​​the former, using pre-trained CLIP and GPT-2 as the main framework of the model, but the difference is that the latter uses a cross-attention structure to connect the encoder and decoder. This method aims to better utilize the benefits of encoder embedding for description generation, and find and enhance the parts of image features that can better guide the decoder to generate descriptions. However, it is often difficult to effectively align the feature offset between visual and text features.

[0006] Heterogeneous mapping networks aim to comprehensively utilize neural networks of different structures to more comprehensively and systematically capture the intrinsic connection between images and texts, which is used to guide the generation of description content. Existing excellent lightweight models mostly design unique mapping modules between encoders and decoders. These mapping modules with different structures give these methods different advantages and characteristics. In order to utilize neural network models of different structures and concentrate the advantages of these models, common methods include (1) treating each neural network model as an independent individual for prediction, and averaging the word probabilities predicted by different neural network models to obtain a comprehensive prediction result. (2) Concatenating the embedding sequences obtained by different models, treating it as a complete embedding sequence and inputting the connection view into the decoder model to generate the corresponding description.

[0007] Existing methods usually use the average method to synthesize the contributions of different mapping networks, but this method usually cannot effectively utilize the advantages of mapping networks with different structures, and different mapping networks usually have different performances when facing different data distributions. Therefore, it is one-sided to use the average value alone to summarize the contributions of different mapping networks. Kuo et al. use the cross-attention mechanism to automatically assign different weights to each mapping network channel, and guide the mechanism to automatically adjust the contributions of different mapping network channels through the generation effect of the decoder, so as to better guide the generation of description content. Summary of the invention

[0008] The purpose of the present invention is to realize a lightweight image description system based on a heterogeneous mapping network and multi-level feature enhancement, so as to automatically generate appropriate descriptions for images to play a role in tasks in different fields and solve the problems mentioned in the above background technology.

[0009] To achieve the above objectives, this method designs the following lightweight image description system based on heterogeneous mapping network and multi-level feature enhancement, including the following steps:

[0010] Step S1: The system is built using the basic framework of encoder-decoder. This method uses the mature multimodal large model CLIP and the language large model GPT-2 as the encoder and decoder of the system respectively, and freezes most of the parameters in the system during the training process to achieve lightweight system.

[0011] Step S2: Based on step S1, a parallel heterogeneous mapping network channel is set between the encoder and the decoder to achieve more comprehensive feature extraction and feature mapping of image features.

[0012] Step S3: Based on steps S1 and S2, the mapping output of each mapping channel is connected to the decoder using the cross attention mechanism to form the first layer feature enhancement mechanism, and the comprehensive output after splicing the mapping outputs of the three channels is connected to the decoder to form the second layer feature enhancement mechanism.

[0013] Step S4: The lightweight framework, heterogeneous mapping network and multi-level feature enhancement mechanism designed in steps S1, S2 and S3 are used to design the overall implementation process and form an overall model to realize the automatic description of the image.

[0014] Figure 1 The Encoder and Decoder parts in the figure are the main framework of this method, which are used to extract basic features of the image and generate the final description. Here, this method freezes most of the parameters and does not participate in the parameter update link during system training, which greatly saves the system training time and realizes the lightweight of the overall system.

[0015] After the Encoder module, this method designs several parallel feature mapping channels. Existing lightweight methods of this type mostly set up a fixed mapping network, such as convolutional neural network (CNN), multi-layer linear perceptron (MLP), etc. The inductive bias of these networks makes them advantageous in processing a certain type of information, but it also limits these networks from capturing a wider range of information. To solve this problem, this method sets |M| networks with different inductive biases in parallel to map the image semantic information extracted by the encoder. For the input image X, Enc(·) represents the primary features of the image extracted by the Encoder module, and Map i (·) represents the features after mapping of each i-th channel, as follows:

[0016] m i =Map i (Enc.(X))

[0017] This method inputs the image into the pre-trained model Encoder module for encoding, and then uses a parallel heterogeneous network for mapping encoding to more comprehensively extract the semantic information in the image from different dimensions.

[0018] This method designs a multi-level mapping network enhancement module, that is, through two layers of cross attention layers, the intra-channel enhancement and inter-channel enhancement of the mapping network embedding are realized. In the first layer, the output m of each channel mapping network is i Perform cross attention operation on the decoder hidden state input as follows:

[0019] Q=W q Y+b q

[0020] K=W k M+b k

[0021] V=W v M+b v

[0022]

[0023] Where Y is the hidden state input of the image description text used for training, and M is the set of mapping networks. In the first layer of cross-attention calculation, the hidden state input of the decoder is mapped as the query vector, and the output M from the heterogeneous mapping module is calculated as the key and value. The generated description is used to guide each mapping network to generate encoding that is conducive to description generation, thereby enhancing the mapping ability of each mapping network channel. In the second layer, we first concatenate the output of the mapping network of each channel:

[0024]

[0025] The concatenated mixed channel matrix is ​​cross-attended again, as shown below:

[0026]

[0027] Here, the decoder hidden state input is used as the query vector again to perform channel-level screening on the heterogeneous mapping module. We make a summary of this module. The multi-level mapping network enhancement module is implemented through a two-layer cross-attention mechanism. The first layer enhances the encoding of the mapping network within the channel, and the second layer of the cross-attention mechanism achieves inter-channel enhancement by adaptively adjusting the parameters of different channels, thereby better guiding description generation.

[0028] The overall structure of the model is as follows Figure 1 As shown in the figure, the pre-trained encoder and decoder are used to realize the preliminary extraction of image features and the final generation of image description. After obtaining the preliminary features using the encoder, we input them into the heterogeneous mapping network module set up in parallel, and rely on mapping networks with different structures to extract specific features of the initial features from different angles. At the same time, the first-layer feature enhancement mechanism is used to guide the mapping network to generate decoder-friendly mapping features within the channel, and then the specific features of each channel are spliced. The second-layer feature enhancement mechanism is used to screen and enhance the spliced ​​features between channels. According to the different friendliness of different mapping features to the decoder, weights are automatically assigned to them and they are selectively enhanced. The comprehensive features that can best help the decoder to generate descriptions are input into the decoder to guide the decoder to generate descriptions of the corresponding images. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 It is the overall architecture of the designed model.

[0030] Figure 2 It is a comparison of the effects of different mapping network combinations of the present invention.

[0031] Figure 3 This is a comparison of the effects of the model invented in this article and other models. DETAILED DESCRIPTION

[0032] The following will be combined with the drawings in the present invention to clearly and completely describe the technical solutions and implementation details in the specific implementation process of the present invention. Obviously, the described implementation examples are only part of the implementation examples of the present invention, not all the embodiments; based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention. The implementation plan of this experiment is as follows:

[0033] Step S1: Determine the experimental training dataset. In order to determine the generalization and versatility of the system, this method will be trained on two mainstream evaluation datasets. Among them, COCO: This dataset contains more than 120,000 pictures and more than 500,000 descriptions. Each picture contains five different descriptions. We use the division method proposed by Karpathy et al., in which 5,000 pictures are used for verification, 5,000 pictures are used for testing, and the remaining pictures are used for training; Flickr: This dataset contains 31,783 pictures that capture people in daily activities and events. Each picture has a descriptive title.

[0034] Step S2: Determine the basic selection of each module and related parameter settings and statistics. The main framework of this system uses CLIP-ViT-B / 32 as the encoder, GPT-2 Base As a decoder, we designed a heterogeneous mapping network aggregation module between the encoder and the decoder to extract more comprehensive features of the image. Here we designed three heterogeneous mapping channels, namely MLP-based, CNN-based and transformer-based mapping channels. In the CNN-based channel, we adopted the Bottleneck structure in ResNet. In the transformer-based channel, we used 12 layers of transformer encoder layers to build the channel.

[0035] Step S3: During the training phase, we freeze the encoder and decoder and only train the heterogeneous mapping channel and the multi-level enhancement module, forming a lightweight image description model with about 32M training parameters. During the training process, we use cross entropy loss as a penalty, AdamW optimizer and set the batch size to 64, and the initial learning rate to 1e-4. We will train on an RTX 4090 GPU for 10 cycles, which will take up about 16GB of video memory and take about 10 hours.

[0036] Step S4: Model testing. In the model testing phase, this method uses two datasets, COCO and Flickr, to evaluate the model. Figure 2 As shown in the figure, the model trained on Flickr by this method is compared with the model trained on the same dataset by the general model. The test pictures are also from the Flickr dataset, where GT(1) and GT(2) represent the two description labels of the picture, i.e., the standard answers. The comparison shows that the invented model can pay more attention to the detailed features in the image compared with the general method, and the generated description is relatively more fluent, such as the use of verbs and the arrangement of subject, predicate and object. Then we conducted an ablation test on the invented model to evaluate the contribution of different mapping channels to the system model, such as Figure 3As shown in the figure, the model without the Transformer mapping channel has the largest loss in accuracy compared to the complete model. This shows that the Transformer-based mapping network contributes the most to the system model. At the same time, we use the following four evaluation indicators to evaluate the invented model:

[0037]

[0038] Among them, candidates are generated description sentences, the given standard description sentences are references, n-gram is a word group of length n, Count clip (n-gram) indicates how many n-grams appear in references, and Count(n-gram′) indicates how many n-grams in a given generated description appear in the standard description sentence. When n=1, this formula is used to measure the accuracy at the word level, and higher-order formulas are used to measure the fluency of the sentence.

[0039] Pen=y·frag β

[0040]

[0041] METEOR = (1-Pen)·F mean

[0042] Among them, P and γ are the precision and recall of the words in the generated description, α and γ are weight parameters, and Pen is the penalty factor for different word order settings. This formula calculates specific sequence matching, synonyms, roots and affixes, and the matching relationship between interpretations, while considering the precision and recall, making this formula more relevant to manual judgment.

[0043]

[0044] where g n (c i ) indicates that in the description statement c i All n-grams of ω k TF-IDF weight vector. This formula is combined with the spatial model. The cosine similarity between each generated description is calculated, and different weights are assigned to different words, which can better evaluate the generation of key information and reduce the impact of non-key sentences on the results.

[0045]

[0046] Where G(·) represents the conversion of text into a scene graph, and T(·) represents the conversion of the scene graph into a set of tuples. This formula mainly evaluates the similarity of nouns and is therefore suitable for image description tasks.

[0047] The main innovation of the present invention is to design a heterogeneous mapping network module to extract more comprehensive and integrated features from images, and to design a hierarchical feature enhancement module to enhance the features of the channels with greater contributions and further enhance the output comprehensive results. By using several different evaluation indicators to conduct comprehensive evaluation on several general data sets, it is proved that the lightweight image description system designed by the present invention has better performance.

[0048] It is obvious to those skilled in the art that the present invention is not limited to the details of the above-mentioned exemplary embodiments, and that the present invention can be implemented in other specific forms without departing from the spirit or essential features of the present invention; therefore, no matter from which point of view, the embodiments should be regarded as exemplary and non-restrictive, and the scope of the present invention is limited by the appended claims rather than the above description, and therefore it is intended to include all changes within the meaning and scope of the equivalent elements of the claims within the present invention; any figure mark in the claims should not be regarded as limiting the claims involved.

[0049] In addition, it should be understood that although the present specification is described according to implementation modes, not every implementation mode contains only one independent technical solution. This description of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment may also be appropriately combined to form other implementation modes that can be understood by those skilled in the art.

Claims

1. A lightweight image description method based on heterogeneous mapping network and multi-level feature enhancement, characterized in that: The steps include: Step S1: The system is built using the basic framework of encoder-decoder, and the mature multimodal large model CLIP and the language large model GPT-2 are used as the encoder and decoder of the system respectively; Step S2: Based on step S1, a parallel heterogeneous mapping network channel is set between the encoder and the decoder to achieve more comprehensive feature extraction and feature mapping of image features; Step S3: Based on steps S1 and S2, the mapping output of each mapping channel is connected to the decoder using a cross attention mechanism to form a first-layer feature enhancement mechanism, and the comprehensive output after the three channel mapping outputs are spliced ​​is connected to the decoder to form a second-layer feature enhancement mechanism; Step S4: The lightweight framework, heterogeneous mapping network and multi-level feature enhancement mechanism designed in steps S1, S2 and S3 are used to design the overall implementation process and form an overall model to realize the automatic description of the image.

2. The lightweight image description method based on heterogeneous mapping network and multi-level feature enhancement according to claim 1 is characterized in that: |M| networks with different inductive biases are set up in parallel to map the image semantic information extracted by the encoder; for the input image X, Enc(·) represents the primary features of the image extracted by the Encoder module, and Map i (·) represents the features after mapping of each i-th channel, as follows: m i =Map i (Enc.(X)) The image is input into the pre-trained model Encoder module for encoding, and then a parallel heterogeneous network is used for mapping encoding to more comprehensively extract the semantic information in the image from different dimensions.

3. The lightweight image description method based on heterogeneous mapping network and multi-level feature enhancement according to claim 2 is characterized in that: A multi-level mapping network enhancement module is designed, that is, two layers of cross attention layers are used to achieve intra-channel enhancement and inter-channel enhancement of the mapping network embedding; in the first layer, the output m of each channel mapping network is i Perform cross attention operation on the decoder hidden state input as follows: Q=W q Y+b q K=W k M+b k V=W v M+b v Where Y is the hidden state input of the image description text used for training, and M is the set of mapping networks. In the first layer of cross-attention calculation, the hidden state input of the decoder is mapped as the query vector, and the output M from the heterogeneous mapping module is calculated as the Key and Value. The generated description is used to guide each mapping network to generate encoding that is conducive to description generation, thereby enhancing the mapping ability of each mapping network channel. In the second layer, the output of the mapping network of each channel is first spliced: The concatenated mixed channel matrix is ​​cross-attended again, as shown below:

4. The lightweight image description method based on heterogeneous mapping network and multi-level feature enhancement according to claim 3 is characterized in that: The decoder hidden state input is used as the query vector to perform channel-level screening on the heterogeneous mapping module to make a summary statement of this module. The multi-level mapping network enhancement module is implemented through a two-layer cross-attention mechanism. The first layer enhances the encoding of the mapping network within the channel, and the second layer of the cross-attention mechanism achieves inter-channel enhancement by adaptively adjusting the parameters of different channels, thereby better guiding description generation.

5. The lightweight image description method based on heterogeneous mapping network and multi-level feature enhancement according to claim 4 is characterized in that: The invented model is evaluated using the following four evaluation metrics: Among them, candidates are generated description sentences, the given standard description sentences are references, n-gram is a word group of length n, Count clip (n-gram) indicates how many n-grams appear in references, and Count(n-gram') indicates how many n-grams appear in the standard description sentence in a given generated description. When n=1, it is used to measure the accuracy at the word level, and higher-order formulas are used to measure the fluency of the sentence. Pen=γ·frag β METEOR=(1-Pen)·F mean Where P and R are the precision and recall of the words in the generated description, α and γ are weight parameters, and Pen is the penalty factor for different word order settings. This formula calculates specific sequence matching, synonyms, roots and affixes, and matching relationships between interpretations, while considering both precision and recall, making this formula more relevant to manual judgment; where g n (c i ) indicates that in the description statement c i All n-grams of ω k TF-IDF weight vector; This formula is combined with the spatial model; Calculate the cosine similarity between each generated description, and assign different weights to different words, which can better evaluate the generation of key information and reduce the impact of non-key sentences on the results; Where G(·) represents the conversion of text into a scene graph, and T(·) represents the conversion of a scene graph into a set of tuples.

Images