Explainable image-to-text description method based on multi-model fusion

Abstract

The application discloses an interpretable image text description method based on multi-model fusion, feature extraction is performed on an input image, then the extracted image features are input into a prediction model, and finally a prediction result is obtained.In terms of data sets, a high-quality data set is formed by inputting strict prompts into ChatGPT to generate corresponding pictures.In terms of models, a CLIP model with an added parsing attention mechanism is applied to extract high-level features of the input image, and its powerful cross-modal understanding capability provides support for subsequent tasks.Three models are applied with LoRA fine-tuning technology, so that the models can more flexibly adapt to the feature relationship of different regions and flexibly adjust the attention in local areas, thereby reducing the computational burden of the model.Finally, the prediction accuracy is improved through the integrated fusion of the prediction results of the three models.In addition, the different parts of the model are modularly designed, improving the scalability and flexibility of the model.

Description

Technical Field

[0001] This invention belongs to the technical field of computer vision and natural language processing methods, specifically relating to an interpretable image textual description method based on multi-model fusion. Background Technology

[0002] In computer vision and natural language processing, image-to-text (image2text) technology aims to understand and describe image content. While humans can quickly summarize a complex image scene in a few sentences within minutes, this process is far more difficult for computers. The main challenges in this field include the complexity and diversity of image content and how to accurately and naturally convert visual information into textual descriptions. Current technologies struggle to fully understand the details, emotions, and contextual information within images, especially when dealing with abstract concepts, metaphors, or complex scenes. Furthermore, current technologies have limitations in explaining the internal workings of model layers, such as decision paths, weight allocation, and activation functions, which restricts a deeper understanding of the model layers. In machine learning, a precise understanding of these internal structures is crucial for enhancing model credibility and improving decision-making transparency. Relying on 'black box' models for critical decisions can be risky, especially in applications that could impact people's lives. Therefore, developing machine learning models that offer higher levels of interpretability to better examine and validate their internal mechanisms is of paramount importance. In addition, its generalization ability, real-time processing ability and accuracy still need to be improved to meet the growing application needs, such as assisting visually impaired people and intelligent content creation, which have high requirements for the accuracy and efficiency of model recognition.

[0003] Existing processing models work by extracting image features and converting them into textual descriptions. While some progress has been made, existing methods suffer from several major shortcomings: First, the models are heavily reliant on training data. The commonly used dataset is the COCO dataset developed and maintained by Microsoft in 2014, which lacks diversity; its images may be biased towards certain regions (e.g., North America), neglecting scenes and object types in other regions. Furthermore, with technological advancements and societal changes, the relevance and timeliness of datasets gradually decrease. These factors lead to insufficient generalization ability of the models, with significant performance degradation when faced with unfamiliar image types or scenes. Finally, because most models employ adaptive neural network methods, establishing more complex image analysis networks, the extracted image features possess anonymity, often resulting in a lack of interpretability.

[0004] Based on the current state of technology, existing image2text models exhibit certain limitations in generalization ability. These models typically perform well on training datasets, but their performance often suffers when applied to new data outside the training dataset, making it difficult for them to effectively handle and recognize image types or scenes not encountered during training. Furthermore, the high computational resources consumed by these models result in a lack of real-time performance. In current research, widely used datasets such as Microsoft COCO, while historically valuable, have outdated data structures and lack reflection of the latest social environments and technological developments. This lack of data timeliness limits their application in the context of a rapidly changing modern society. Due to limitations in understanding and interpreting complex elements, levels of detail, and abstract concepts in images, existing technologies may lack accuracy in generating text descriptions, especially when describing complex or abstract image content, leading to omissions or misunderstandings. Moreover, existing models often lack diversity in generating text descriptions due to architectural limitations. This means that when dealing with diverse and uncommon image content, the models may fail to generate creative and adaptable text descriptions. Summary of the Invention

[0005] The purpose of this invention is to provide an interpretable image textual description method based on multi-model fusion, which solves the problem of existing limitations in computing resource requirements.

[0006] The technical solution adopted in this invention is an interpretable image textual description method based on multi-model fusion. The input image is subjected to feature extraction, and then the extracted image features are input into the prediction model to obtain the prediction result.

[0007] The invention is further characterized by:

[0008] The specific steps are as follows:

[0009] Step 1: Initialize the preprocessed dataset;

[0010] Step 2, Model Preparation;

[0011] Step 3: Instantiate the predictor;

[0012] Step 4: Integrate and fuse the prediction results.

[0013] Step 1 uses the DiffusionDB_2M dataset, prioritizing datasets relevant to the image description task. Duplicate data with similarity scores higher than 0.8 are removed. High-quality and diverse prompts are then input into ChatGPT to generate more compliant images, ensuring a one-to-one correspondence between the input prompts and the generated images. The input prompts are dynamically adjusted based on the model training results, ultimately forming a high-quality prompt dataset.

[0014] Step 2 defines the model architecture, adds a custom attention mechanism to the model, and defines two PyTorch modules.

[0015] In step 3, the predictor loads the trained model, including reading the model file stored in a specific path and loading it into memory. After the model is loaded, the predictor also loads the dataset to be predicted, including reading data from a database, file system, or network source, and performing necessary preprocessing to ensure that the data format is consistent with the training format. Then, the data is passed forward by the predictor through the model. During this process, the data passes through each layer of the model, and the model analyzes and processes the data based on its learned features and patterns. After the forward pass is completed, the model outputs the original prediction result.

[0016] The beneficial effects of this invention are:

[0017] This invention presents an interpretable image textual description method based on multi-model fusion. Regarding the dataset, a high-quality dataset is generated by providing stringent prompts to ChatGPT. For the models, the CLIP model with an added parsed attention mechanism is applied to extract high-level features from the input images, leveraging its powerful cross-modal understanding capabilities to support subsequent tasks. LoRA fine-tuning techniques are applied to the three models respectively, enabling them to more flexibly adapt to feature relationships in different regions and adjust attention flexibly in local regions, thereby reducing the computational burden on the models. Finally, the prediction accuracy is improved by integrating and fusing the prediction results of the three models. Furthermore, modular design of different parts of the model enhances its scalability and flexibility. Attached Figure Description

[0018] Figure 1 This is a flowchart of the algorithm model of the present invention;

[0019] Figure 2 This is a flowchart of the LoRALinearLayer and LoRALinear algorithm of this invention;

[0020] Figure 3 This is a flowchart of the LoRA algorithm applied to the BLIP2 model layer of this invention;

[0021] Figure 4 This is a flowchart of the LoRA algorithm applied to the ConvNeXtxxlarge model layer of this invention;

[0022] Figure 5 This is a flowchart of the forward propagation process of the LoRACLIPAttention module of this invention. Detailed Implementation

[0023] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0024] This invention provides an interpretable image textual description method based on multi-model fusion, such as... Figure 1 As shown, the input image undergoes feature extraction, the extracted image features are then input into the prediction model, and finally the prediction result is obtained.

[0025] The specific steps are as follows:

[0026] Step 1: Initialize the preprocessed dataset;

[0027] The DiffusionDataset class is used to load image data and target embedding vectors. To enhance the accuracy of model recognition, diverse and high-quality data is crucial. To achieve this, step 1 uses the DiffusionDB_2M dataset, prioritizing datasets relevant to the image description task, including COCOCaptions, VizWiz, Open Image, ADE20K, Flickr30K, and TextCaps. Duplicate data with similarity scores higher than 0.8 are removed. High-quality and diverse prompts are then input into ChatGPT to generate more compliant images, ensuring a one-to-one correspondence between input prompts and generated images. The input prompts are dynamically adjusted based on the model's training results, ultimately forming a dataset of approximately 2 million high-quality prompts, termed "PROMPTS_HQ".

[0028] For each cue, two images are generated using a different random seed. COYO-700M is a large-scale dataset containing 747M image-text pairs, and this dataset was used for pre-training. From this large dataset, approximately 6.6 million samples with a similarity greater than 0.3 between the images and text were selected, and duplicates were removed. These texts were then used as cue points and fed into Stable Diffusion. This dataset is called "PROMPTS_LQ".

[0029] Step 2, Model Preparation;

[0030] In step 2, the architecture of the model is defined, a custom attention mechanism is added to the model, and two PyTorch modules are defined.

[0031] Step 3: Instantiate the predictor;

[0032] In step 3, the predictor loads the trained model, including reading the model file stored in a specific path and loading it into memory. After the model is loaded, the predictor will also load the dataset to be predicted, including reading data from a database, file system, or network source and performing necessary preprocessing to ensure that the data format is consistent with the training format. Then the data is passed forward by the predictor through the model. During this process, the data passes through each layer of the model. The model analyzes and processes the data according to its learned features and rules. After the forward pass is completed, the model outputs the original prediction result.

[0033] Step 4: Integrate and fuse the prediction results.

[0034] This invention provides an interpretable image textual description method based on multi-model fusion, as illustrated in the following embodiments:

[0035] Example 1

[0036] In step 2, the model architecture is defined, a custom attention mechanism is added, and two PyTorch modules, LoRALinearLayer and LoRALinear, are defined. These modules implement a special type of neural network layer called LoRA (Low-Rank Adaptation), as follows: Figure 2 As shown. LoRA is a technique for tuning pre-trained deep learning models. Instead of directly fine-tuning all weights, it modifies the model's weights by introducing a low-rank matrix, significantly reducing the number of parameters that need to be trained, thus saving memory and computing resources.

[0037] LoRA technology was applied to the BLIP2, ConvNeXtxxlarge, and CLIP model layers to improve model adaptability and performance. Specifically, LoRA technology was applied to certain layers of the BLIP2 model. Figure 3 As shown. First, calculate the total number of encoding layers in the model, then determine the number of layers to use LoRA based on the LoRA rate. Based on the number of layers, traverse the last few layers in the encoder. For each layer, obtain the current attention layer and multilayer perceptron (MLP) layer, replace the attention layer with a LoRA attention layer, and for each fully connected layer of the MLP, use LoRALiner to create a LoRA version and replace the original layer.

[0038] LoRA techniques applied to specific layers of the ConvNeXtxxlarge model, such as Figure 4As shown, the main modifications to the ConvNeXt-XXLarge model are to the convolutional layers and MLP layers in its tail stage. This selective adjustment helps to effectively fine-tune the model while maintaining its core structure and pre-trained knowledge, making the model more interpretable.

[0039] The forward propagation process of the LoRACLIPAttention module applied to the CLIP model layer is as follows: Figure 5 As shown, the module receives the input hidden state, which is transformed into a corresponding vector through query, key, and value projection layers. LoRA (Low-Rank Adaptation) is used to scale the query and key vectors, enhancing the model's expressive power. Next, a tensor is prepared for attention mechanism computation through a reshaping operation. Causal attention masks and additional attention masks are applied to these vectors to ensure correctness and focus of attention during sequence generation. Attention weights are calculated using the Softmax function, and LoRA scaling is applied again. To reduce overfitting, a Dropout mechanism is introduced. Finally, the calculated attention weights are combined with the value vector to generate the attention output, which is then passed through a linear projection layer to obtain the final module output. This process optimizes the model's computational efficiency and significantly improves the accuracy of image feature extraction.

[0040] Example 2

[0041] In step 3, the predictor is instantiated, which involves creating an instance of the Predictor class. This process is not simply about instantiating a class; it ensures that the predictor can effectively and accurately manage the entire prediction process. To this end, various parameters of the predictor are carefully configured, including the model loading path, preprocessing and postprocessing configurations, and any necessary hardware optimization settings, such as GPU acceleration. These configurations guarantee that the predictor can process data quickly and accurately.

[0042] The predictor is next responsible for loading the trained model. This step typically involves reading the model file stored at a specific path and loading it into memory. After the model is loaded, the predictor will also load the dataset to be predicted. This includes reading data from a database, file system, or network source and performing necessary preprocessing, such as scaling, cropping, or normalization, to ensure that the data format is consistent with the format used during training.

[0043] The data is then passed forward by the predictor through the model. During this process, the data passes through various layers of the model, which analyzes and processes the data based on its learned features and patterns. After the forward pass is complete, the model outputs the raw prediction results.

[0044] Example 3

[0045] In step 4, the prediction results of the three models are integrated and fused to obtain a powerful prediction model that combines the advantages of each individual model. This integration method not only combines the unique capabilities of each model in handling different features and data, but also enhances the robustness of the final model in the face of various data variations. Specifically, this can be achieved through various integration techniques, such as averaging, weighted averaging, stacking, blending, or voting mechanisms (soft voting and hard voting).

[0046] This invention is based on a multi-model fusion-based interpretable image textual description method. It utilizes ChatGPT to construct a large dataset, increasing the accuracy and diversity of the training image set and making the dataset more relevant to everyday use cases. The model incorporates an improved attention mechanism analysis layer—the CLIP model—that analyzes and interprets its internal attention layers. This improved attention mechanism provides in-depth analysis and interpretation of the model's internal attention layers. By dynamically adjusting weights, this mechanism accurately reflects the interdependencies between different input features. Whenever the model makes a prediction, this mechanism provides a clear explanation path, detailing which features play a key role in the prediction result and how these features interact to form the final output, thus improving the interpretability of the prediction results when processing image input. Furthermore, through this analysis and visualization of the internal attention layers, this invention optimizes the efficiency of the feature extraction process, enabling the model to process and understand complex image content more efficiently. Finally, the prediction results of multiple improved models are integrated and fused, further enhancing the ability to understand and describe complex image content. Specifically, an attention-based deep learning framework is introduced into the three models, enabling them to more accurately focus on key regions of images and generate more accurate and natural text descriptions. Furthermore, LoRA (Local Rank Adaptation) fine-tuning technology is introduced into the self-attention layer, allowing for effective fine-tuning while keeping most parameters fixed. This allows the models to be trained efficiently on smaller datasets without retraining the entire network, reducing the computational burden. Through these improvements, this invention is expected to demonstrate superior performance in various application scenarios. Specifically, in image content interpretation, this invention can more accurately analyze and identify elements and scenes in images, making it applicable to image content search; in assisting visually impaired individuals, it can provide clearer and more targeted visual descriptions, reducing computational resource requirements in practical edge deployments while maintaining recognition efficiency and accuracy to improve their daily life experience; in the field of intelligent content creation, this invention can automatically generate creative and high-quality visual and textual content.

Claims

1. A method for interpretable image textual description based on multi-model fusion, characterized in that, The input image undergoes feature extraction, the extracted image features are then input into the prediction model, and finally the prediction result is obtained. The specific steps are as follows: Step 1: Initialize the preprocessed dataset; Step 2, Model Preparation; Step 3: Instantiate the predictor; Step 4: Integrate and fuse the prediction results; In step 1, the DiffusionDB_2M dataset is used, and datasets related to the image description task are selected first. Duplicate data with similarity scores higher than 0.8 are removed. Then, high-quality and diverse prompts are input into ChatGPT to generate images that are more in line with the specifications, so that the input prompts and the generated images correspond one-to-one. The input prompts are dynamically adjusted according to the results of model training, and finally a high-quality prompt dataset is formed. In step 2, the model architecture is defined, and a custom attention mechanism is added to the model. Two PyTorch modules, LoRALinearLayer and LoRALinear, are defined. LoRA modifies the model's weights by introducing a low-rank matrix. LoRA technology is applied to specific layers of the BLIP2 model. First, the total number of encoding layers in the model is calculated, and then the number of layers to use LoRA is determined based on the LoRA rate. Based on the number of layers, the last few layers in the encoder are traversed. For each layer, the current attention layer and the multilayer perceptron (MLP) layer are obtained, and the attention layer is replaced with a LoRA attention layer. For each fully connected layer of the MLP, a LoRA version is created using LoRALiner and the original layer is replaced. The forward propagation of the LoRACLIPAttention module applied to the CLIP model layer involves receiving the input hidden state, which is transformed into a corresponding vector through a query, key, and value projection layer. LoRA scaling is used to scale the query and key vectors, enhancing the model's expressive power. Next, a reshaping operation is performed to prepare tensors for attention mechanism computation. Causal attention masks and additional attention masks are applied to these vectors to ensure correctness and focus during sequence generation. Attention weights are calculated using the Softmax function, and LoRA scaling is applied again. To reduce overfitting, a Dropout mechanism is introduced. Finally, the calculated attention weights are combined with the value vector to generate the attention output, which is then passed through a linear projection layer to obtain the final module output.

2. The interpretable image textual description method based on multi-model fusion according to claim 1, characterized in that, In step 3, the predictor loads the trained model, including reading the model file stored in a specific path and loading it into memory. After the model is loaded, the predictor will also load the dataset to be predicted, including reading data from a database, file system or network source and preprocessing it to ensure that the data format is consistent with the training format. The data is then passed forward by the predictor through the model. During this process, the data passes through each layer of the model, which analyzes and processes the data based on the features and patterns it has learned. After the forward pass is completed, the model outputs the original prediction result.

Images