Image description method and device

Abstract

The invention discloses an image description method and device, and the method comprises the steps: obtaining an image and a text instruction; the image and the text instruction are input into an image description model for reasoning, text description information corresponding to the image is obtained, and the text description information corresponding to the image is obtained by comparing visual codes, output by a visual encoder in the image description model, of the image. According to the image description model and text description information output by the image through the visual encoder, the projection layer and the large language model in the image description model, the similarity between the visual encoder and the text description information is obtained, and parameters of the image description model are adjusted based on the similarity.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of artificial intelligence, and relate to, but are not limited to, an image description method and device. Background Art

[0002] Current multimodal large language models often experience hallucinations when describing image content. Content hallucination occurs when large language models often identify objects that don't exist in an image and then, based on the identified presence or absence of these objects, engage in a formal conversation with humans. Addressing content hallucinations has become a pressing technical challenge. Summary of the Invention

[0003] In view of this, embodiments of the present application provide an image description method and apparatus.

[0004] The technical solution of the embodiment of the present application is implemented as follows:

[0005] In a first aspect, an embodiment of the present application provides an image description method, comprising:

[0006] Get image and text instructions;

[0007] The image and text instructions are input into the image description model for inference to obtain the text description information corresponding to the image. The similarity between the visual encoding and the text description information is obtained by comparing the visual encoding output by the visual encoder in the image description model and the text description information output by the visual encoder, projection layer and large language model in the image description model, and the parameters of the image description model are adjusted based on the similarity.

[0008] In a second aspect, an embodiment of the present application provides an image description device, comprising:

[0009] Acquisition module, used to obtain image and text instructions;

[0010] The inference module is used to input the image and text instructions into the image description model for inference to obtain the text description information corresponding to the image. The similarity between the visual encoding and the text description information is obtained by comparing the visual encoding output by the visual encoder in the image description model and the text description information output by the visual encoder, projection layer and large language model in the image description model, and the parameters of the image description model are adjusted based on the similarity.

[0011] In a third aspect, an embodiment of the present application provides an electronic device comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and the processor obtains image and text instructions when executing the program; the image and text instructions are input into an image description model for inference to obtain text description information corresponding to the image, wherein the similarity between the visual encoding and the text description information is obtained by comparing the visual encoding output by the visual encoder in the image description model and the text description information output by the visual encoder, projection layer and large language model in the image description model, and the parameters of the image description model are adjusted based on the similarity.

[0012] In a fourth aspect, an embodiment of the present application provides a storage medium storing executable instructions for obtaining image and text instructions when executed by a processor; the image and text instructions are input into an image description model for inference to obtain text description information corresponding to the image, wherein the similarity between the visual encoding and the text description information is obtained by comparing the visual encoding output by the visual encoder in the image description model and the text description information output by the visual encoder, projection layer and large language model in the image description model, and the parameters of the image description model are adjusted based on the similarity.

[0013] In a fifth aspect, an embodiment of the present application provides a computer program product, comprising a computer program or instructions, which, when executed by a processor, realizes the acquisition of images and text instructions; the image and text instructions are input into an image description model for inference to obtain text description information corresponding to the image, wherein the similarity between the visual encoding and the text description information is obtained by comparing the visual encoding output by the visual encoder in the image description model and the text description information output by the visual encoder, projection layer and large language model in the image description model, and the parameters of the image description model are adjusted based on the similarity. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 A schematic diagram of an implementation flow of an image description method provided in an embodiment of the present application;

[0015] Figure 2 A schematic diagram of an implementation flow of a training image description model provided in an embodiment of the present application;

[0016] Figure 3 A schematic diagram of an implementation flow for adjusting image description model parameters using a target loss value provided in an embodiment of the present application;

[0017] Figure 4A A schematic diagram of a process for obtaining a multimodal large language model provided in an embodiment of the present application;

[0018] Figure 4BA diagram showing the overall structure of a network in the training phase provided in an embodiment of the present application;

[0019] Figure 4C A diagram showing the overall structure of a network in the training phase provided in an embodiment of the present application;

[0020] Figure 4D A diagram of the overall network structure in the use phase provided by an embodiment of the present application;

[0021] Figure 5 A schematic diagram of the structure of an image description device provided in an embodiment of the present application;

[0022] Figure 6 A schematic diagram of a hardware entity of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0023] To make the purpose, technical solutions and advantages of the embodiments of the present application more clear, the specific technical solutions of the embodiments of the present application will be further described in detail below in conjunction with the drawings in the embodiments of the present application. The following embodiments are used to illustrate the present application but are not intended to limit the scope of the present application.

[0024] In the following description, reference is made to “some embodiments”, which describes a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0025] In the following description, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0026] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application pertains. The terms used herein are for the purpose of describing the embodiments of this application only and are not intended to limit this application.

[0027] The present application embodiment provides an image description method, such as Figure 1 As shown, the method includes:

[0028] Step S110: obtaining image and text instructions;

[0029] Image captioning is a technology that combines computer vision and natural language processing, with the goal of generating a natural language description for a given image.

[0030] During implementation, the image to be described may be preprocessed into a format required by the image description model (eg, 224x224 resolution, normalized pixel values, etc.).

[0031] Convert the image to a tensor format (such as Tensor for PyTorch or tf.Tensor for TensorFlow).

[0032] The image description model supports multimodal input. For example, it can obtain text instructions (for example, "Describe the details of this image") and use a large language model to encode the text instructions into a text feature vector.

[0033] The acquired text instruction may be an instruction for prompting image processing, that is, a prompt word for processing the image. For example, an image may be acquired, and at the same time, a text instruction for processing the image may be acquired, which is "Describe this image with words."

[0034] Step S120: Inferring the image and the text instruction image description model to obtain text description information corresponding to the image, wherein the similarity between the visual encoding and the text description information is obtained by comparing the visual encoding output by the visual encoder in the image description model and the text description information output by the visual encoder, projection layer and large language model in the image description model, and adjusting the parameters of the image description model based on the similarity.

[0035] Here, the image description model includes a visual encoder, a projection layer, and a large language model.

[0036] During implementation, the image is passed through a visual encoder to extract features and output visual features. For example, a pre-trained visual model, such as the Contrastive Language-Image Pre-Training (CLIP) visual encoder, can be used to extract visual features.

[0037] Use a projection layer to map the visual features to the same dimension as the text features.

[0038] A Large Language Model (LLM) is used to generate a text description of the image based on visual features and text instructions. The LLM can be a model such as LlaMA2 or Vicuna.

[0039] Before using the image description model, the image description model is trained, including: first obtaining the visual encoding (visual features) output by the visual encoder in the image description model; the text description information output by the visual encoder, projection layer and large language model in the image description model; then comparing the visual encoding and the text description information to obtain the similarity between the visual encoding and the text description information; finally, adjusting the parameters of the image description model based on the similarity.

[0040] In some embodiments, during the process of the image description model performing the reasoning task, the similarity between the visual encoding and the text description information can be obtained by comparing the visual encoding output by the visual encoder of the input image and the text description information output by the visual encoder, the projection layer and the large language model, and the parameters of the image description model can be adjusted and optimized in real time using the similarity, so that the image description model's description of the image becomes more and more accurate and effective.

[0041] In an embodiment of the present application, the image description model obtains similarity during the training process by comparing the visual encoding output by the visual encoder in the image description model and the text description information output by the visual encoder, projection layer and large language model in the image description model, and then adjusts the parameters of the image description model based on the similarity. In this way, by minimizing the semantic gap between the visual encoding and the text description, the image description model can learn a more accurate image-text mapping relationship, thereby generating a description that is highly relevant to the image content. The similarity-based loss function can be directly used for end-to-end training, which simplifies the training process and improves efficiency. The trained image description model can significantly improve the accuracy, fluency and generalization ability of generating text description information.

[0042] In some embodiments, the present application provides a method for training an image description model, such as Figure 2 As shown, this can be achieved by following the steps below:

[0043] Step S210: Encode the first text description information using a text encoder to obtain a first text encoding, wherein the first text description information is obtained by inferring the first training image using the image description model;

[0044] Here, the text encoder and the image encoder in the image inference model correspond to the same Comparative Language Image Pretraining (CLIP) model. The CLIP model uses contrastive learning to map images and text into a shared vector space, enabling cross-modal retrieval and classification. Its core structure consists of two parts: an image encoder (ImageEncoder), which converts input images into a high-dimensional vector representation; and a text encoder (TextEncoder), which converts input text into a similar high-dimensional vector representation.

[0045] The text encoder and the image encoder in the image inference model correspond to the same CLIP model, and the text and visual alignment effect can be ensured through contrastive learning.

[0046] The first text description information is obtained by inferring the first training image by the image description model, that is, the first training image in the image set used for training is input into the image description model, and after processing by the visual encoder, projection layer and large language model, the first text description information used to describe the first training image is obtained.

[0047] During implementation, the first text description information may be encoded using a text encoder to obtain a first text encoding corresponding to the first text description information, that is, the first text description information is converted into a high-dimensional vector representation.

[0048] Step S220: Encode the first training image using the visual encoder to obtain a first image code, wherein the text encoder and the image encoder correspond to the same comparative language image pre-training model;

[0049] During the implementation process, the first training image is processed using the visual encoding in the image description model to obtain a first image encoding corresponding to the first training image, that is, the first training image is converted into a high-dimensional vector representation.

[0050] Here, the text encoder and the image encoder in the image inference model correspond to the same CLIP, and the text and visual alignment effect can be ensured through contrastive learning.

[0051] There is no restriction on the order of execution of step S210 and step S220, and they may be executed simultaneously.

[0052] Step S230: Adjust parameters of the image description model based on the similarity between the first image code and the first text code.

[0053] Here, since the text encoder and the image encoder in the image inference model correspond to the same CLIP model, the high-dimensional vector representation converted based on the first text description information and the high-dimensional vector representation converted based on the first training image are similar high-dimensional vector representations. By comparing the two high-dimensional vectors, the similarity between the first image encoding and the first text encoding can be obtained.

[0054] During implementation, by adjusting the parameters of the image description model so that the similarity between the first image code and the first text code reaches an optimization target, the image and text matching performance can be improved.

[0055] In the embodiment of the present application, a text encoder is first used to encode the first text description information to obtain a first text code, and a visual encoder is used to encode the first training image to obtain a first image code. Based on the similarity between the first image code and the first text code, the parameters of the image description model are adjusted. Thus, by adjusting the parameters of the image description model to optimize the similarity between the first image code and the first text code, the image-text matching performance can be improved, thereby further enhancing the effectiveness and accuracy of the image description model in generating image descriptions.

[0056] In some embodiments, the above step S210 "encoding the first text description information using a text encoder to obtain a first text encoding" can be implemented by the following steps:

[0057] Step 211: extract key words from the first text description information;

[0058] During the implementation process, a key concept extraction module can be set up to extract key objects from the first text description information output by the image description model. For example, the key concept extraction module can be combined with natural language processing (NLP) technology, such as named entity recognition (NER), keyword extraction or semantic analysis, to identify the core objects described in the text. The first text description information generated by the image description model (such as "a red car parked next to the tree"). The key concept output module can output: the key objects or concepts extracted from the first text description information (such as ["car" "tree"]). The key concept extraction module can accurately and efficiently extract the entities most relevant to the first text description information.

[0059] Step 212: Generate a target sentence based on the key words;

[0060] During implementation, a prompt word expression module can be set up to organize the extracted keywords into complete sentences. For example, natural language generation (NLG) technology can be used to convert discrete keywords into natural language descriptions. A list of keywords extracted from the image description (such as ["car", "tree", "bird"]). Output: A complete sentence organized based on keywords (such as "A red car is parked next to the tree, and a small bird is parked on it."). The prompt word expression module can generate sentences that are grammatically correct, semantically coherent, and relevant to the content of the image (the first text description information).

[0061] Step 213: Encode the target sentence using the text encoder to obtain the first text encoding.

[0062] During implementation, the text encoder can encode the target sentence into a specific machine language, encoding the first text represented by a high-dimensional vector. It encodes semantic information into a machine-processable form while preserving semantic integrity and distinctiveness, achieving a natural language to machine language mapping.

[0063] In this embodiment, key words are first extracted from the first text description information; a target sentence is then generated based on the key words; and finally, a text encoder is used to encode the target sentence to obtain a first text encoding. This allows the first text description information to be converted into a first text encoding represented by a high-dimensional vector, preserving semantic integrity and distinctiveness, allowing for further comparison with a first image encoding obtained based on the same first training image.

[0064] In some embodiments, the above step S230 "adjusting the parameters of the image description model based on the similarity between the first image encoding and the first text encoding" can be implemented by the following steps:

[0065] Step 231: determining a target loss value based on the first image code and the first text code;

[0066] Here, the target loss value is used to measure the semantic consistency between image and text encoding, thereby guiding the parameter optimization of the image description model.

[0067] The first image encoding may be a high-dimensional vector representation of a fixed dimension (such as a 2048-dimensional ResNet feature), and the first text encoding may be a high-dimensional vector representation of the same fixed dimension as the first image encoding.

[0068] During implementation, a suitable loss function (such as contrast loss, triple loss, etc.) may be selected according to actual needs. A target loss value between the first image encoding and the first text encoding is calculated.

[0069] Step 232: Use the target loss value to adjust the parameters of the image description model.

[0070] In some embodiments, a back-propagation algorithm may be used to update the parameters of the image description model to minimize the target loss value.

[0071] In some embodiments, other optimization techniques (such as learning rate scheduling, weight decay, etc.) can also be combined to improve the training effect.

[0072] In an embodiment of the present application, a target loss value is determined based on the first image encoding and the first text encoding; the target loss value is then used to adjust the parameters of the image description model. In this way, the target loss value can be used to guide the update of the image description model parameters, so that the semantic consistency between the image and text encodings is improved. During the training process, the convergence of the model can be evaluated by monitoring the changes in the target loss value. Select a suitable loss function according to the specific task requirements to ensure that the model can effectively solve practical problems. By selecting a suitable loss function, the semantic consistency between the image and text encodings can be effectively measured and the model optimization can be guided.

[0073] In some embodiments, the above step 231 “determining a target loss value based on the first image encoding and the first text encoding” may be implemented by the following steps:

[0074] Step 2311: Determine a first similarity loss value based on the first image code and the first text code;

[0075] Here, the semantic consistency between the first training image and the first text description information may be measured based on the first image encoding and the first text encoding.

[0076] During implementation, a similarity metric (such as cosine similarity) may be used to calculate the similarity between the first image encoding and the first text encoding, and a loss value (such as contrast loss, InfoNCE loss, etc.) may be calculated based on the similarity.

[0077] Step 2312: Determine a second similarity loss value based on the first text code and a preset text;

[0078] Here, the first text encoding and the preset text encoding can measure the proximity of the generated text to the preset text (the identification text corresponding to the first training image), that is, evaluate the accuracy of the text generation. The similarity between the first text encoding and the preset text is calculated based on a similarity metric (such as cosine similarity or Euclidean distance). A loss value (such as mean square error loss, cross entropy loss, etc.) is calculated based on the similarity.

[0079] For example, the original golden loss function can be used to supervise text generation or align text encoding with preset text. The original golden loss function can refer to the supervision loss based on the target text (ground truth), such as cross entropy loss or mean squared error.

[0080] During the implementation process, the execution order of step 2311 and step 2312 is not limited, and they can also be executed simultaneously.

[0081] Step 2313: Perform a weighted average of the first similarity loss value and the second similarity loss value to obtain the target loss value.

[0082] During the implementation process, the two loss values (the first similarity loss value and the second similarity loss value) can be fused through the weighted average method to balance cross-modal alignment and supervised learning.

[0083] In some embodiments, the weighting coefficient corresponding to the first similarity loss value or the second similarity loss value can be dynamically adjusted according to the training stage. For example, in the early stage of training, more attention is paid to cross-modal alignment, and the weighting coefficient corresponding to the first similarity loss value can be increased; in the later stage, more attention is paid to supervised learning, and the weighting coefficient corresponding to the second similarity loss value can be increased.

[0084] In an embodiment of the present application, a first similarity loss value is determined based on a first image code and a first text code; a second similarity loss value is determined based on the first text code and a preset text; and finally, a weighted average is performed on the first similarity loss value and the second similarity loss value to obtain a target loss value. In this way, the image description model is adjusted by fusing the two losses to obtain a target loss value, which can improve the cross-modal alignment capability of the image description model and improve the semantic consistency between the image and the text. This improves the performance of supervised learning and enhances the accuracy of text generation or classification of the image description model. A balance is found between cross-modal alignment and supervised learning to improve the generalization capability of the image description model.

[0085] The present application embodiment provides a method for adjusting image description model parameters using target loss value, such as Figure 3 As shown, this can be achieved by following the steps below:

[0086] Step S310: Using the target loss value, based on the first data set, adjust the parameters of the visual encoder, projection layer, and large language model in the image description model to obtain an image description model that has completed the first stage of parameter adjustment;

[0087] Training the image description model can be divided into two phases. The first phase involves updating the parameters of the visual encoder, projection layer, and large language model based on a first dataset (e.g., general image-text pairs), achieving cross-modal semantic alignment and enabling the image description model to initially generate image descriptions. The first dataset is large (millions of samples) and covers a wide range of scenarios; strict labeling quality is not required; the focus is on diverse modal alignment.

[0088] During the first stage of training, the parameter adjustment strategy includes: adjusting the parameters of the visual encoder to extract image features and optimize the visual semantic representation of the target domain; adjusting the parameters of the projection layer to map the visual features to a shared semantic space and align them with the text features; and adjusting the parameters of the large language model to learn text generation capabilities and optimize the accuracy of the description of image semantics.

[0089] Step S320: Using the target loss value, while fixing the parameters of the visual encoder and the large language model, adjust the parameters of the projection layer based on the second data set to obtain an image description model that completes the second stage parameter adjustment;

[0090] The quality of the second data set is higher than the data quality of the second data set.

[0091] In the second phase of training, based on a second dataset (e.g., high-quality image-text pairs from a specific domain), the projection layer's parameters are adjusted to improve the semantic mapping accuracy of the target domain. This second dataset is smaller (consisting of approximately 100,000 samples) but has extremely high-quality annotations. It is selected based on the domain of interest and has strong domain relevance (e.g., medical images, product descriptions, etc.).

[0092] During the first stage of training, the parameter adjustment strategies include: fixing the visual encoder and freezing the parameters to avoid overfitting to a specific domain; fixing the large language model and freezing the parameters to maintain general text generation capabilities; and fine-tuning the projection layer, optimizing only the parameters of the projection layer to make it more suitable for the visual-text semantic mapping of the target domain.

[0093] In an embodiment of the present application, the first stage of training updates the parameters of the visual encoder, projection layer, and large language model based on the first data set; in the second stage of training, the visual encoder and large language model are fixed based on the second data set, and the parameters of the projection layer are adjusted to improve the semantic mapping accuracy of the projection layer in the target domain. In this way, in the first training stage, the image features are mapped to the same semantic space as the text features through the visual encoder and projection layer to ensure cross-modal semantic consistency. In the second training stage, by freezing the parameters of the visual encoder and the large language model, the overfitting problem caused by the small size of the data set is avoided. Furthermore, the high-quality data set of the second stage enables the projection layer to learn the specific semantic mapping of the target domain, which can improve the performance of the image description model on specific tasks.

[0094] In some embodiments, the parameters of the large language model include parameters of a low-rank adapter; and the above step 310 "using the target loss value to adjust the parameters of the visual encoder, projection layer, and large language model in the image description model based on the first dataset to obtain an image description model in which the parameters are adjusted in the first stage" can be implemented by the following steps:

[0095] Utilizing the target loss value, the parameters of the visual encoder, the projection layer, and the low-rank adapter in the image description model are adjusted based on the first data set to obtain the image description model that completes the first-stage parameter adjustment.

[0096] Here, a Low-Rank Adaptation of Large Language Models (LoRA) is added to the large language model in multimodal learning. LoRA is a parameter-efficient fine-tuning (PEFT) technology that aims to solve the fine-tuning problem of large language models (LLMs) on specific tasks. It reduces the number of trainable parameters, reduces computing and storage costs, and can maintain or improve model performance.

[0097] During the implementation process, since the large language model in the multimodal environment adds low-rank adaptation of the large language model, in the first training stage, the image description model is given general multimodal capabilities by updating the visual encoder, projection layer and low-rank adaptation of the large language model.

[0098] In the embodiment of the present application, a low-rank adapter is added to the large language model in the multimodal environment. The purpose is to make the image description model more flexible to adapt to new tasks by adjusting a small number of parameters. In the first training phase, the low-rank adapter of the visual encoder, projection layer, and large language model is updated to give the image description model a general multimodal capability. And because the low-rank adaptation of the large language model is updated, the low-rank decomposition technology significantly reduces the number of parameters and computing resources required for fine-tuning while maintaining model performance, that is, reduces the number of trainable parameters, reduces computing and storage costs, and can maintain or improve model performance.

[0099] In some embodiments, the present invention provides a method for initializing an image description model, which can be implemented by the following steps:

[0100] Step S140: Before training the image description model, load a multimodal model to initialize the image description model.

[0101] Multimodal models (such as CLIP, ViLT, and ALBEF) have been trained on large-scale multimodal data and have learned the complex relationships between images and text, thus possessing powerful visual and language understanding capabilities. By loading these pre-trained multimodal models, the image description model can:

[0102] Inheriting visual feature extraction capabilities: The visual encoder in the multimodal model is able to extract high-level semantic features of the image, which are crucial for image description tasks.

[0103] Leveraging cross-modal alignment knowledge: Multimodal models learn the alignment relationship between image regions and text words through contrastive learning or cross-modal attention mechanisms, which helps image description models generate more accurate descriptions.

[0104] Accelerated convergence: Pre-trained weights provide a good initialization for the model, allowing the training process to converge to the optimal solution faster and reducing dependence on large-scale labeled data.

[0105] During implementation, you can choose a pre-trained multimodal model based on the task requirements and data size. For example, for general image description tasks, you can choose CLIP or ALBEF; for specific domains (such as medical images), you can choose a domain-specific multimodal model.

[0106] In the embodiment of the present application, before training the image description model, the multimodal model is loaded to initialize the image description model. In this way, loading the multimodal model to initialize the image description model before training the image description model is an effective strategy that can utilize the rich knowledge accumulated by the multimodal model in cross-modal tasks to improve the performance of the image description model.

[0107] In some embodiments, the step S140 of "loading the multimodal model to initialize the image description model" may be implemented by the following steps:

[0108] Step 141: Migrating the weight parameters of the visual encoder in the multimodal model to the visual encoder in the image description model;

[0109] The visual encoders in multimodal models (such as CLIP, ALBEF, and ViLT) are trained on large-scale image-text data to extract high-level semantic features of images. Transferring these weight parameters to the visual encoder of the image captioning model can reduce the need for large-scale annotated image data and accelerate model convergence.

[0110] Step 142: Migrate the weight parameters of the text encoder in the multimodal model to the text encoder.

[0111] Here, the text encoder in the multimodal model (such as the Transformer encoder) learns rich language representations and is capable of processing text sequences. Transferring these weight parameters to the text encoder in the image captioning model can improve the model's understanding of text generation tasks.

[0112] The weight parameters of the visual encoder in the same multimodal model are transferred to the visual encoder in the image captioning model, and the weight parameters of the text encoder are transferred to the text encoder. This ensures that the outputs of the visual encoder and text encoder have a significant correlation in terms of semantic space consistency and feature distribution alignment. For example, the image features of "cat" extracted by the visual encoder and the text features of "cat" generated by the text encoder are close in semantic space, facilitating subsequent fusion. In image captioning, the distribution alignment of visual and text features can improve the accuracy and fluency of the generated captions.

[0113] In an embodiment of the present application, the weight parameters of the visual encoder in the multimodal model are migrated to the visual encoder in the image description model; the weight parameters of the text encoder in the multimodal model are migrated to the text encoder. In this way, the multimodal model usually maps the visual encoder and the text encoder to the same semantic space through contrastive learning (such as CLIP) or cross-modal alignment training. After migration, the visual encoder and the text encoder of the image description model still share this semantic space, ensuring that the visual features (such as objects and scenes in the image) and the text features (such as descriptive vocabulary and semantic concepts) are consistent at the semantic level. The migrated encoder retains the feature distribution learned in the multimodal model, so that the distribution of visual features and text features in the vector space is similar. This alignment helps to reduce the semantic gap between modalities and improve the performance of cross-modal tasks. During multimodal model training, the visual encoder and the text encoder learn the correlation between modalities through attention mechanism or cross-modal interaction. After migration, this correlation is preserved, allowing visual features to better guide text generation, or text features to more accurately interpret visual content, effectively improving the training effect of the image description model, that is, improving the alignment effect of text features and visual features during training, and improving training accuracy and efficiency.

[0114] Current large multimodal language models often experience hallucinations when describing image content. Training large multimodal language models requires significant data construction and training overhead. Dedicated models are less versatile and difficult to generalize to other multimodal models. Furthermore, the need to introduce data from other modalities (knowledge graphs) complicates large multimodal language models. Relying on specialized models for various tasks requires large language models to be constructed to avoid hallucinations, and manual correction of raw output results is required, making the inference process cumbersome.

[0115] Based on the above technical problems, the embodiments of the present application provide an automated general solution to content hallucination. During the model training process, a hallucination correction module is introduced to guide network training, while the hallucination correction module can be removed during the inference process.

[0116] Figure 4A A flowchart of a method for obtaining a multimodal large language model is provided in an embodiment of the present application, such as Figure 4A As shown, this can be achieved by following the steps below:

[0117] Step S410: Building a network structure;

[0118] Figure 4B This is a diagram of the overall network structure in the training phase provided by the embodiment of the present application, such as Figure 4B As shown, the overall structure diagram of the network includes a multimodal large language model 41 and a content hallucination correction module 42, wherein:

[0119] The multimodal large language model 41 includes a visual encoding network (Image Encoder) 411, a projection module (projection layer) 412, and a pre-trained large language model (LLM) 413. The multimodal large language model 413 incorporates a low-rank adaptation of large language models (LoRA) to adapt the model more flexibly to new tasks by adjusting a small number of parameters. The pre-trained large language model 413 can be a model such as LlaMA2 or Vicuna.

[0120] A content hallucination correction module 42 is introduced to supervise the multimodal structure during training and reduce its hallucination phenomenon. The content hallucination correction module 42 includes a key concept extraction module 421, a prompt word representation module 422, a text encoder (TextEncoder) 423, and a similarity calculation module 424. Among them, the key concept extraction module 421 is used to extract key objects from the prediction results of the large language model; the prompt word representation module 422 is used to organize the extracted keywords into complete sentences; the text encoder 423 is used to encode the prompt word sentence into a specific machine language; and finally, the similarity calculation module 424 is used to calculate the similarity loss between the visual encoding result and the text encoding result.

[0121] During the implementation process, the key concept (content) extraction module (extract key concepts 421): GPT-3.5 can be used to extract key concepts, such as key nouns, from the output of MLLM.

[0122] Prompt word management module (prompt word expression 422): synthesize text sentences based on extracted keywords

[0123] Text encoding module 423: responsible for encoding the synthesized text sentences, which are then used to calculate the similarity loss with the output of the visual encoding network (image encoder) 411.

[0124] here, Figure 4B The visual encoding network 411 and text encoder 423 used in the present invention are derived from the same set of contrastive language-image pre-training (CLIP) through contrastive learning to ensure the alignment of text and vision.

[0125] Step S420: Load the multimodal pre-trained model and initialize the multimodal pre-trained model;

[0126] Before the model officially starts training, you can load the multimodal model to initialize the network. That is, you can use the visual encoding and text encoding in CLIP to initialize the network. Figure 4B The visual encoding network 411 and text encoder 423 are initialized. The pre-trained LLM model 413 and the models in the content hallucination correction module 42 are frozen throughout the training process. This is because these two components have very large parameters (usually over a billion) and can be directly inherited from the pre-trained model without affecting the final effect, so they are frozen.

[0127] Step S430: training the multimodal pre-training model in two stages using a loss function;

[0128] Here, a two-stage model training method is provided. During the two-stage training process, the loss function adopts the weighted average of the original golden loss function and the similarity loss function.

[0129] During the first stage of training, the weights of the visual coding network 411, the projection module 412, and the LoRA are updated (e.g. Figure 4B Add the texture filling module as a module that needs to update parameters).

[0130] During the second stage of training, only the weights of the projection module 412 are updated, and the parameters of the rest of the network remain frozen ( Figure 4C Add the texture filling module as a module that needs to update parameters).

[0131] Step S440: Obtain a multimodal large language model with content hallucination correction function.

[0132] After the model training is completed, remove the hallucination correction module, such as Figure 4D As shown, the visual encoding network 411, the projection module 412, the LLM and the LoRA module 412 are retained for multimodal large language model reasoning.

[0133] In the embodiment of the present application, an illusion correction module is introduced during model training to guide network training, while the illusion correction module can be removed during inference. A two-stage model training method is provided: in the first stage, the image encoding module, projection layer, and large language model LoRA are updated to give the model universal multimodal capabilities; in the second stage, the projection layer parameters are updated to address model accuracy issues. This achieves end-to-end integrated training and inference, without introducing additional computing power during inference and without generating additional training data.

[0134] Based on the foregoing embodiments, an embodiment of the present application provides an image description device, which includes modules, each module includes sub-modules, each sub-module includes a unit, and can be implemented by a processor in an electronic device; of course, it can also be implemented by a specific logic circuit; in the implementation process, the processor can be a central processing unit (CPU), a microprocessor (MPU), a digital signal processor (DSP) or a field programmable gate array (FPGA), etc.

[0135] Figure 5 A schematic diagram of the structure of the image description device provided in the embodiment of the present application is shown as follows: Figure 5 As shown, the apparatus 500 includes:

[0136] An acquisition module 510 is used to acquire images and text instructions;

[0137] The inference module 520 is used to input the image and the text instruction into the image description model for inference to obtain text description information corresponding to the image, wherein the similarity between the visual encoding and the text description information is obtained by comparing the visual encoding output by the visual encoder in the image description model and the text description information output by the visual encoder, projection layer and large language model in the image description model, and the parameters of the image description model are adjusted based on the similarity.

[0138] In some embodiments, the image description device further includes a first encoding module, a second encoding module and an adjustment module, wherein the first encoding module is used to encode first text description information using a text encoder to obtain a first text encoding during the process of training the image description model, wherein the first text description information is obtained by inferring the first training image by the image description model; the second encoding module is used to encode the first training image using the visual encoder to obtain a first image encoding, wherein the text encoder and the image encoder correspond to the same comparative language image pre-training model; and the adjustment module is used to adjust the parameters of the image description model based on the similarity between the first image encoding and the first text encoding.

[0139] In some embodiments, the first encoding module includes an extraction submodule, a generation submodule and an encoding submodule, wherein the extraction submodule is used to extract key words in the first text description information; the generation submodule is used to generate a target sentence based on the key words; and the encoding submodule is used to encode the target sentence using the text encoder to obtain the first text encoding.

[0140] In some embodiments, the adjustment module includes a determination submodule and an adjustment submodule, wherein the determination submodule is used to determine a target loss value based on the first image encoding and the first text encoding; and the adjustment submodule is used to adjust the parameters of the image description model using the target loss value.

[0141] In some embodiments, the determination submodule includes a first determination unit, a second determination unit and a weighted averaging unit, wherein the first determination unit is used to determine a first similarity loss value based on the first image encoding and the first text encoding; the second determination unit is used to determine a second similarity loss value based on the first text encoding and a preset text; and the weighted averaging unit is used to perform a weighted average of the first similarity loss value and the second similarity loss value to obtain the target loss value.

[0142] In some embodiments, the adjustment submodule includes a first adjustment unit and a second adjustment unit, wherein the first adjustment unit is used to use the target loss value to adjust the parameters of the visual encoder, projection layer and large language model in the image description model based on the first data set to obtain an image description model that completes the first stage parameter adjustment; the second adjustment unit is used to use the target loss value to fix the parameters of the visual encoder and the large language model, and adjust the parameters of the projection layer based on the second data set to obtain an image description model that completes the second stage parameter adjustment; wherein the quality of the second data set is higher than the data quality of the second data set.

[0143] In some embodiments, the parameters of the large language model include parameters of the low-rank adapter; the first adjustment unit is also used to use the target loss value to adjust the parameters of the visual encoder, the projection layer and the low-rank adapter in the image description model based on the first data set to obtain the image description model that completes the first stage parameter adjustment.

[0144] In some embodiments, the image description device further includes an initialization module for loading a multimodal model to initialize the image description model before training the image description model.

[0145] In some embodiments, the initialization module includes a first migration submodule and a second migration submodule, wherein the first migration submodule is used to migrate the weight parameters of the visual encoder in the multimodal model to the visual encoder in the image description model; the second migration submodule is used to migrate the weight parameters of the text encoder in the multimodal model to the text encoder.

[0146] The description of the above device embodiment is similar to the description of the above method embodiment and has similar beneficial effects as the method embodiment. For technical details not disclosed in the device embodiment of this application, please refer to the description of the method embodiment of this application for understanding.

[0147] It should be noted that, in the embodiment of the present application, if the above method is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable an electronic device (which can be a mobile phone, tablet computer, laptop computer, desktop computer, etc.) to execute all or part of the method described in each embodiment of the present application. The aforementioned storage medium includes various media that can store program code, such as a U disk, a mobile hard disk, a read-only memory (ROM), a magnetic disk or an optical disk. In this way, the embodiment of the present application is not limited to any specific combination of hardware and software.

[0148] Correspondingly, an embodiment of the present application provides a storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the image description method provided in the above embodiment are implemented.

[0149] Correspondingly, an embodiment of the present application provides an electronic device, Figure 6 A hardware entity diagram of an electronic device provided in an embodiment of the present application, such as Figure 6 As shown, the hardware entity of the device 600 includes: a memory 601 and a processor 602, the memory 601 stores a computer program that can be run on the processor 602, and the processor 602 implements the steps of the image description method provided in the above embodiment when executing the program.

[0150] The memory 601 is configured to store instructions and applications executable by the processor 602, and can also cache data to be processed or processed by the processor 602 and various modules in the electronic device 600 (for example, image data, audio data, voice communication data and video communication data), which can be implemented through flash memory (FLASH) or random access memory (RAM).

[0151] It should be noted that the description of the above storage medium and device embodiments is similar to the description of the above method embodiments and has similar beneficial effects as the method embodiments. For technical details not disclosed in the storage medium and device embodiments of this application, please refer to the description of the method embodiments of this application for understanding.

[0152] It should be understood that "one embodiment" or "an embodiment" mentioned throughout the specification means that the specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It should be understood that in the various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application. The above-mentioned serial numbers of the embodiments of the present application are for description only and do not represent the advantages and disadvantages of the embodiments.

[0153] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or apparatus comprising the element.

[0154] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0155] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units; they may be located in one place or distributed across multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the scheme of this embodiment.

[0156] In addition, all functional units in the embodiments of the present application can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the above-mentioned integrated units can be implemented in the form of hardware or in the form of hardware plus software functional units.

[0157] Those skilled in the art will understand that all or part of the steps of implementing the above-mentioned method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above-mentioned method embodiment; and the aforementioned storage medium includes: mobile storage devices, read-only memories (ROM), magnetic disks or optical disks, and other media that can store program codes.

[0158] Alternatively, if the above-mentioned integrated unit of the present application is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application can essentially or in other words be embodied in the form of a software product that contributes to the relevant technology. The computer software product is stored in a storage medium and includes several instructions for enabling an electronic device (which can be a mobile phone, tablet computer, laptop computer, desktop computer, etc.) to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0159] The methods disclosed in the several method embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments.

[0160] The features disclosed in the several product embodiments provided in this application can be arbitrarily combined without conflict to obtain new product embodiments.

[0161] The features disclosed in the several method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments or device embodiments.

[0162] The above is merely an embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. A method for describing an image, comprising: Get image and text instructions; The image and the text instruction are input into an image description model for reasoning to obtain text description information corresponding to the image, wherein the similarity between the visual encoding and the text description information is obtained by comparing the visual encoding output by the visual encoder in the image description model and the text description information output by the visual encoder, projection layer and large language model in the image description model, and the parameters of the image description model are adjusted based on the similarity.

2. The method according to claim 1, further comprising: In the process of training the image description model, encoding first text description information using a text encoder to obtain a first text encoding, wherein the first text description information is obtained by the image description model by inferring the first training image; encoding the first training image using the visual encoder to obtain a first image encoding, wherein the text encoder and the image encoder correspond to the same comparative language image pre-training model; Based on the similarity between the first image encoding and the first text encoding, parameters of the image description model are adjusted.

3. The method according to claim 2, wherein encoding the first text description information using a text encoder to obtain the first text encoding comprises: Extracting key words from the first text description information; generating a target sentence based on the key words; The target sentence is encoded using the text encoder to obtain the first text encoding.

4. The method according to claim 2, wherein adjusting parameters of the image description model based on the similarity between the first image encoding and the first text encoding comprises: determining a target loss value based on the first image encoding and the first text encoding; Parameters of the image description model are adjusted using the target loss value.

5. The method of claim 4, wherein determining the target loss value based on the first image encoding and the first text encoding comprises: determining a first similarity loss value based on the first image encoding and the first text encoding; Determining a second similarity loss value based on the first text encoding and a preset text; A weighted average is performed on the first similarity loss value and the second similarity loss value to obtain the target loss value.

6. The method of claim 4, wherein adjusting parameters of the image description model using the target loss value comprises: Using the target loss value, based on the first data set, adjust the parameters of the visual encoder, the projection layer, and the large language model in the image description model to obtain an image description model that completes the first stage of parameter adjustment; Using the target loss value, while fixing the parameters of the visual encoder and the large language model, adjusting the parameters of the projection layer based on the second data set to obtain an image description model that completes the second stage parameter adjustment; The quality of the second data set is higher than the data quality of the second data set.

7. The method of claim 6, wherein the parameters of the large language model include parameters of a low-rank adapter; The method of adjusting the parameters of the visual encoder, the projection layer, and the large language model in the image description model based on the first data set by using the target loss value to obtain the image description model in which the parameters of the first stage are adjusted includes: Utilizing the target loss value, the parameters of the visual encoder, the projection layer, and the low-rank adapter in the image description model are adjusted based on the first data set to obtain the image description model that completes the first-stage parameter adjustment.

8. The method according to any one of claims 2 to 7, further comprising: Before training the image description model, a multimodal model is loaded to initialize the image description model.

9. The method according to claim 8, wherein loading the multimodal model to initialize the image description model comprises: Migrating weight parameters of the visual encoder in the multimodal model to the visual encoder in the image description model; The weight parameters of the text encoder in the multimodal model are transferred to the text encoder.

10. An image description device, comprising: Acquisition module, used to obtain image and text instructions; An inference module is used to input the image and the text instruction into an image description model for inference to obtain text description information corresponding to the image, wherein the similarity between the visual encoding and the text description information is obtained by comparing the visual encoding output by the visual encoder in the image description model and the text description information output by the visual encoder, projection layer and large language model in the image description model, and the parameters of the image description model are adjusted based on the similarity.

Images