Fine-grained image caption method and system based on instruction fine-tuning and multi-modal large model

Abstract

The present application relates to the technical field of computers. Provided are a fine-grained image caption method and system based on instruction fine-tuning and a multi-modal large model. The method comprises: acquiring a first vector sequence corresponding to a target image, wherein the first vector sequence corresponding to the target image is obtained by means of encoding a second vector sequence corresponding to the target image, the second vector sequence corresponding to the target image being obtained on the basis of advanced semantic information extracted from the target image; and acquiring fine-grained caption text of the target image on the basis of the first vector sequence and a first prompt template, wherein the first prompt template is used for providing instruction information required for performing fine-grained caption on the target image. By means of the present application, the attributes and features of important targets in an image can be accurately identified and captioned on the basis of little instruction information required for performing fine-grained caption on the image, and a vector sequence carrying advanced semantic information of the image, so as to realize fine-grained caption of the image, thus providing richer and more specific image information.

Description

Image fine-grained description method and system for fine-tuning multimodal large models

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to Chinese patent application No. 2023112732419, filed on September 28, 2023, entitled “Image fine-grained description method and system for instruction fine-tuning multimodal large models”, which is incorporated herein by reference in its entirety. Technical Field

[0003] The present application relates to the field of computer technology, and in particular to a method and system for fine-grained image description by fine-tuning a multimodal large model. Background Art

[0004] In the field of artificial intelligence, multimodal large models and instruction fine-tuning are two major research directions. In the past two years, research on multimodal large models has seen explosive growth, and instruction fine-tuning has also demonstrated its effectiveness in practical applications in downstream tasks in the field of natural language processing.

[0005] The goal of a large multimodal model is to enable the model to understand and process different types of data, such as text, images, and sounds. During the pre-training phase, the large multimodal model is trained with full supervision or self-supervision on a large-scale multimodal dataset to learn the underlying patterns of the multimodal data and perform preliminary alignment of the multimodal semantic information. However, the multimodal data in the pre-training phase usually contains a lot of noise, which causes the large multimodal model to be less robust in downstream tasks. This is usually manifested as an incomplete understanding of multimodal information in multimodal understanding tasks, leading to incorrect classification and prediction; or the unimodal representation generated by the multimodal generation task is completely inconsistent with human thinking logic and perception. Therefore, in order to adapt the large multimodal model to downstream tasks, it is necessary to fine-tune the large multimodal model.

[0006] Instruction fine-tuning is a new approach to fine-tuning large models that allows them to adapt to downstream tasks by learning a simple set of instructions, without the need for complex fine-tuning. The main advantage of this approach is that it reduces training time and enables large models to adapt to new downstream tasks more quickly. Instruction fine-tuning is achieved by including the task description as part of the input to the large model. This approach allows large models to learn new tasks without the need for additional labeled data, significantly reducing the amount of data required for large model training.

[0007] Although large multimodal models and instruction fine-tuning techniques have made significant progress in the field of artificial intelligence, they still face many challenges. For example, for large multimodal models, how to effectively integrate information from different modalities and how to design effective pre-training strategies remain issues that require further research. Regarding instruction fine-tuning, how to design effective instructions and how to handle the uncertainty of large models also require further research.

[0008] On the other hand, in the multimodal field, image description tasks have become a hot research topic. Among them, fine-grained image captioning has attracted widespread attention for its in-depth understanding and precise description of image content. It not only requires identifying important objects in an image, but also describes their specific attributes and features, such as shape, color, and texture. This image description method can provide richer and more specific image information, which is of great value for many practical application scenarios. However, the fine-grained image description task also faces many challenges, such as how to accurately identify and describe the attributes and features of important objects, and how to deal with the complexity and diversity of information in images.

[0009] Summary of the Invention

[0010] The method and system for fine-grained image description of a multimodal large model using instruction fine-tuning provided in this application are used to solve the problem existing in the prior art of how to accurately identify and describe the attributes and features of important targets in an image and achieve fine-grained description of the image.

[0011] This application provides a method for fine-grained image description for fine-tuning a multimodal large model, including:

[0012] Obtaining a first vector sequence corresponding to a target image, wherein the first vector sequence corresponding to the target image is obtained by encoding a second vector sequence corresponding to the target image, and the second vector sequence corresponding to the target image is obtained based on high-level semantic information extracted from the target image;

[0013] A fine-grained description text of the target image is obtained according to the first vector sequence and a first prompt template, where the first prompt template is used to provide instruction information required for a fine-grained description of the target image.

[0014] This application also provides a fine-grained image description system for fine-tuning a multimodal large model, including:

[0015] a data acquisition module, configured to acquire a first vector sequence corresponding to a target image, wherein the first vector sequence corresponding to the target image is obtained by encoding a second vector sequence corresponding to the target image, wherein the second vector sequence corresponding to the target image is obtained based on high-level semantic information extracted from the target image;

[0016] An image description module is used to obtain a fine-grained description text of the target image according to the first vector sequence and a first prompt template, wherein the first prompt template is used to provide instruction information required for a fine-grained description of the target image.

[0017] The present application also provides an electronic device, comprising a processor and a memory storing a computer program, wherein when the processor executes the program, it implements a method for fine-tuning a multimodal large model's fine-grained image description according to any one of the instructions described above.

[0018] The present application also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a method for fine-tuning a multimodal large model's fine-grained image description by any of the instructions described above.

[0019] The present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the image fine-grained description method of fine-tuning a multimodal large model as described in any of the above instructions.

[0020] The image fine-grained description method and system for instruction fine-tuning multimodal large models provided in this application can accurately identify and describe the attributes and features of important targets in the image based on a small amount of instruction information required for fine-grained description of the image and a vector sequence carrying high-level semantic information of the image, thereby achieving fine-grained description of the image and providing richer and more specific image information. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the present application or the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0022] FIG1 is a flow chart of a method for fine-grained image description by fine-tuning a multimodal large model using instructions provided in this application;

[0023] FIG2 is a schematic diagram of the model structure of the visual encoder in the multimodal large model provided by this application;

[0024] FIG3 is a schematic diagram of the model structure of the text decoder in the multimodal large model provided by this application;

[0025] FIG4 is a schematic diagram of the next word segmentation prediction task provided by this application;

[0026] FIG5 is a schematic diagram of a self-attention mask matrix for causal language modeling provided by this application;

[0027] FIG6 is a schematic diagram of the forward propagation of the text decoder provided by the present application during the inference / testing phase;

[0028] FIG7 is a schematic diagram of a flow chart of instruction fine-tuning of a multimodal large model provided by this application;

[0029] FIG8 is a schematic diagram of a process for constructing fine-grained description text of an image provided by the present application;

[0030] FIG9 is a second flow chart of the method for fine-tuning a multimodal large model for fine-grained image description provided by this application;

[0031] FIG10 is a schematic diagram of the structure of the image fine-grained description system for fine-tuning a multimodal large model provided by the present application;

[0032] FIG11 is a schematic diagram of the physical structure of the electronic device provided in this application. DETAILED DESCRIPTION

[0033] To make the objectives, technical solutions, and advantages of this application more clear, the technical solutions of this application will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments of this application, all other embodiments obtained by ordinary technicians in this field without making any creative efforts are within the scope of protection of this application.

[0034] The method for fine-grained image description of a multimodal large model using instruction fine-tuning technology provided in this application successfully implements instruction fine-tuning of the multimodal large model in downstream image description tasks using a very small amount of multimodal instruction data, thereby achieving fine-grained description of the image. The specific implementation is as follows.

[0035] FIG1 is one of the flow charts of the method for fine-grained image description of instruction fine-tuning multimodal large models provided in this application. As shown in FIG1 , the method includes the following steps.

[0036] Step 110: Obtain a first vector sequence corresponding to the target image, where the first vector sequence corresponding to the target image is obtained by encoding a second vector sequence corresponding to the target image, and the second vector sequence corresponding to the target image is obtained based on high-level semantic information extracted from the target image.

[0037] Step 120 : Acquire a fine-grained description text of the target image according to the first vector sequence and a first prompt template, where the first prompt template is used to provide instruction information required for a fine-grained description of the target image.

[0038] It should be noted that the execution subject of the above method may be a computer device.

[0039] Optionally, the target image may be an image to be described in a fine-grained manner, more specifically a three-channel RGB image. The first vector sequence corresponding to the target image is obtained by encoding a second vector sequence corresponding to the target image, wherein the dimension of the second vector sequence is the same as that of the first vector sequence. The first vector sequence may be composed of multiple one-dimensional vectors of fixed dimensions.

[0040] The second vector sequence corresponding to the target image is a numerical carrier of the high-level semantic information of the target image, which can be obtained by extracting the high-level semantic information of the target image, more specifically by inputting the target image into a visual encoder (such as a visual Transformer model). Among them, this high-level semantic information can be specifically used to characterize the attributes and features of important objects in the target image, which can be widely applied to various downstream tasks such as object detection, instance segmentation, image depth estimation, etc.

[0041] The prompt template can also be called an instruction template, which is any text prompt template (or instruction information) randomly extracted from the text prompt template set. The instruction information used to describe the target image in fine granularity is the first prompt template.

[0042] Optionally, the target image is described in a fine-grained manner using the instruction information in the first prompt template and the first vector sequence corresponding to the acquired target image, thereby generating a fine-grained description text corresponding to the target image, or referred to as a fine-grained image description text.

[0043] The image fine-grained description method of the instruction fine-tuning multimodal large model provided in this application can accurately identify and describe the attributes and features of important targets in the image based on a small amount of instruction information required for fine-grained description of the image and a vector sequence carrying high-level semantic information of the image, thereby achieving fine-grained description of the image and providing richer and more specific image information.

[0044] In one embodiment, obtaining the fine-grained description text of the target image according to the first vector sequence and the first hint template may include:

[0045] Inputting the first vector sequence and the first hint template into a target text decoder to obtain the fine-grained description text, the target text decoder is used to generate the fine-grained description text corresponding to the target image based on the first hint template, the first vector sequence corresponding to the target image is obtained by inputting the second vector sequence corresponding to the target image into a target multimodal information adaptation module in a multimodal large model, the target multimodal information adaptation module is used to encode the second vector sequence corresponding to the target image to obtain the first vector sequence corresponding to the target image;

[0046] The target text decoder and the target multimodal information adaptation module are obtained by:

[0047] splicing a first vector sequence and a second hint template corresponding to each image in the image dataset to obtain a first text sequence, wherein the second hint template is used to provide instruction information required for fine-grained description of the images in the image dataset, and the text decoder is used to generate a fine-grained description text corresponding to each image according to the second hint template;

[0048] splicing the fine-grained description text corresponding to each image in the image dataset onto the end of the first text sequence to obtain a second text sequence;

[0049] Inputting the second text sequence into a text decoder in the multimodal large model, and fine-tuning the multimodal information adaptation module and the text decoder in the multimodal large model until the target loss function converges;

[0050] Using the converged multimodal information adaptation module as the target multimodal information adaptation module;

[0051] The converged text decoder is used as the target text decoder.

[0052] In the quest for artificial general intelligence (AGI), researchers have built large multimodal models for cross-modal understanding and generation tasks. These models are capable of processing and understanding data from multiple modalities, such as text, images, and audio. However, the true power of these models lies not solely in their multimodal capabilities, but rather in their extremely low fine-tuning costs and their robust few-shot understanding and generation capabilities when applied to downstream tasks.

[0053] Fine-tuning is a method for adapting a model to a specific downstream task. Through fine-tuning, the model focuses on understanding the details of a specific downstream task. The potential of fine-tuning is enormous, as it allows us to achieve a deeper understanding of downstream tasks and more accurately understand and generate modal semantic information at minimal data and hardware costs.

[0054] However, research on fine-tuning methods for large multimodal models is still in its developmental stages, and researchers need more techniques to explore and optimize the fast fine-tuning of large multimodal models. Researchers need to fine-tune large multimodal models more efficiently, understand the semantic information of each modality more accurately, and apply this method to more fields.

[0055] Although large multimodal models perform well on some downstream tasks, challenges remain in fine-grained image description. This is because fine-grained image description requires the model to have a deep understanding of the image content, including the ability to identify subtle details in the image, understand the meaning of these details, and be able to convert this understanding into accurate and detailed text descriptions. Therefore, this application focuses on a fine-grained image description method based on instruction fine-tuning of large multimodal models, which will not only promote the development of large multimodal models, but also provide new ideas and methods for solving this challenge.

[0056] The core idea of ​​each embodiment of this application is to organically combine a large multimodal model and instruction fine-tuning technology for fine-grained image description downstream tasks.

[0057] This application can not only be used in the field of fine-grained image description, but also has a positive driving effect on other research fields of multimodal large models. First, this application provides a new perspective to understand and describe multimodal data, which enables us to have a deeper understanding of the intrinsic connections between multimodal data and more accurately describe the content of multimodal data. This accuracy of understanding and depth of description can provide new ideas and methods for researchers in other fields; secondly, it promotes the research on fine-tuning strategies of multimodal large models in downstream tasks. The construction of instruction fine-tuning and instruction data sets in this application can provide new ideas and methods for research in other fields of multimodal large models; finally, this application promotes the understanding and application of multimodal large models. By studying this application, we can deeply understand the working principles of multimodal large models, explore the best application strategies of multimodal large models in downstream tasks, and optimize the application effects of multimodal large models.

[0058] Optionally, by inputting the first vector sequence corresponding to the target image and its corresponding first prompt template into the target text decoder in the multimodal large model, the target text decoder generates a fine-grained description text corresponding to the target image according to the first prompt template.

[0059] Among them, the first vector sequence corresponding to the target image is obtained by inputting the second vector sequence corresponding to the target image into the target multimodal information adaptation module in the multimodal information large model, and encoding the second vector sequence corresponding to the target image based on the target multimodal information adaptation module to obtain the first vector sequence corresponding to the target image.

[0060] In one embodiment, a method for acquiring the second vector sequence corresponding to the target image may include:

[0061] The target image is input into a visual encoder in a multimodal large model to obtain the second vector sequence corresponding to the target image. The visual encoder is used to extract high-level semantic information in the target image and obtain the second vector sequence corresponding to the target image.

[0062] Optionally, the second vector sequence of the target image can be specifically obtained by inputting the target image into a visual encoder in the multimodal large model and encoding the target image into an image embedding vector sequence, i.e., a second vector sequence, based on the visual encoder. The visual encoder is configured to extract high-level semantic information from the target image to obtain the second vector sequence corresponding to the target image.

[0063] Optionally, the target text decoder and the target multimodal information adaptation module in the multimodal large model are obtained as follows:

[0064] Build the overall architecture of a large multimodal model, which mainly consists of three parts: a visual encoder, a multimodal information adaptation module, and a text decoder;

[0065] The images in the image dataset, the fine-grained description text corresponding to the image, and the corresponding prompt template (i.e., the second prompt template) are used as input to the multimodal large model to perform instruction fine-tuning on the multimodal large model. Among them, the images in the image dataset are all three-channel RGB images.

[0066] Input each image in the image dataset into a visual encoder, extract high-level semantic information in the image based on the visual encoder, and obtain a second vector sequence corresponding to each image;

[0067] The second vector sequence corresponding to each image is input into the multimodal information adaptation module, the second vector sequence is encoded based on the multimodal information adaptation module, and the first sequence vector corresponding to the target image is output.

[0068] Figure 2 is a schematic diagram of the model structure of the visual encoder in the multimodal large model provided by this application. Referring to Figure 2, the visual encoder adopts a visual Transformer structure, and the input is a three-channel RGB image. The size of the visual encoder input image is fixed (for example, 224x224), and the input image is evenly divided into image blocks of the same size. The image block size is artificially specified (for example, 14x14, 16x16, etc.), and then all image blocks are mapped to feature vectors of the image blocks using a fully connected layer. The image block feature vector undergoes a self-attention operation in the visual Transformer (ViT) to output an image embedding vector sequence, i.e., a second vector sequence. The image embedding vector sequence consists of multiple one-dimensional vectors of fixed dimensions. The visual encoder extracts high-level semantic information of the image, and the pre-trained visual encoder can be widely used in a variety of downstream tasks, such as target detection, instance segmentation, and image depth estimation.

[0069] The dimension of the second vector sequence output by the multimodal information adaptation module is the same as the dimension of the first vector sequence input by the text decoder. The functions of the multimodal information adaptation module are: (1) changing the dimension of the second vector sequence output by the visual encoder to adapt it to the input of the text decoder; (2) since there is a modality gap between high-level semantic information of different modalities, the multimodal information adaptation module further encodes the second vector sequence of the target image so that the visual modality information extracted by the visual encoder can adapt to the text decoder, eliminating the modality gap between the visual and text modalities and semantically aligning the visual modality information with the text modality information. The multimodal information adaptation module can generally adopt a single fully connected layer or a 2-layer or 3-layer multi-layer perceptron.

[0070] The first vector sequence and the second prompt template corresponding to each image are spliced ​​together to obtain a first text sequence, and the fine-grained description text corresponding to each image is spliced ​​to the end of the first text sequence to obtain a second text sequence. The second prompt template can be specifically any text prompt template (or called instruction information) randomly selected from the text prompt template set, which can be specifically used to provide the instruction information required for fine-grained description of images in the image dataset.

[0071] Inputting the second text sequence into the text decoder to fine-tune the multimodal information adaptation module and the text decoder in the multimodal large model until the target loss function converges;

[0072] The converged multimodal information adaptation module is used as the target multimodal information adaptation module, and the converged text decoder is used as the target text decoder.

[0073] Optionally, Figure 3 is a schematic diagram of the model structure of the text decoder in the multimodal large model provided by the present application. Referring to Figure 3, the text decoder in the present application can adopt Transformer as its structure (in order to ensure the structural consistency of the visual encoder and the text decoder, the visual encoder adopts the visual Transformer structure), and the input of the text decoder in the instruction fine-tuning stage consists of three parts, namely, the second vector sequence output by the multimodal information adaptation module, the second prompt template, and the fine-grained description text corresponding to the image in the image dataset. The input of the text decoder in the reasoning / testing stage consists of two parts, namely, the prompt template and the image embedding vector sequence output by the multimodal information adaptation module.

[0074] Among them, the role of the prompt template is to prompt the text decoder to perform fine-grained image description tasks, or to issue instructions to the text decoder to generate fine-grained description text corresponding to the image. For example, the prompt template can be defined as: generate its corresponding fine-grained description text based on the image <image>, and the <image> in the prompt template is replaced by the image embedding vector sequence (i.e., the second vector sequence) output by the multimodal information adaptation module.

[0075] Optionally, Figure 4 is a schematic diagram of the next word segmentation prediction task provided by the present application. Referring to Figure 4, the text decoder performs the next word segmentation prediction task in the instruction fine-tuning stage, also known as the autoregressive task, that is, each word segmentation only predicts what the next word segmentation adjacent to it is, without predicting the word segmentation before it and after the next word segmentation. The corresponding loss function of the next word segmentation prediction task (i.e., the target loss function) is the cross-entropy loss. The next word segmentation prediction task of the text decoder in the training stage uses the fine-grained description text of the image in the image dataset as the prediction target, which is also the true value of the text decoder, that is, the cross-entropy loss value only includes the loss value of the predicted fine-grained text description part of the image, and does not include the cross-entropy loss value of the second prompt template and the second vector sequence corresponding to the image in the image dataset in the next word segmentation prediction task.

[0076] Optionally, Figure 5 is a schematic diagram of the self-attention mask matrix of the causal language modeling provided by this application. Referring to Figure 5, in order to adapt to the next word segmentation prediction task, the self-attention operation of the text decoder adopts the causal language modeling mechanism.

[0077] The principle of causal language modeling is that a word in the input sequence of the text decoder performs self-attention operations only on the word before it, and not on the word after it. The causal language model is specifically implemented using a self-attention mask matrix. The self-attention query sequence (query) and the key sequence (key) are subjected to the dot product operation in the conventional self-attention operation to obtain the logits matrix. The self-attention mask matrix and the logits matrix are then added element by element. The self-attention matrix is ​​a lower triangular matrix with all diagonal and lower triangular element values ​​being 0 and all upper triangular element values ​​being negative infinity. The result of adding the self-attention mask matrix and the logistic matrix element by element is called the causal logistic matrix. The causal self-attention coefficient matrix is ​​obtained by further performing the conventional self-attention softmax operation on the causal logistic matrix. The upper triangular elements in the causal logistic matrix are all negative infinity. Therefore, after the softmax operation, the upper triangular elements in the causal self-attention coefficient matrix are all 0, which achieves the purpose of performing self-attention operation on each segmentation in the self-attention operation only for itself and the previous segmentation, preventing information leakage and model overfitting in conventional self-attention operation.

[0078] Optionally, Figure 6 is a schematic diagram of the forward propagation of the text decoder provided by the present application in the reasoning / testing stage. Referring to Figure 6, the text decoder only generates one segmentation word for each forward propagation in the reasoning / testing stage until a special segmentation word marking the end of the sentence is generated. Similar to the instruction fine-tuning stage, the text decoder only generates the next adjacent segmentation word in the reasoning / testing stage. Assuming that the text decoder generates a total of n segmentations (including the special segmentation word marking the end of the sentence) in the reasoning / verification stage, the text decoder needs to forward propagate n times in total, and each forward propagation adds the segmentation word generated by the previous forward propagation to the end of the input text sequence of the text decoder as a new input text sequence to the text decoder to predict the next segmentation word.

[0079] Optionally, during the fine-tuning phase, the visual encoder and text decoder load pre-trained model parameter values. For example, if a visual transformer is used as the visual encoder, the pre-trained parameter values ​​of CLIP-ViT-large-patch14 can be directly loaded; if LLaMA is used as the text decoder, the pre-trained parameter values ​​of Vicuna-7B or Vicuna-13B can be directly loaded. The parameters of the multimodal information adaptation module are randomly initialized. During the instruction fine-tuning phase, the weights of the visual encoder are frozen, and the weights of the multimodal information adaptation module and text decoder are fine-tuned.

[0080] Optionally, the fine-grained image description based on the multimodal large model needs to design a text prompt template set in the instruction fine-tuning stage, and the image, the second prompt template and the fine-grained description text corresponding to the image are used as the input of the multimodal large model. The text prompt template set in the instruction fine-tuning stage is used as the text prompt template set of the text decoder. In each iteration, a prompt template is randomly selected from the text prompt template set, and the selected prompt template, the image sequence and the fine-grained image description are spliced ​​in a certain order to obtain the input sequence of the text decoder. The prompt template has two main functions. (1) The second vector sequence corresponding to the image output by the multimodal information adaptation module is spliced ​​with the second prompt template to obtain the first text sequence, and the fine-grained description text of the image is reasonably spliced ​​to the back of the first text sequence to obtain a text sequence that the text decoder can understand, that is, the second text sequence. The second vector sequence output by the multimodal information adaptation module can be regarded as another foreign language completely different from the fine-grained description text. (2) The prompt template contains rich instruction semantic information, and the text decoder performs the fine-grained image description text generation task based on the rich instruction semantic information in the prompt template. For example, a text prompt template set could consist of the following prompt templates:

[0081] “<image embedding vector sequence start word><image embedding vector sequence><image embedding sequence end word>Please describe this image in detail.”;

[0082] "<image embedding sequence start word><image embedding vector sequence><image embedding sequence end word> Please describe the important objects in this image, including the object's texture, color, relationship, quantity and other detailed features.";

[0083] “<image embedding sequence start word><image embedding vector sequence><image embedding sequence end word> Look carefully at this image and describe its contents in as much detail as possible.”;

[0084] "<image embedding sequence start word><image embedding vector sequence><image embedding sequence end word> Please provide a sufficiently detailed text description of this image.";

[0085] “<image embedding sequence start word><image embedding vector sequence><image embedding sequence end word> Can you describe the content of this image in detail for me?”.

[0086] The instruction semantics of the prompt template are exactly the same, the only difference is the text expression. The <image embedding sequence start word> and <image embedding sequence end word> in the prompt template are special words that mark the start and end of the image embedding vector sequence respectively.

[0087] In one embodiment, the objective loss function is obtained by:

[0088] The target loss function is calculated based on the predicted probability of the predicted segmented words in the first segmented word sequence corresponding to the second text sequence and the integer index of the fine-grained description text corresponding to the image in the image dataset in a pre-constructed segmented word table, wherein the predicted probability is obtained based on the first segmented word sequence, the second segmented word sequence and the third segmented word sequence, the first segmented word sequence is obtained by removing the predicted segmented words in the target segmented word sequence, the target segmented word sequence is obtained by segmenting the fine-grained description text corresponding to each image in the second text sequence, the second segmented word sequence is obtained by segmenting the second prompt template corresponding to each image, and the third segmented word sequence is obtained by segmenting the first vector sequence corresponding to each image.

[0089] Optionally, Figure 7 is a flow chart of instruction fine-tuning of the multimodal large model provided in the present application. Referring to Figure 7, the fine-grained image description method of the multimodal large model takes the image, the fine-grained description text corresponding to the image, and the second prompt template as the input of the multimodal large model, and the visual encoder extracts visual semantic information from the image to obtain an image embedding vector sequence, i.e., a second vector sequence. The multimodal information adaptation module eliminates the semantic gap inherent in the image embedding vector sequence output by the visual encoder. In the instruction fine-tuning stage, a prompt template is randomly extracted from the text prompt template set, and then reasonably spliced ​​with the image embedding vector sequence and the fine-grained description text and used as the input of the text decoder. In the instruction fine-tuning stage, the text decoder adopts the next word segmentation prediction task, and the loss function adopts the conventional cross entropy loss.

[0090] Based on the predicted probability of the predicted word in the first word segmentation sequence corresponding to the second text sequence by the text decoder and the integer index of the fine-grained description text corresponding to the image in the image dataset in the word segmentation vocabulary, a target loss function is calculated, and the word segmentation corresponding to the fine-grained description text corresponding to the integer index is used as the true target value of the cross-entropy loss function to calculate the target loss function value. The pre-constructed word segmentation vocabulary (or word segmentation dictionary) can be obtained based on an existing word segmentation algorithm.

[0091] The target word segmentation sequence X is obtained by segmenting the fine-grained description text corresponding to each image in the second text sequence. c ,in, q represents the length of the word segmentation sequence obtained after the fine-grained description text of the image is segmented.

[0092] The first segmentation sequence is obtained by removing the predicted segmentation words in the target segmentation sequence, and the second segmentation sequence X is obtained by segmenting the second prompt template corresponding to each image in the image dataset. p ,in, n represents the length of the word segmentation sequence obtained by the second hint template segmentation; the third word segmentation sequence X is obtained by segmenting the first vector sequence corresponding to each image. v ,in, m represents the length of the word segmentation sequence obtained after the first vector sequence is segmented.

[0093] Calculate the text decoder for the predicted word segmentation according to the first word segmentation sequence, the second word segmentation sequence and the third word segmentation sequence The predicted probability, that is, the probability of the next word predicted by the text decoder

[0094] Here, θ represents the weights updated by the multimodal large model during the instruction fine-tuning phase. The multimodal large model only fine-tunes the parameters of the multimodal information adaptation module and the text decoder. That is, θ represents the parameters / weights of the multimodal information adaptation module and the text decoder.

[0095] The significance of the next word segmentation prediction task is that the text decoder predicts the probability of a word in the fine-grained description text based only on three parts: (1) the first vector sequence of the image; (2) the second word segmentation sequence corresponding to the second prompt template; (3) the word segmentation sequence that precedes the predicted word in the target word segmentation sequence after the fine-grained description text segmentation, that is, the first word segmentation sequence.

[0096] The predicted probability of the predicted word in the first word sequence corresponding to the second text sequence according to the text decoder And the integer index Label of the fine-grained description text corresponding to the image in the image dataset in the word segmentation vocabulary i , calculate the cross entropy loss of the multimodal large model in the instruction fine-tuning stage, that is, the target loss function loss:

[0097] Among them, the cross entropy loss function value calculation formula has two inputs, namely and Label i The cross entropy function CrossEntropy in the cross entropy loss calculation formula only calculates the cross entropy loss value of a word in the fine-grained description text corresponding to the image. The first input item in the cross entropy loss calculation formula is The one-dimensional probability vector obtained by the text decoder and softmax operation for a word in the fine-grained description text is in the range [0,1]. The dimension of the vector is equal to the total number of words in the word list. The one-dimensional probability vector is named the logistic vector of the word corresponding to the fine-grained description text. The second input item in the cross entropy loss calculation formula is Label. i , represents the integer index of the fine-grained description text in the word segmentation vocabulary, which is also the word segmentation category label of the logistic value. The word segmentation vocabulary is obtained by tokenizing the second prompt template and the fine-grained description text corresponding to each image.

[0098] The calculation method of the cross entropy function CrossEntropy can be expressed by the following formula:

[0099] Among them, in the calculation method of the cross entropy function, the logistic value corresponding to the word segmentation category label in the logistic vector is taken, the natural logarithm with base e is taken, and then the inverse is taken to obtain the loss value of a single word segmentation in the fine-grained description text. The loss values ​​of all word segmentations in the fine-grained description text are averaged, which is the loss function value of the next word segmentation prediction task.

[0100] The image fine-grained description method of the instruction fine-tuning multimodal large model provided in this application uses the multimodal large model in the downstream fine-grained image description field, fully tapping the understanding and generation capabilities of the multimodal large model; organically combining the instruction fine-tuning technology with the multimodal large model, greatly reducing the training cost of the multimodal large model for downstream tasks, achieving fine-grained description of the image, and fully tapping the information in the image.

[0101] In one embodiment, a method for obtaining the image dataset and the fine-grained description text corresponding to each image includes:

[0102] Constructing the image dataset based on the images in the dataset according to the public graphics and text;

[0103] Based on seed instruction text data, the public image-text pair dataset, and a high-performance dialogue model, obtaining the instruction fine-tuning dataset, wherein the seed instruction text data includes at least one first image-text pair, wherein the first image-text pair is composed of an image obtained from the public image-text pair dataset and manually written fine-grained description text corresponding to the image;

[0104] The high-performance dialogue model is used to filter and denoise the fine-grained description text of the images in the instruction fine-tuning dataset to obtain the fine-grained description text corresponding to each image.

[0105] Optionally, FIG8 is a second flow chart of the process of constructing fine-grained description text of an image provided by the present application. Referring to FIG8 , constructing fine-grained description text corresponding to each image in the image dataset mainly includes the following three steps: (1) designing seed instruction text data; (2) constructing an instruction fine-tuning dataset for fine-grained image description based on the seed instruction text data, a public image-text pair dataset, and a high-performance dialogue model (such as ChatGPT, GPT4, etc.); (3) using the high-performance dialogue model to filter and denoise the fine-grained description text in the instruction fine-tuning dataset for fine-grained image description to obtain the fine-grained description text corresponding to each image.

[0106] The image dataset may be constructed based on images in a public image-text pair dataset. The public image-text pair dataset may include existing high-quality image-text datasets, such as COCO and CC3M.

[0107] At present, all instruction fine-tuning data are text instruction data. In order to fine-tune the instructions of the multimodal large model to achieve fine-grained description of the image, it is necessary to construct a high-quality instruction fine-tuning dataset for fine-grained image description. First, the seed instruction text data of the instruction fine-tuning dataset for fine-grained image description is designed. In order to ensure the quality of the seed instruction text data and the subsequently generated fine-grained description text, the seed instruction text data is completely manually designed. The seed instruction text data includes at least one first image-text pair, which is composed of an image obtained from a public image-text pair dataset and a manually written fine-grained description text corresponding to the image.

[0108] To minimize the impact of noise, we sampled a certain number of high-quality image-text pairs (i.e., seed image-text pairs) from existing public high-quality image-text datasets (such as COCO and CC3M). These high-quality image-text pairs serve as metadata for the seed instruction text data, and the number is generally less than 200. Because image descriptions in existing public high-quality image-text datasets are generally short, we manually compose fine-grained description text based on the metadata of the seed instruction text data (a certain number of high-quality image-text pairs). For example, the original image description of a certain image-text pair in the metadata of the seed instruction text data is: "Three puppies sitting on a woman's lap, looking out the car window."

[0109] The manually written / designed fine-grained description text is: "This is an image filled with warmth and harmony. Inside a car, a woman sits comfortably on her lap with three attentive puppies, all captivated by the view outside the window. In front of the car, a mobile phone sits quietly, perhaps prepared by the woman to record this beautiful moment. This scene is filled with a relaxed and joyful atmosphere, a moment that brings joy." A fine-grained description text is written for each image-text pair in the metadata of the seed instruction text data. The manually written fine-grained description text replaces the original image description in each image-text pair in the metadata to obtain the first image-text pair, thus forming the manually designed seed instruction text data.

[0110] In one embodiment, obtaining the instruction fine-tuning dataset based on the seed instruction text data, the public image-text pair dataset, and the high-performance dialogue model may include:

[0111] Selecting a first preset number of target image-text pairs from the first image-text pairs included in the seed instruction text data as seed instruction text examples;

[0112] The seed instruction text example, target text, and construction instruction are spliced ​​together to obtain a third text sequence, wherein the target text is obtained by converting size information of the image in the second image-text pair, original description text corresponding to the image in the second image-text pair, and detection boxes of all objects in the image in the second image-text pair into text form, the second image-text pair is a second preset number of image-text pairs extracted from the public image-text pair dataset, the original description text is the description text of the second image-text pair in the public image-text pair dataset, and the construction instruction is used to provide target instruction information for the high-performance dialogue model, and the target instruction information is instruction information required for fine-grained description of the images in the image dataset based on the seed instruction text example and the target text;

[0113] The third text sequence is input into the high-performance dialogue model to obtain the instruction fine-tuning dataset.

[0114] Optionally, a fine-grained image description instruction fine-tuning dataset is constructed based on seed instruction text data, an existing public image-text pair dataset, and a high-performance dialogue model (such as ChatGPT, GPT4, etc.). The existing public image-text pair dataset can be COCO, and a second preset number (for example, 50,000) of image-text pairs (i.e., second image-text pairs) are extracted from the existing public image-text pair dataset. The images in the second image-text pairs, the detection boxes of all objects in the images in the second image-text pairs, such as rectangular box annotations, and the size information of the images in the second image-text pairs (such as width and height) are used as metadata of the fine-grained image description instruction fine-tuning dataset; the seed instruction text data and the metadata of the instruction fine-tuning dataset are used as input to the high-performance dialogue model, and the instruction following capability inherent in the high-performance dialogue model is used to obtain fine-grained description text of the image.

[0115] Specifically, (1) a first preset number of target image-text pairs are selected from the first image-text pairs included in the seed instruction text data as input examples / prompts, and are named seed instruction text examples; (2) a second preset number of image-text pairs are selected from the instruction fine-tuning dataset to obtain a second image-text pair, and the original description text of the image in the second image-text pair (i.e., the description text of the second image-text pair in the public image-text pair dataset), the rectangular box annotations of all objects in the image in the second image-text pair, and the size information of the image are converted into text form, and are named as the textual metadata of the image, i.e., the target text; (3) the seed instruction text example and the target text are concatenated and used as the input of the high-performance dialogue model; (4) an instruction text for constructing the instruction fine-tuning dataset is selected and named as the construction instruction of the instruction fine-tuning dataset. The function of the construction instruction is to enable the high-performance dialogue model to generate a fine-grained description text of the image based on the seed instruction text example and the target text; (5) the long text obtained by concatenating the seed instruction text example and the target text, and the third text sequence obtained by concatenating the long text and the construction instruction in a reasonable manner are used as the input of the high-performance dialogue model. Generally speaking, the input of the high-performance dialogue model consists of three parts: seed instruction text examples, target text, and construction instructions. The high-performance dialogue model uses its inherent instruction-following ability and the input third text sequence to generate fine-grained description text of the image; (6) The above steps are iterated until the construction of the instruction fine-tuning dataset is completed.

[0116] In one embodiment, using the high-performance dialogue model, filtering and denoising the fine-grained description text of the images in the instruction fine-tuning dataset to obtain the fine-grained description text corresponding to each image may include:

[0117] The fine-grained description text and post-processing instructions of the images in the instruction fine-tuning dataset are spliced ​​and input into the high-performance dialogue model to obtain the fine-grained description text corresponding to each image. The post-processing instructions are used to guide the high-performance dialogue model to filter and denoise the fine-grained description text of the images in the instruction fine-tuning dataset.

[0118] Optionally, a high-performance dialogue model is used to filter and denoise the fine-grained description text of the images in the instruction fine-tuning dataset, obtaining fine-grained description text corresponding to each image. This step requires the instruction text to guide the high-performance dialogue model in filtering and denoising post-processing, and the corresponding instruction text is named post-processing instruction. Each iteration of this step extracts a certain amount of fine-grained description text from the instruction fine-tuning dataset. The post-processing instructions and fine-grained description text are concatenated in a reasonable manner as input to the high-performance dialogue model, and the output of the high-performance dialogue model is the denoised fine-grained description text.

[0119] The post-processing instructions can use the following text template: Fix errors in the given paragraph. Delete any duplicate sentences, meaningless characters, non-Chinese sentences, etc. Remove unnecessary repetitions. Rewrite any incomplete sentences. Return the result directly without explanation. If the input paragraph is correct, return the input paragraph directly without explanation.

[0120] FIG9 is a second flow chart of the method for fine-grained image description by fine-tuning a multimodal large model with instructions provided in this application. Referring to FIG9 , the method includes:

[0121] Build the overall architecture of a large multimodal model, which mainly includes a visual encoder, a multimodal information adaptation module, and a text decoder;

[0122] Design seed instruction text data and build a fine-tuning dataset of fine-grained image description instructions based on a high-performance dialogue model, i.e., fine-grained image description text.

[0123] Fine-tuning large multimodal models using images, fine-grained description text, and prompt templates.

[0124] The fine-grained image description method for instruction fine-tuning of a large multimodal model provided in this application improves the convergence speed of instruction fine-tuning of a large multimodal model by constructing high-quality fine-grained description text, enabling researchers to fine-tune the large multimodal model more efficiently and understand the semantic information of each modality more accurately, thereby applying it to more fields.

[0125] The image fine-grained description system for instruction fine-tuning a multimodal large model provided in this application is described below. The image fine-grained description system for instruction fine-tuning a multimodal large model described below and the image fine-grained description method for instruction fine-tuning a multimodal large model described above can be referenced to each other.

[0126] FIG10 is a schematic diagram of the structure of the fine-grained image description system for fine-tuning a multimodal large model provided by this application, as shown in FIG10 , including:

[0127] A data acquisition module 1010 is configured to acquire a first vector sequence corresponding to a target image, wherein the first vector sequence corresponding to the target image is obtained by encoding a second vector sequence corresponding to the target image, wherein the second vector sequence corresponding to the target image is obtained based on high-level semantic information extracted from the target image;

[0128] The image description module 1011 is configured to obtain a fine-grained description text of the target image according to the first vector sequence and a first prompt template, wherein the first prompt template is configured to provide instruction information required for a fine-grained description of the target image.

[0129] The image fine-grained description system of the instruction fine-tuning multimodal large model provided in this application can accurately identify and describe the attributes and features of important targets in the image based on a small amount of instruction information required for fine-grained description of the image and a vector sequence carrying high-level semantic information of the image, thereby achieving fine-grained description of the image and providing richer and more specific image information.

[0130] FIG11 is a schematic diagram of the physical structure of an electronic device provided by the present application. As shown in FIG11 , the electronic device may include: a processor 1110, a communication interface 1111, a memory 1112, and a bus 1113. The processor 1110, the communication interface 1111, and the memory 1112 communicate with each other via the bus 1113. The processor 1110 may call the logic instructions in the memory 1112 to execute the following method:

[0131] Obtaining a first vector sequence corresponding to a target image, wherein the first vector sequence corresponding to the target image is obtained by encoding a second vector sequence corresponding to the target image, and the second vector sequence corresponding to the target image is obtained based on high-level semantic information extracted from the target image;

[0132] A fine-grained description text of the target image is obtained according to the first vector sequence and a first prompt template, where the first prompt template is used to provide instruction information required for a fine-grained description of the target image.

[0133] In addition, the logical instructions in the above-mentioned memory can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer power screen (which can be a personal computer, a server, or a network power screen, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0134] Furthermore, the present application discloses a computer program product, comprising a computer program stored on a non-transitory computer-readable storage medium, wherein the computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the instructions provided in the above-mentioned method embodiments to fine-tune the image fine-grained description method of a multimodal large model, for example, including:

[0135] Obtaining a first vector sequence corresponding to a target image, wherein the first vector sequence corresponding to the target image is obtained by encoding a second vector sequence corresponding to the target image, and the second vector sequence corresponding to the target image is obtained based on high-level semantic information extracted from the target image;

[0136] A fine-grained description text of the target image is obtained according to the first vector sequence and a first prompt template, where the first prompt template is used to provide instruction information required for a fine-grained description of the target image.

[0137] On the other hand, the present application also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the method for fine-tuning a multimodal large model image fine-grained description by executing the instructions provided in the above embodiments is implemented, for example, including:

[0138] Obtaining a first vector sequence corresponding to a target image, wherein the first vector sequence corresponding to the target image is obtained by encoding a second vector sequence corresponding to the target image, and the second vector sequence corresponding to the target image is obtained based on high-level semantic information extracted from the target image;

[0139] A fine-grained description text of the target image is obtained according to the first vector sequence and a first prompt template, where the first prompt template is used to provide instruction information required for a fine-grained description of the target image.

[0140] The system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units. They may be located in one place or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of this embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0141] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, or of course by hardware. Based on this understanding, the above technical solution, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product, which can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for enabling a computer power screen (which can be a personal computer, a server, or a network power screen, etc.) to execute the methods described in each embodiment or certain parts of the embodiments.

[0142] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A fine-grained image description method for fine-tuning a multimodal large model by instructions, comprising: Acquire a first vector sequence corresponding to a target image, wherein the first vector sequence corresponding to the target image is obtained by encoding a second vector sequence corresponding to the target image, and the second vector sequence corresponding to the target image is obtained according to high-level semantic information extracted from the target image; According to the first vector sequence and the first prompt template, a fine-grained description text of the target image is obtained, wherein the first prompt template is used to provide instruction information required for a fine-grained description of the target image.

2. The image fine-grained description method for fine-tuning a multimodal large model according to claim 1, wherein: The step of acquiring the fine-grained description text of the target image according to the first vector sequence and the first prompt template includes: The first vector sequence and the first prompt template are input into a target text decoder to obtain the fine-grained description text, wherein the target text decoder is used to generate the fine-grained description text corresponding to the target image according to the first prompt template, the first vector sequence corresponding to the target image is obtained by inputting the second vector sequence corresponding to the target image into a target multimodal information adaptation module in a multimodal large model, and the target multimodal information adaptation module is used to encode the second vector sequence corresponding to the target image to obtain the first vector sequence corresponding to the target image, Wherein, the target text decoder and the target multimodal information adaptation module are obtained in the following manner: splicing a first vector sequence and a second prompt template corresponding to each image in the image data set to obtain a first text sequence, wherein the second prompt template is used to provide instruction information required for fine-grained description of the image in the image data set, and the text decoder is used to generate a fine-grained description text corresponding to each image according to the second prompt template; splicing the fine-grained description text corresponding to each image in the image data set to the end of the first text sequence to obtain a second text sequence; Inputting the second text sequence into a text decoder in the multimodal large model, and fine-tuning the multimodal information adaptation module and the text decoder in the multimodal large model until the target loss function converges; Using the converged multimodal information adaptation module as the target multimodal information adaptation module; The converged text decoder is used as the target text decoder.

3. The image fine-grained description method for fine-tuning a multimodal large model according to claim 2, wherein: The calculation method of the target loss function includes: The target loss function is calculated according to the predicted probability of the predicted segmented words in the first segmented word sequence corresponding to the second text sequence and the integer index of the fine-grained description text corresponding to the image in the image data set in a pre-constructed segmented word table, wherein the predicted probability is obtained according to the first segmented word sequence, the second segmented word sequence and the third segmented word sequence, the first segmented word sequence is obtained by removing the predicted segmented words in the target segmented word sequence, the target segmented word sequence is obtained by segmenting the fine-grained description text corresponding to each image in the second text sequence, the second segmented word sequence is obtained by segmenting the second prompt template corresponding to each image, and the third segmented word sequence is obtained by segmenting the first vector sequence corresponding to each image.

4. The image fine-grained description method for fine-tuning a multimodal large model according to claim 2, wherein: The method for obtaining the image dataset and the fine-grained description text corresponding to each image includes: Constructing the image dataset according to the images in the dataset based on the public graphics and texts; Based on the seed instruction text data, the public image-text pair dataset and the high-performance dialogue model, an instruction fine-tuning dataset is obtained, wherein the seed instruction text data includes at least one first image-text pair, and the first image-text pair is composed of an image obtained from the public image-text pair dataset and a manually written fine-grained description text corresponding to the image; The high-performance dialogue model is used to filter and denoise the fine-grained description text of the images in the instruction fine-tuning dataset to obtain the fine-grained description text corresponding to each image.

5. The image fine-grained description method for fine-tuning a multimodal large model according to claim 4, wherein: The step of obtaining the instruction fine-tuning dataset based on the seed instruction text data, the public image-text pair dataset, and the high-performance dialogue model includes: Selecting a first preset number of target image-text pairs from the first image-text pairs included in the seed instruction text data as seed instruction text examples; The seed instruction text example, the target text and the construction instruction are spliced ​​to obtain the first three text sequences, wherein the target text is obtained by converting the size information of the image in the second image-text pair, the original description text corresponding to the image in the second image-text pair, and the detection boxes of all objects in the image in the second image-text pair into text form, the second image-text pair is a second preset number of image-text pairs extracted from the public image-text pair dataset, the original description text is the description text of the second image-text pair in the public image-text pair dataset, and the construction instruction is used to provide target instruction information for the high-performance dialogue model, and the target instruction information is the instruction information required for fine-grained description of the image in the image dataset according to the seed instruction text example and the target text; The third text sequence is input into the high-performance dialogue model to obtain the instruction fine-tuning dataset.

6. The image fine-grained description method for fine-tuning a multimodal large model according to claim 4, wherein: Using the high-performance dialogue model, the fine-grained description text of the image in the instruction fine-tuning dataset is filtered and denoised to obtain the fine-grained description text corresponding to each image, including: The fine-grained description text and post-processing instructions of the images in the instruction fine-tuning dataset are spliced ​​and input into the high-performance dialogue model to obtain the fine-grained description text corresponding to each image, wherein the post-processing instructions are used to guide the high-performance dialogue model to filter and denoise the fine-grained description text of the images in the instruction fine-tuning dataset.

7. The image fine-grained description method for fine-tuning a multimodal large model according to any one of claims 2 to 4, wherein: The method of acquiring the second vector sequence corresponding to the target image includes: The target image is input into a visual encoder in a multimodal large model to obtain the second vector sequence corresponding to the target image, wherein the visual encoder is used to extract high-level semantic information in the target image to obtain the second vector sequence corresponding to the target image.

8. A fine-grained image description system for fine-tuning a multimodal large model, comprising: A data acquisition module, used to acquire a first vector sequence corresponding to a target image, wherein the first vector sequence corresponding to the target image is obtained by encoding a second vector sequence corresponding to the target image, and the second vector sequence corresponding to the target image is obtained based on high-level semantic information extracted from the target image; An image description module is used to obtain, according to the first vector sequence and the first prompt template The fine-grained description text of the target image, wherein the first prompt template is used to provide instruction information required for a fine-grained description of the target image.

9. An electronic device comprising a processor and a memory storing a computer program, wherein: When the processor executes the computer program, it implements the method for fine-tuning the image fine-grained description of a multimodal large model according to any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, wherein: When the computer program is executed by a processor, the method for fine-tuning the image fine-grained description of a multimodal large model according to any one of claims 1 to 7 is implemented.

Images