Method for enhancing fine-grained perception capability of multimodal large model, image processing method and device based on multimodal large model

Abstract

Disclosed are a method for enhancing the fine-grained perception capability of a multimodal large model, and an image processing method and device based on the multimodal large model. The method comprises: obtaining a target detection dataset and an image-text pair dataset; for each image in the two datasets, inputting the current image into a visual encoder to obtain a first visual feature; inputting the first visual feature into an enhanced coding module to obtain a second visual feature; if the current image belongs to the target detection dataset, performing target detection processing on the current image based on the second visual feature to obtain a prediction result; adjusting parameters in the visual encoder and the enhanced coding module based on a detection loss determined by the prediction result and a first true label; if the current image belongs to the image-text pair dataset, inputting the second visual feature and a text question into a large language model to obtain an estimated answer; and adjusting parameters in the visual encoder, the enhanced coding module, and the large language model based on a first generation loss determined by the estimated answer and the second true label.

Description

Technical Field

[0001] The present disclosure generally relates to the field of artificial intelligence, and more specifically, to a method for enhancing the fine-grained perception capability of a multimodal large model, and an image processing method and apparatus based on the multimodal large model. Background Art

[0002] Large multimodal models are a crucial technical foundation for visual question answering and a crucial step towards general artificial intelligence. Visual question answering tasks require large multimodal models to simultaneously process and understand data from both visual (images, videos) and linguistic (question, text description) modalities and generate appropriate responses. Structurally, large multimodal models typically consist of a visual encoder, a connection layer, and a large language model. Their capabilities are primarily reflected in the following aspects: 1) Modal fusion: Through cross-modal attention mechanisms and multimodal alignment techniques, they effectively fuse visual and linguistic features to understand questions and locate relevant areas in images; 2) Language generation: Large multimodal models typically employ autoregressive generation, leveraging the powerful generation capabilities of pre-trained large language models to provide fluent, context-consistent answers to questions.

[0003] However, the pre-trained visual encoders currently used in large multimodal models do not have strong fine-grained visual perception capabilities, resulting in poor performance of large multimodal models on some visual question-answering tasks that require fine-grained visual perception. Summary of the Invention

[0004] The embodiments of the present disclosure provide a method for enhancing the fine-grained perception capability of a large multimodal model, an image processing method and device based on a large multimodal model, which can effectively solve the problem of insufficient fine-grained perception capability of a large multimodal model in the prior art.

[0005] In a general aspect, a method for enhancing the fine-grained perception capability of a multimodal large model is provided, wherein the multimodal large model includes a visual encoder, an enhanced encoding module, and a large language model, and the method includes: obtaining a target detection dataset and a picture-text pair dataset, wherein the target detection dataset includes a plurality of first images and a first true label of each first image, and the picture-text pair dataset includes a plurality of picture-text pairs and a second true label of each picture-text pair, the first true label includes the true category and true location information of the object contained in the corresponding first image, and the second true label includes the true answer to the text question in the corresponding picture-text pair; for each image in the target detection dataset and the picture-text pair dataset, performing the following processing: inputting the current image into the visual encoder, obtaining a first visual feature; inputting the first visual feature into the enhanced coding module to obtain a second visual feature; in response to the current image belonging to the target detection dataset, performing target detection processing on the current image based on the second visual feature to obtain a prediction result corresponding to the current image; adjusting the parameters of the visual encoder and the enhanced coding module based on the detection loss determined by the prediction result and the first true label of the current image; in response to the current image belonging to the image-text pair dataset, inputting the second visual feature and the text question in the image-text pair where the current image is located into the large language model to obtain an estimated answer to the text question; adjusting the parameters of the visual encoder, the enhanced coding module and the large language model based on the first generation loss determined based on the estimated answer and the second true label of the image-text pair where the current image is located.

[0006] Optionally, target detection processing is performed on the current image based on the second visual feature to obtain a prediction result corresponding to the current image, including: performing cross-attention processing on the second visual feature and a preset query vector to obtain a cross-processing vector, wherein the query vector indicates the category and position information of the object contained in the current image queried from the second visual feature; inputting the cross-processing vector into a decoder to obtain a decoded feature; and inputting the decoded feature into a feedforward neural network to obtain a prediction result corresponding to the current image.

[0007] Optionally, when the target detection dataset also includes a text question corresponding to the current image and the first true label also includes a true answer to the text question, before adjusting the parameters of the visual encoder and the enhanced coding module based on the detection loss determined based on the prediction result and the first true label of the current image, it also includes: inputting the second visual feature and the text question corresponding to the current image into the large language model to obtain an estimated answer to the text question corresponding to the current image; determining a second generation loss based on the estimated answer and the true answer to the text question corresponding to the current image; wherein, adjusting the parameters of the visual encoder and the enhanced coding module based on the detection loss determined based on the prediction result and the first true label of the current image includes: adjusting the parameters of the visual encoder, the enhanced coding module and the large language model based on the detection loss and the second generation loss.

[0008] Optionally, before adjusting the parameters in the visual encoder and the enhanced coding module based on the detection loss determined based on the prediction result and the first true label of the current image, the method further includes: for each object contained in the current image, determining the matching relationship between the prediction result of the current object and the true category and true position information of the current object in the first true label of the current image; and determining the detection loss based on the matching relationship of each object and the first true label.

[0009] Optionally, the multimodal large model also includes an image projection layer and a text projection layer, wherein the second visual feature and the text question in the image-text pair where the current image is located are input into the large language model to obtain an estimated answer to the text question, including: inputting the second visual feature into the image projection layer to obtain a third visual feature; inputting the text question in the image-text pair where the current image is located into the text projection layer to obtain a first text feature; splicing the third visual feature and the first text feature to obtain a first splicing result; inputting the first splicing result into the large language model to obtain an estimated answer to the text question in the image-text pair where the current image is located.

[0010] In another general aspect, an image processing method based on a multimodal large model is provided, wherein the multimodal large model includes a visual encoder, an enhanced coding module and a large language model, and the image processing method includes: inputting the image to be processed into the visual encoder to obtain a first visual feature; inputting the first visual feature into the enhanced coding module to obtain a second visual feature; inputting the second visual feature and a text question in the image-text pair in which the image to be processed is located into the large language model to obtain an answer to the text question, wherein the parameters of the visual encoder, the enhanced coding module and the large language model in the multimodal large model are obtained by the above method.

[0011] In another general aspect, a device for enhancing the fine-grained perception capability of a multimodal large model is provided, the multimodal large model including a visual encoder, an enhanced encoding module and a large language model, the device including: an acquisition unit, configured to acquire a target detection dataset and a picture-text pair dataset, wherein the target detection dataset includes a plurality of first images and a first true label of each first image, the picture-text pair dataset includes a plurality of picture-text pairs and a second true label of each picture-text pair, the first true label includes the true category and true location information of the object contained in the corresponding first image, and the second true label includes the true answer to the text question in the corresponding picture-text pair; an execution unit, configured to perform the following processing for each image in the target detection dataset and the picture-text pair dataset: input the current image into Input the visual encoder to obtain a first visual feature; input the first visual feature into the enhanced coding module to obtain a second visual feature; in response to the current image belonging to the target detection dataset, perform target detection processing on the current image based on the second visual feature to obtain a prediction result corresponding to the current image; adjust the parameters of the visual encoder and the enhanced coding module based on the detection loss determined by the prediction result and the first true label of the current image; in response to the current image belonging to the image-text pair dataset, input the second visual feature and the text question in the image-text pair where the current image is located into the large language model to obtain an estimated answer to the text question; adjust the parameters of the visual encoder, the enhanced coding module and the large language model based on the first generation loss determined by the estimated answer and the second true label of the image-text pair where the current image is located.

[0012] Optionally, the execution unit is further configured to perform cross-attention processing on the second visual feature and a preset query vector to obtain a cross-processed vector, wherein the query vector indicates the category and position information of the object contained in the current image queried from the second visual feature; input the cross-processed vector into the decoder to obtain a decoded feature; and input the decoded feature into the feedforward neural network to obtain a prediction result corresponding to the current image.

[0013] Optionally, the execution unit is further configured to, when the target detection dataset also includes a text question corresponding to the current image and the first true label also includes a true answer to the text question, input the second visual feature and the text question corresponding to the current image into the large language model before adjusting the parameters of the visual encoder and the enhanced coding module based on the detection loss determined based on the prediction result and the first true label of the current image, to obtain an estimated answer to the text question corresponding to the current image; determine a second generation loss based on the estimated answer and the true answer to the text question corresponding to the current image; and adjust the parameters of the visual encoder, the enhanced coding module and the large language model based on the detection loss and the second generation loss.

[0014] Optionally, the execution unit is further configured to determine, for each object contained in the current image, a matching relationship between the prediction result of the current object and the true category and true position information of the current object in the first true label of the current image before adjusting the parameters of the visual encoder and the enhanced coding module based on the detection loss determined based on the prediction result and the first true label of the current image; and determine the detection loss based on the matching relationship and the first true label of each object.

[0015] Optionally, the multimodal large model also includes an image projection layer and a text projection layer, wherein the execution unit is further configured to input the second visual feature into the image projection layer to obtain a third visual feature; input the text question in the image-text pair where the current image is located into the text projection layer to obtain a first text feature; splice the third visual feature and the first text feature to obtain a first splicing result; and input the first splicing result into the large language model to obtain an estimated answer to the text question in the image-text pair where the current image is located.

[0016] In another general aspect, an image processing device based on a multimodal large model is provided, wherein the multimodal large model includes a visual encoder, an enhanced coding module and a large language model, and the device includes: a first acquisition unit, configured to input the image to be processed into the visual encoder to obtain a first visual feature; a second acquisition unit, configured to input the first visual feature into the enhanced coding module to obtain a second visual feature; and a third acquisition unit, configured to input the second visual feature and a text question in the image-text pair in which the image to be processed is located into the large language model to obtain an answer to the text question, wherein the parameters of the visual encoder, the enhanced coding module and the large language model in the multimodal large model are obtained by any of the above methods.

[0017] In another general aspect, a computer-readable storage medium storing instructions is provided, wherein, when the instructions are executed by at least one computing device, the at least one computing device is prompted to perform any of the above-mentioned methods for enhancing the fine-grained perception capability of a multimodal large model or image processing methods based on a multimodal large model.

[0018] In another general aspect, a system is provided comprising at least one computing device and at least one storage device storing instructions, wherein the instructions, when executed by the at least one computing device, cause the at least one computing device to execute any of the above-mentioned methods for enhancing the fine-grained perception capability of a multimodal large model or image processing methods based on a multimodal large model.

[0019] In another general aspect, a computer program product is provided, comprising computer instructions, which, when executed by a processor, implement any of the above-mentioned methods for enhancing the fine-grained perception capability of a multimodal large model or image processing methods based on a multimodal large model.

[0020] According to the method for enhancing the fine-grained perception capability of a multimodal large model, and the image processing method and device based on the multimodal large model according to the embodiments of the present disclosure, on the basis of the original generation task training, a target detection data set is introduced, and a target detection process for the image is added, so that the category and position information of the object contained in the image can be predicted, and based on the detection loss determined by this part of the prediction results and its true category and true position information, the parameters of the visual encoder and the enhanced coding module are updated to enhance the fine-grained visual perception capability of the visual features in the encoding process. That is, the present disclosure additionally optimizes the target detection task on the basis of the generation task, and enhances the fine-grained perception capability of the visual features encoded by the multimodal large model through the target detection task, which is a visual downstream task, thereby obtaining a multimodal large model with better generation effect. Therefore, through the present disclosure, the problem of insufficient fine-grained perception capability of the multimodal large model in the prior art can be effectively solved.

[0021] Additional aspects and / or advantages of the present general inventive concept will be set forth in part in the following description and in part will be apparent from the description, or may be learned through practice of the present general inventive concept. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The above and other objects and features of the embodiments of the present disclosure will become more apparent through the following description in conjunction with the accompanying drawings showing the embodiments, in which:

[0023] Figure 1 is a flow chart illustrating a method for enhancing the fine-grained perception capability of a multimodal large model according to an embodiment of the present disclosure;

[0024] Figure 2 is an architectural diagram illustrating an embodiment of the present disclosure for enhancing the fine-grained perception capability of a multimodal large model;

[0025] Figure 3 is a general flow chart illustrating an embodiment of the present disclosure for enhancing the fine-grained perception capability of a multimodal large model;

[0026] Figure 4 is a flowchart illustrating an image processing method based on a multimodal large model according to an embodiment of the present disclosure;

[0027] Figure 5 is a block diagram illustrating an apparatus for enhancing the fine-grained perception capability of a multimodal large model according to an embodiment of the present disclosure;

[0028] Figure 6 2 is a block diagram illustrating an image processing apparatus based on a multimodal large model according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0029] The following detailed description is provided to help the reader gain a comprehensive understanding of the methods, devices and / or systems described herein. However, various changes, modifications and equivalents of the methods, devices and / or systems described herein will be clear after understanding the disclosure of the present application. For example, the order of operations described herein is merely an example and is not limited to those orders set forth herein, but can be changed as will be clear after understanding the disclosure of the present application, except for operations that must occur in a specific order. In addition, for greater clarity and conciseness, descriptions of features known in the art may be omitted.

[0030] The features described herein can be implemented in different forms and should not be construed as limited to the examples described herein. Rather, the examples described herein are provided to illustrate only some of the many possible ways to implement the methods, devices, and / or systems described herein, which will become clear after understanding the disclosure of this application.

[0031] As used herein, the term "and / or" includes any one of the associated listed items and any combination of any two or more.

[0032] Although terms such as "first," "second," and "third" may be used herein to describe various members, components, regions, layers, or portions, these members, components, regions, layers, or portions should not be limited by these terms. Instead, these terms are used solely to distinguish one member, component, region, layer, or portion from another member, component, region, layer, or portion. Thus, what is referred to as a first member, first component, first region, first layer, or first portion in the examples described herein may also be referred to as a second member, second component, second region, second layer, or second portion without departing from the teachings of the examples.

[0033] In the specification, when an element (such as a layer, region, or substrate) is described as being “on,” “connected to,” or “coupled to” another element, the element may be directly “on,” “connected to,” or “coupled to” the other element, or one or more other elements may be present therebetween. Conversely, when an element is described as being “directly on,” “directly connected to,” or “directly coupled to” another element, there may be no other elements present therebetween.

[0034] The terms used herein are only used to describe various examples and are not intended to limit the disclosure. Unless the context clearly indicates otherwise, the singular is intended to include the plural. The terms "comprise," "include," and "have" indicate the presence of the recited features, quantities, operations, components, elements, and / or combinations thereof, but do not preclude the presence or addition of one or more other features, quantities, operations, components, elements, and / or combinations thereof.

[0035] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present disclosure pertains after understanding the present disclosure. Unless expressly defined as such herein, terms (such as those defined in commonly used dictionaries) should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and the present disclosure, and should not be interpreted in an idealized or overly formal manner.

[0036] Furthermore, in describing the examples, when it is deemed that a detailed description of well-known related structures or functions would cause ambiguous interpretation of the present disclosure, such detailed description will be omitted.

[0037] During the fine-tuning phase of visual instructions for large multimodal models, the model is typically configured for causal language modeling. The input sequence (visual features + question text) is concatenated into a continuous sequence. Through autoregressive generative tasks, the next token is gradually predicted, ultimately generating a complete answer. This approach's ability to perceive fine-grained visual features is limited by the large multimodal model's visual encoder. Recent research has shown that visual encoders commonly used in large multimodal models, such as clip, have inherent limitations in fine-grained perception due to their training paradigm, resulting in weak fine-grained visual perception capabilities in large multimodal models.

[0038] The present disclosure provides a fine-grained perception enhancement method for the visual features of a large multimodal model, which can improve the fine-grained perception ability of the visual encoder during the visual instruction fine-tuning stage, thereby more accurately answering questions that require fine-grained visual perception in visual question-answering tasks. Specifically, an enhanced encoding module is added after the visual encoder and a target detection branch is added when training the large multimodal model. That is, in the visual instruction fine-tuning stage, an additional target detection task is introduced on the basis of the autoregressive generation task, and the fine-grained visual perception ability of the large multimodal model during encoding is enhanced through the target detection task, a visual downstream task.

[0039] The method for enhancing the fine-grained perception capability of a multimodal large model, and the image processing method and apparatus based on the multimodal large model disclosed in the present invention are described in detail below with reference to the accompanying drawings.

[0040] This paper proposes a method to enhance the fine-grained perception capability of multimodal large models. Figure 1 1 is a flow chart illustrating a method for enhancing the fine-grained perception capability of a multimodal large model according to an embodiment of the present disclosure. Figure 1 The multimodal large model includes a visual encoder, an enhanced encoding module, and a large language model. The method for enhancing the fine-grained perception capability of the multimodal large model includes the following steps:

[0041] In step S101, a target detection dataset and a picture-text pair dataset are obtained, wherein the target detection dataset includes multiple first images and a first true label for each first image, and the picture-text pair dataset includes multiple picture-text pairs and a second true label for each picture-text pair, the first true label includes the true category and true location information of the object contained in the corresponding first image, and the second true label includes the true answer to the text question in the corresponding picture-text pair.

[0042] As an example, the first image included in the above object detection dataset only needs to contain an object and the corresponding object's true category and true location information, and there is no requirement for the first image to correspond to a text question. Suppose the image is an image of two playful puppies. The true category of the object contained in the image is dog, and the true location is the actual position of the two dogs in the image. For example, the two dogs can be framed separately, and the location of each frame is the true location of each dog. It should be noted that an image can contain both a dog and a cow, which means that the image contains two objects, and the true category of each object is dog and cow respectively.

[0043] As an example, the aforementioned image-text pair dataset contains image-text pairs, including images and corresponding text questions, for training generative tasks. It should be noted that the images in the image-text pair dataset are generally not suitable for object detection. This prevents object detection samples from appearing in the image-text pair dataset, which could result in duplicate samples in both datasets and lead to redundant computation.

[0044] In step S102, for each image in the target detection dataset and the image-text pair dataset, the following processing is performed: the current image is input into the visual encoder to obtain a first visual feature; the first visual feature is input into the enhanced coding module to obtain a second visual feature; in response to the current image belonging to the target detection dataset, target detection processing is performed on the current image based on the second visual feature to obtain a prediction result corresponding to the current image; based on the detection loss determined by the prediction result and the first true label of the current image, the parameters of the visual encoder and the enhanced coding module are adjusted; in response to the current image belonging to the image-text pair dataset, the second visual feature and the text question in the image-text pair where the current image is located are input into the large language model to obtain an estimated answer to the text question; based on the estimated answer and the first generation loss determined by the second true label of the image-text pair where the current image is located, the parameters of the visual encoder, the enhanced coding module and the large language model are adjusted.

[0045] As an example, the above processing can be performed on the images in the target detection dataset first, and after the processing of the images in the target detection dataset is completed, the above processing can be performed on the images in the image-text pair dataset; the above processing can also be performed on the images in the image-text pair dataset first, and after the processing of the images in the image-text pair dataset is completed, the above processing can be performed on the images in the target detection dataset; and after the processing of some images in the target detection dataset is completed, such as after the processing of a batch of images in the target detection dataset is completed, the above processing can be performed on a batch of images in the image-text pair dataset, and then the next batch of images in the target detection dataset can be processed, and then the above processing can be performed on the next batch of images in the image-text pair dataset, until the data in both the target detection dataset and the image-text pair dataset are processed.

[0046] As an example, for each image in the object detection dataset and the image-text pair dataset, it can first be input into the visual encoder to obtain the first visual feature (also known as the visual token). The first visual feature is then input into the enhanced encoding module to re-encode the first visual feature to further enhance the visual fine-grained perception and obtain the second visual feature. After obtaining the second visual feature, subsequent processing needs to determine which dataset the current image belongs to. Specifically:

[0047] If the current image belongs to the object detection dataset, target detection processing is performed on the current image to obtain a prediction result corresponding to the current image, where the prediction result includes the predicted category and predicted location information of the object in the current image. Then, a detection loss is determined based on the predicted category, predicted location information, and the true category and true location information in the first true label of the current image. The parameters of the visual encoder and the enhanced coding module are adjusted based on the detection loss.

[0048] If the current image belongs to an image-text pair dataset, the second visual feature and the text question in the image-text pair in which the current image is located are input into the large language model to obtain an estimated answer to the text question. Then, a first generation loss is determined based on the estimated answer and the true answer in the second true label of the image-text pair in which the current image is located, and the parameters of the visual encoder, the enhanced encoding module, and the large language model are adjusted using the first generation loss.

[0049] After the above processing is performed on all images in the target detection dataset and the image-text dataset, a trained multimodal large model is obtained.

[0050] It should be noted that, in order to adapt to the output of the visual encoder of the multimodal large model, the target detection branch of the present disclosure can adopt an end-to-end target detection framework composed of pure transformers, and the present disclosure does not limit this. The enhanced coding module can be an additional coding layer in the transformer encoder compared to the original visual coding of the multimodal large model. Here, only the transformer encoder can be connected, but only the additional coding layer in the transformer encoder compared to the original visual coding of the multimodal large model is used. The visual encoder and the enhanced coding module can also be replaced with a transformer encoder, and the present disclosure does not limit this.

[0051] The object detection branch of this disclosure utilizes the visual encoder of a large multimodal model as the backbone network, with the integrated transformer encoder serving as additional training parameters. This re-encodes the visual tokens transmitted by the encoder to inject fine-grained perceptual information. The following example illustrates this approach using a pure transformer-based end-to-end object detection framework.

[0052] According to an embodiment of the present disclosure, target detection processing can be performed on the current image based on the second visual feature to obtain a prediction result corresponding to the current image by: performing cross-attention processing on the second visual feature and a preset query vector to obtain a cross-processed vector, wherein the query vector indicates the category and location information of the object contained in the current image queried from the second visual feature; inputting the cross-processed vector into a decoder to obtain a decoded feature; and inputting the decoded feature into a feedforward neural network to obtain a prediction result corresponding to the current image. Through this embodiment, target detection processing can be performed quickly and accurately.

[0053] As an example, the above-mentioned query vector can be pre-set and initialized as needed, and the feature information of the object (i.e., the above-mentioned cross-processing vector) can be obtained from the second visual feature through the query vector. For example, the preset query vector can indicate the category and position information of the object contained in the current image queried from the second visual feature, and this disclosure is not limited to this.

[0054] As an example, object detection can be achieved through a decoder, a feedforward neural network with the previous visual encoder, and an enhanced coding model.

[0055] Specifically, assuming that the current image belongs to the target detection dataset, cross-attention processing is performed on the second visual feature and the preset query vector to obtain a cross-processed vector. Then, the cross-processed vector is input into the decoder to obtain a decoded feature, and then the decoded feature is input into the feedforward neural network to obtain the prediction result corresponding to the current image, that is, the predicted category and predicted position information of the object contained in the current image (such as the bounding box used to frame the object).

[0056] It should be noted that if the image does not contain the required object, the feedforward neural network can input information that there is no object, such as the object category is "no object", which is not limited in this disclosure.

[0057] According to an embodiment of the present disclosure, before adjusting the parameters of the visual encoder and the enhanced coding module based on the detection loss determined based on the prediction result and the first true label of the current image, the detection loss can be determined. Specifically, for each object contained in the current image, the matching relationship between the prediction result of the current object and the true category and true position information of the current object in the first true label of the current image is determined; based on the matching relationship of each object and the first true label, the detection loss is determined.

[0058] Through this embodiment, by utilizing the matching relationship between the prediction results and the real categories and real position information in the real labels, a detection loss that is more consistent with reality can be determined, thereby enabling better training of multimodal large models.

[0059] As an example, the Hungarian matching cost function can be used to calculate the matching cost between the predicted result and the true label. The matching cost of each object in the current image can be calculated by the following formula:

[0060] )

[0061] in, ( ) indicates the first i The true labels of the objects, including the categories of the objects and bounding box (i.e. location information); ( ) indicates the first i The prediction results of the objects, including the predicted categories and predicted bounding boxes ; Represents the classification loss, which is used to evaluate the difference between the predicted category and the true category; represents the L1 loss, which is used to evaluate the coordinate difference between the predicted bounding box and the true bounding box; represents the GIoU loss, which is used to evaluate the overlap quality between the predicted bounding box and the true bounding box; and Represents the loss weight, which is used to balance the contribution of L1 and GIoU losses.

[0062] For each object in the current image, the best matching relationship between the predicted result and the true label can be found by minimizing the corresponding Hungarian matching cost function. Then, the detection loss is determined based on the matching relationship of each object. The detection loss is obtained based on the Hungarian matching result and can be as follows:

[0063]

[0064] in, It represents the best matching relationship between the prediction result and the true label found by using the Hungarian matching cost function, and N represents the number of object categories contained in the current image.

[0065] According to an embodiment of the present disclosure, the multimodal large model also includes an image projection layer and a text projection layer, wherein the second visual feature and the text question in the image-text pair where the current image is located are input into the large language model to obtain an estimated answer to the text question, including: inputting the second visual feature into the image projection layer to obtain a third visual feature; inputting the text question in the image-text pair where the current image is located into the text projection layer to obtain a first text feature; splicing the third visual feature and the first text feature to obtain a first splicing result; inputting the first splicing result into the large language model to obtain an estimated answer to the text question in the image-text pair where the current image is located.

[0066] Through this embodiment, the original training path of the multimodal large model can be completed conveniently and quickly.

[0067] As an example, this embodiment is the original training path of the multimodal large model, that is, training for the image-text pair dataset, with the purpose of aligning visual features and text features and generating text. Specifically, after obtaining the second visual feature of the image in the image-text pair, the text question in the image-text pair is input into the tokenizer to obtain the segmentation result. Then, the second visual feature and the segmentation result are mapped to the same dimension through the image projection layer and the text projection layer respectively, that is, the third visual feature and the first text feature of the same dimension are obtained. After the third visual feature and the first text feature are spliced, they are input into the large language model to obtain the corresponding estimated answer, and autoregressive training is performed based on the estimated answer and the true answer.

[0068] As an example, the generation loss in autoregressive training is as follows:

[0069]

[0070] in, T Indicates the total length of the generated sequence (i.e., the estimated answer); The actual token representing time step t; represents the generated token sequence before time step t; xrepresents the result of concatenating the third visual feature and the first text feature; Indicates that the multimodal large model is under the condition x and the generated token sequence Next prediction probability.

[0071] After obtaining the first generation loss through the above generation loss function, the parameters of the visual encoder, enhanced encoding module and large language model can be adjusted based on the first generation loss to achieve autoregressive training.

[0072] According to an embodiment of the present disclosure, when the target detection dataset also includes a text question corresponding to the current image and the first true label also includes a true answer to the text question, before adjusting the parameters of the visual encoder and the enhanced coding module based on the detection loss determined based on the prediction result and the first true label of the current image, the second visual feature and the text question corresponding to the current image are input into the large language model to obtain an estimated answer to the text question corresponding to the current image; a second generation loss is determined based on the estimated answer and the true answer to the text question corresponding to the current image; wherein, adjusting the parameters in the visual encoder and the enhanced coding module based on the detection loss determined based on the prediction result and the first true label of the current image includes: adjusting the parameters of the visual encoder, the enhanced coding module and the large language model based on the detection loss and the second generation loss.

[0073] Through this embodiment, when some first images in the target detection dataset correspond to text questions, these first images can also go through the generation task, thereby enhancing the training of the large language model.

[0074] As an example, when some first images in the target detection dataset correspond to text questions, the second visual features and the text questions corresponding to the first image can also be input into the large language model to obtain an estimated answer to the text question corresponding to the first image. Then, based on the estimated answer and the true answer corresponding to the first image, the second generation loss is determined, and then the total loss is obtained based on the second generation loss and the detection loss of the first image, and then the parameters of the visual encoder, enhanced coding module and large language model are adjusted through the total loss.

[0075] It should be noted that the above-mentioned total loss may be the sum of the second generation loss and the detection loss, which is not limited in the present disclosure.

[0076] In order to better understand the present disclosure, Figure 2 and Figure 3 Provide a description of the system.

[0077] Figure 2 The architecture diagram shows how to enhance the fine-grained perception capabilities of large multimodal models. Figure 3A flowchart is shown for enhancing the fine-grained perception capability of a large multimodal model. Figure 2 and Figure 3 As shown, the process includes the following parts:

[0078] First, obtain the target detection dataset and the image-text pair dataset, and obtain the current image and the true label (first true label or second true label) corresponding to the current image from the target detection dataset or the image-text pair dataset;

[0079] Second, the current image ( Figure 2 The image contains two puppies) is input into the visual encoder to obtain the first visual feature;

[0080] Again, the first visual feature is input into the enhanced coding module to obtain a second visual feature;

[0081] Again, if the current image is an image in the target detection dataset, the second visual feature and the preset query vector are cross-attention processed to obtain a cross-processing result (that is, the above-mentioned cross-processing vector). The cross-processing result is the feature information of the puppy in the image obtained from the second visual feature based on the query vector, and then the cross-processing vector is input into the Transformer decoder to obtain the decoded feature; the decoded feature is input into the feedforward neural network (FFN) to obtain the prediction result corresponding to the current image, which includes the category (CLASS) - dog and the bounding box (BOX) of the two dogs; the prediction result is matched one-to-one with the true label, and the Hungarian matching cost function is used to calculate the matching cost between the prediction result and the true label, and the detection loss of the current image is determined based on the matching cost; it should be noted that, considering that an image may contain multiple objects, multiple FFNs are generally set, and one FFN corresponds to the detection of one object. If there is no corresponding object, the corresponding FFN outputs no object (NO OBEJECT);

[0082] Again, if the current image is an image in the image-text pair dataset, the second visual feature is input into the image projection layer to obtain the third visual feature, and the text question in the image-text pair where the current image is located is input into the word segmenter to obtain the word segmentation result, and the word segmentation result is input into the text projection layer to obtain the text feature. Then, the third visual feature and the text feature are spliced ​​and input into the large language model to obtain the estimated answer to the text question. The estimated answer is that there are two dogs in the picture, and their positions in the picture are<x1,y1,x2,y2> ,<x3,y3,x4,y4> ;Determine the first generation loss based on the estimated answer and the true answer;

[0083] Finally, if the current image is an image in the target detection dataset and the current image does not have a corresponding text question, the parameters of the visual encoder and the enhanced coding module are adjusted based on the above-mentioned detection loss; if the current image is an image in the target detection dataset and the current image has a corresponding text question, the second visual feature is input into the image projection layer to obtain the third visual feature, and the text question in the image-text pair where the current image is located is input into the word segmenter to obtain the word segmentation result, and the word segmentation result is input into the text projection layer to obtain the text feature. Then, the third visual feature and the text feature are spliced ​​and input into the large language model to obtain the estimated answer to the text question; the second generation loss is determined based on the estimated answer and the true answer, and the parameters of the visual encoder, enhanced coding module and large language model are adjusted through the detection loss and the second generation loss; if the current image is an image in the image-text pair dataset, the parameters of the visual encoder, enhanced coding module and large language model are adjusted through the first generation loss; until all samples in the target detection dataset and the image-text pair dataset are processed.

[0084] This disclosure also proposes an image processing method based on a multimodal large model. Figure 4 : is a flowchart illustrating an image processing method based on a multimodal large model according to an embodiment of the present disclosure. Figure 4 The multimodal large model includes a visual encoder, an enhanced coding module and a large language model. The image processing method based on the multimodal large model includes the following steps:

[0085] In step S401, the image to be processed is input into a visual encoder to obtain a first visual feature;

[0086] In step S402, the first visual feature is input into an enhanced coding module to obtain a second visual feature;

[0087] In step S403, the second visual feature and the text question in the image-text pair containing the image to be processed are input into the large language model to obtain an answer to the text question.

[0088] The parameters of the visual encoder, enhanced encoding module and large language model in the multimodal large model described above are obtained by the method described above.

[0089] In summary, the present disclosure belongs to the field of multimodal artificial intelligence, and is based on visual question-answering tasks, especially for the part of visual question-answering tasks that requires fine-grained perception. The specific purpose is to enhance the visual fine-grained perception ability of the multimodal large model, so as to solve the problem that the multimodal large model performs poorly for visual question-answering tasks that require fine-grained visual perception. Specifically, an enhanced coding module is added after the visual encoder and a target detection branch is added when training the multimodal large model, that is, in the visual instruction fine-tuning stage, an additional target detection task is introduced on the basis of the autoregressive generation task, and the fine-grained perception ability of the multimodal large model during visual encoding is enhanced through the visual downstream task of the target detection task. In the inference settlement after the training is completed, the target detection branch can be omitted, and only the enhanced coding module used to inject fine-grained visual perception ability and the original multimodal large model structure are retained.

[0090] Figure 5 is a block diagram illustrating an apparatus for enhancing the fine-grained perception capability of a multimodal large model according to an embodiment of the present disclosure, such as Figure 5 As shown, the multimodal large model includes a visual encoder, an enhanced encoding module and a large language model, and the device includes: an acquisition unit 50 and an execution unit 52.

[0091] The acquisition unit 50 is configured to acquire a target detection dataset and a picture-text pair dataset, wherein the target detection dataset includes a plurality of first images and a first true label of each first image, and the picture-text pair dataset includes a plurality of picture-text pairs and a second true label of each picture-text pair, the first true label includes the true category and true location information of the object contained in the corresponding first image, and the second true label includes the true answer to the text question in the corresponding picture-text pair; the execution unit 52 is configured to perform the following processing on each image in the target detection dataset and the picture-text pair dataset: input the current image into the visual encoder to obtain the first visual feature; input the first visual feature into the enhanced coding model block, obtains a second visual feature; in response to the current image belonging to the target detection dataset, performs target detection processing on the current image based on the second visual feature to obtain a prediction result corresponding to the current image; based on the detection loss determined by the prediction result and the first true label of the current image, adjusts the parameters of the visual encoder and the enhanced coding module; in response to the current image belonging to the image-text pair dataset, inputs the second visual feature and the text question in the image-text pair where the current image is located into the large language model to obtain an estimated answer to the text question; based on the estimated answer and the first generation loss determined by the second true label of the image-text pair where the current image is located, adjusts the parameters of the visual encoder, the enhanced coding module and the large language model.

[0092] According to an embodiment of the present disclosure, the execution unit 52 is further configured to perform cross-attention processing on the second visual feature and a preset query vector to obtain a cross-processed vector, wherein the query vector indicates the category and position information of the object contained in the current image queried from the second visual feature; input the cross-processed vector into the decoder to obtain a decoded feature; and input the decoded feature into a feedforward neural network to obtain a prediction result corresponding to the current image.

[0093] According to an embodiment of the present disclosure, the execution unit 52 is further configured to, when the target detection dataset also includes a text question corresponding to the current image and the first true label also includes a true answer to the text question, input the second visual feature and the text question corresponding to the current image into the large language model before adjusting the parameters of the visual encoder and the enhanced coding module based on the detection loss determined based on the prediction result and the first true label of the current image, to obtain an estimated answer to the text question corresponding to the current image; determine a second generation loss based on the estimated answer and the true answer to the text question corresponding to the current image; and adjust the parameters of the visual encoder, the enhanced coding module and the large language model based on the detection loss and the second generation loss.

[0094] According to an embodiment of the present disclosure, the execution unit 52 is further configured to determine, for each object contained in the current image, a matching relationship between the prediction result of the current object and the true category and true position information of the current object in the first true label of the current image before adjusting the parameters in the visual encoder and the enhanced coding module based on the detection loss determined based on the prediction result and the first true label of the current image; and determine the detection loss based on the matching relationship and the first true label of each object.

[0095] According to an embodiment of the present disclosure, the multimodal large model also includes an image projection layer and a text projection layer, wherein the execution unit 52 is further configured to input the second visual feature into the image projection layer to obtain a third visual feature; input the text question in the image-text pair where the current image is located into the text projection layer to obtain a first text feature; splice the third visual feature and the first text feature to obtain a first splicing result; and input the first splicing result into the large language model to obtain an estimated answer to the text question in the image-text pair where the current image is located.

[0096] Figure 6 is a block diagram illustrating an image processing apparatus based on a multimodal large model according to an embodiment of the present disclosure, Figure 6 As shown, the multimodal large model includes a visual encoder, an enhanced encoding module and a large language model, and the device includes a first acquisition unit 60, a second acquisition unit 62 and a third acquisition unit 64.

[0097] The first acquisition unit 60 is configured to input the image to be processed into a visual encoder to obtain a first visual feature; the second acquisition unit 62 is configured to input the first visual feature into an enhanced coding module to obtain a second visual feature; the third acquisition unit 64 is configured to input the second visual feature and the text question in the image-text pair containing the image to be processed into the large language model to obtain an answer to the text question, wherein the parameters of the visual encoder, enhanced coding module and large language model in the multimodal large model are obtained by the method described above.

[0098] According to an embodiment of the present disclosure, a computer-readable storage medium storing instructions is provided, wherein, when the instructions are executed by at least one computing device, the at least one computing device is prompted to execute a method for enhancing the fine-grained perception capability of a multimodal large model or an image processing method based on a multimodal large model as described in any of the above embodiments.

[0099] According to an embodiment of the present disclosure, a system is provided comprising at least one computing device and at least one storage device storing instructions, wherein the instructions, when executed by the at least one computing device, prompt the at least one computing device to execute a method for enhancing the fine-grained perception capability of a multimodal large model or an image processing method based on a multimodal large model as described in any of the above embodiments.

[0100] In another general aspect, a computer program product is provided, comprising computer instructions, which, when executed by a processor, implement a method for enhancing the fine-grained perception capability of a multimodal large model or an image processing method based on a multimodal large model as described in any of the above embodiments.

[0101] While some embodiments of the present disclosure have been shown and described, it will be appreciated by those skilled in the art that changes may be made to these embodiments without departing from the principles and spirit of the disclosure, the scope of which is defined by the claims and their equivalents.

Claims

1. A method for enhancing the fine-grained perception capability of a large multimodal model, characterized in that: The multimodal large model includes a visual encoder, an enhanced encoding module and a large language model, and the method includes: Obtain an object detection dataset and an image-text pair dataset, wherein the object detection dataset includes multiple first images and a first true label for each first image, and the image-text pair dataset includes multiple image-text pairs and a second true label for each image-text pair, wherein the first true label includes the true category and true location information of the object contained in the corresponding first image, and the second true label includes the true answer to the text question in the corresponding image-text pair; For each image in the object detection dataset and the image-text pair dataset, perform the following processing: Inputting the current image into the visual encoder to obtain a first visual feature; Inputting the first visual feature into the enhanced coding module to obtain a second visual feature; In response to the current image belonging to the object detection dataset, performing object detection processing on the current image based on the second visual feature to obtain a prediction result corresponding to the current image; and adjusting parameters of the visual encoder and the enhanced coding module based on the prediction result and a detection loss determined based on a first true label of the current image; In response to the current image belonging to the image-text pair dataset, inputting the second visual feature and a text question in the image-text pair in which the current image is located into the large language model to obtain an estimated answer to the text question; adjusting parameters of the visual encoder, the enhanced encoding module, and the large language model based on a first generation loss determined based on the estimated answer and a second true label of the image-text pair in which the current image is located; Wherein, when the object detection dataset further includes a text question corresponding to the current image and the first true label further includes a true answer to the text question, before adjusting the parameters of the visual encoder and the enhanced coding module based on the detection loss determined based on the prediction result and the first true label of the current image, the method further includes: Inputting the second visual feature and the text question corresponding to the current image into a large language model to obtain an estimated answer to the text question corresponding to the current image; and determining a second generation loss based on the estimated answer and the true answer to the text question corresponding to the current image; The step of adjusting parameters of the visual encoder and the enhanced coding module based on the prediction result and the detection loss determined by the first true label of the current image comprises: Parameters of the visual encoder, the enhanced encoding module, and the large language model are adjusted based on the detection loss and the second generation loss.

2. The method according to claim 1, wherein The performing target detection processing on the current image based on the second visual feature to obtain a prediction result corresponding to the current image includes: performing cross-attention processing on the second visual feature and a preset query vector to obtain a cross-processed vector, wherein the query vector indicates category and position information of an object contained in the current image queried from the second visual feature; Inputting the cross-processed vector into a decoder to obtain a decoding feature; The decoded features are input into a feedforward neural network to obtain a prediction result corresponding to the current image.

3. The method according to claim 1, wherein Before adjusting the parameters of the visual encoder and the enhanced coding module based on the detection loss determined based on the prediction result and the first true label of the current image, the method further includes: For each object included in the current image, determining a matching relationship between a prediction result of the current object and the true category and true location information of the current object in the first true label of the current image; A detection loss is determined based on the matching relationship of each object and the first true label.

4. The method according to claim 1, wherein The multimodal large model also includes an image projection layer and a text projection layer. The step of inputting the second visual feature and the text question in the image-text pair containing the current image into a large language model to obtain an estimated answer to the text question includes: inputting the second visual feature into the image projection layer to obtain a third visual feature; Inputting a text question in the image-text pair where the current image is located into the text projection layer to obtain a first text feature; splicing the third visual feature and the first text feature to obtain a first splicing result; The first splicing result is input into a large language model to obtain an estimated answer to the text question in the image-text pair where the current image is located.

5. An image processing method based on a multimodal large model, characterized in that: The multimodal large model includes a visual encoder, an enhanced encoding module and a large language model, and the method includes: Inputting the image to be processed into the visual encoder to obtain a first visual feature; Inputting the first visual feature into the enhanced coding module to obtain a second visual feature; Inputting the second visual feature and the text question in the image-text pair containing the image to be processed into the large language model to obtain an answer to the text question, The parameters of the visual encoder, the enhanced coding module and the large language model in the multimodal large model are obtained by the method according to any one of claims 1 to 4.

6. A device for enhancing the fine-grained perception capability of a large multimodal model, characterized in that: The multimodal large model includes a visual encoder, an enhanced coding module and a large language model, and the device includes: an acquisition unit configured to acquire an object detection dataset and an image-text pair dataset, wherein the object detection dataset includes a plurality of first images and a first true label for each first image, and the image-text pair dataset includes a plurality of image-text pairs and a second true label for each image-text pair, wherein the first true label includes true category and true location information of an object contained in the corresponding first image, and the second true label includes a true answer to a text question in the corresponding image-text pair; An execution unit is configured to perform the following processing on each image in the object detection dataset and the image-text pair dataset: Inputting the current image into the visual encoder to obtain a first visual feature; Inputting the first visual feature into the enhanced coding module to obtain a second visual feature; In response to the current image belonging to the object detection dataset, performing object detection processing on the current image based on the second visual feature to obtain a prediction result corresponding to the current image; and adjusting parameters of the visual encoder and the enhanced coding module based on the prediction result and a detection loss determined based on a first true label of the current image; In response to the current image belonging to the image-text pair dataset, inputting the second visual feature and a text question in the image-text pair in which the current image is located into the large language model to obtain an estimated answer to the text question; adjusting parameters of the visual encoder, the enhanced encoding module, and the large language model based on a first generation loss determined based on the estimated answer and a second true label of the image-text pair in which the current image is located; The execution unit is further configured to, when the target detection dataset also includes a text question corresponding to the current image and the first true label also includes a true answer to the text question, input the second visual feature and the text question corresponding to the current image into the large language model before adjusting the parameters of the visual encoder and the enhanced coding module based on the detection loss determined based on the prediction result and the first true label of the current image, to obtain an estimated answer to the text question corresponding to the current image; determine a second generation loss based on the estimated answer and the true answer to the text question corresponding to the current image; and adjust the parameters of the visual encoder, the enhanced coding module, and the large language model based on the detection loss and the second generation loss.

7. An image processing device based on a multimodal large model, characterized in that: The multimodal large model includes a visual encoder, an enhanced coding module and a large language model, and the device includes: A first acquisition unit is configured to input the image to be processed into the visual encoder to obtain a first visual feature; a second acquiring unit configured to input the first visual feature into the enhanced coding module to obtain a second visual feature; The third acquisition unit is configured to input the second visual feature and the text question in the image-text pair of the image to be processed into the large language model to obtain an answer to the text question, The parameters of the visual encoder, the enhanced coding module and the large language model in the multimodal large model are obtained by the method according to any one of claims 1 to 4.

8. A computer-readable storage medium storing instructions, characterized in that: When the instructions are executed by at least one computing device, the at least one computing device is prompted to execute the method for enhancing the fine-grained perception capability of a multimodal large model as described in any one of claims 1 to 4 or the image processing method based on a multimodal large model as described in claim 5.

9. A system comprising at least one computing device and at least one storage device storing instructions, characterized in that: When the instructions are executed by the at least one computing device, they prompt the at least one computing device to execute the method for enhancing the fine-grained perception capability of a multimodal large model as described in any one of claims 1 to 4 or the image processing method based on a multimodal large model as described in claim 5.

10. A computer program product comprising computer instructions, characterized in that When the computer instructions are executed by a processor, the method for enhancing the fine-grained perception capability of a multimodal large model as described in any one of claims 1 to 4 or the image processing method based on a multimodal large model as described in claim 5 is implemented.

Images