Data processing method and device based on MLLM and MLLM training method and device based on MLLM
By combining fine-tuning and pre-training image encoder in MLLM, integrating image features and fusing text features, the problem of fine-tuning models ignoring the general ability of pre-training models is solved, and the accuracy of data processing and image expression capabilities are improved.
Patent Information
- Application Number
- CN202411045471.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-31
- Publication Date
- 2024-11-12
AI Technical Summary
In the prior art, fine-tuning models for specific tasks ignore the general capability of pre-trained models, resulting in reduced accuracy of data processing.
A MLLM-based method is adopted, combining a fine-tuned image encoder and a pretrained image encoder, the target image is encoded, image features are integrated, and text features are fused to obtain more accurate data processing results.
By integrating the general capabilities of the pre-trained model and the dedicated capabilities of the fine-tuning model, the accuracy of data processing and image expression capabilities are improved, and the effectiveness of object detection and generation tasks is improved.
Smart Images

Figure CN118939972A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of artificial intelligence technology, specifically to the technical fields of large models, multimodal data processing, etc., and in particular to a data processing and MLLM training method and device based on MLLM. Background Art
[0002] Multimodal Large Language Model (MLLM) is a new research hotspot that has emerged in recent years. It further introduces multimodal data processing capabilities based on the powerful generalization and reasoning capabilities of the Large Language Model (LLM).
[0003] The training process of a model usually includes a pre-training process and a fine-tuning process. The corresponding models can be called pre-trained models and fine-tuned models, respectively. The pre-trained model is used for general tasks, and the fine-tuned model is obtained by adjusting the pre-trained model based on a specific task. Summary of the invention
[0004] The present invention provides a data processing and MLLM training method and device based on MLLM.
[0005] According to one aspect of the present disclosure, a data processing method based on MLLM is provided, including: the MLLM includes: a fine-tuned image encoder and a pre-trained image encoder, and the method includes: using the fine-tuned image encoder to encode a target image to obtain a first image feature; using the pre-trained image encoder to encode the target image to obtain a second image feature; integrating the first image feature and the second image feature to obtain a target image feature; obtaining a target prompt feature based on an original prompt text; and obtaining a data processing result based on the target image feature and the target prompt feature.
[0006] According to another aspect of the present disclosure, an MLLM training method is provided, the MLLM comprising: a fine-tuning image encoder and a pre-training image encoder, the method comprising: using the fine-tuning image encoder to encode a target image sample to obtain a first image feature; using the pre-training image encoder to encode the target image sample to obtain a second image feature; integrating the first image feature and the second image feature to obtain a target image feature; obtaining a prediction result based on the target image feature and the target prompt feature; constructing a loss function based on the prediction result, and adjusting the model parameters of the fine-tuning image encoder based on the loss function, and fixing the model parameters of the pre-training image encoder.
[0007] According to another aspect of the present disclosure, a data processing device based on MLLM is provided, wherein the MLLM includes: a fine-tuned image encoder and a pre-trained image encoder, and the device includes: a first image encoding module, used to encode a target image using a fine-tuned image encoder to obtain a first image feature; a second image encoding module, used to encode the target image using a pre-trained image encoder to obtain a second image feature; an integration module, used to integrate the first image feature and the second image feature to obtain a target image feature; a first acquisition module, used to obtain a target prompt feature according to an original prompt text; and a second acquisition module, used to obtain a data processing result according to the target image feature and the target prompt feature.
[0008] According to another aspect of the present disclosure, an MLLM training device is provided, the MLLM comprising: a fine-tuning image encoder and a pre-training image encoder, the device comprising: a first image encoding module, for encoding a target image sample using the fine-tuning image encoder to obtain a first image feature; a second image encoding module, for encoding the target image sample using the pre-training image encoder to obtain a second image feature; an integration module, for integrating the first image feature and the second image feature to obtain a target image feature; an acquisition module, for acquiring a prediction result based on the target image feature and the target prompt feature; an adjustment module, for constructing a loss function based on the prediction result, and adjusting the model parameters of the fine-tuning image encoder based on the loss function, and fixing the model parameters of the pre-training image encoder.
[0009] According to another aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute any method as described in any of the above aspects.
[0010] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute any one of the methods according to any one of the above aspects.
[0011] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, wherein when the computer program is executed by a processor, the computer program implements any one of the methods described in any one of the above aspects.
[0012] The present disclosure can improve the accuracy of data processing.
[0013] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS
[0014] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.
[0015] Figure 1 is a schematic diagram according to a first embodiment of the present disclosure;
[0016] Figure 2 is a schematic diagram of an application scenario for implementing an embodiment of the present disclosure;
[0017] Figure 3 is a schematic diagram of data processing based on MLLM according to an embodiment of the present disclosure;
[0018] Figure 4 is a schematic diagram of a cache model provided according to an embodiment of the present disclosure;
[0019] Figure 5 is a schematic diagram according to a second embodiment of the present disclosure;
[0020] Figure 6 is a schematic diagram according to a third embodiment of the present disclosure;
[0021] Figure 7 is a schematic diagram according to a fourth embodiment of the present disclosure;
[0022] Figure 8 is a schematic diagram according to a fifth embodiment of the present disclosure;
[0023] Fig. 9 is a schematic diagram according to a sixth embodiment of the present disclosure;
[0024] Fig.10 It is a schematic diagram of an electronic device used to implement the MLLM-based data processing method or MLLM training method of an embodiment of the present disclosure. DETAILED DESCRIPTION
[0025] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.
[0026] In the related art, different specific tasks are usually processed by fine-tuning models corresponding to different tasks. For example, for the object detection task, for example, for the animal detection task, a model fine-tuned by animal image samples is used for detection, and for the vehicle recognition task, a model fine-tuned by vehicle image samples is used for detection.
[0027] Fine-tuning models corresponding to specific tasks ignores the general capabilities of pre-trained models and reduces the accuracy of data processing.
[0028] In order to improve data processing accuracy, the present disclosure provides the following embodiments.
[0029] Figure 1 is a schematic diagram according to the first embodiment of the present disclosure. This embodiment provides a data processing method based on MLLM, wherein the MLLM comprises: a fine-tuning image encoder and a pre-training image encoder, and the method comprises:
[0030] 101. Use the fine-tuning image encoder to encode a target image to obtain a first image feature.
[0031] 102. Use the pre-trained image encoder to encode the target image to obtain a second image feature.
[0032] 103. Integrate the first image feature and the second image feature to obtain a target image feature.
[0033] 104. Obtain target prompt features based on the original prompt text.
[0034] 105. Obtain data processing results based on the target image features and the target prompt features.
[0035] Among them, MLLM introduces multimodal data on the basis of LLM. In the embodiment of the present disclosure, the multimodal data includes images and texts.
[0036] Specifically, during data processing, the input of MLLM includes the target image and the original prompt text, and the output is the data processing result.
[0037] MLLM can be applied to target detection scenarios. In this scenario, the target image is the image to be detected, which includes one or more categories of objects. The original prompt text is used to describe the category of the object to be detected, and the data processing result is the target detection result. For example, the target image is an image of a school, which includes a teaching building and a school gate. The original prompt text is a text similar to "Identify the school gate". Then, after MLLM processing, the school gate in the target image can be obtained, such as marking the "school gate" with a rectangular frame.
[0038] MLLM can also be applied to other scenarios, such as generation scenarios. Specifically, if the target image contains one or more categories of objects, and the original prompt text is used to describe the modification of a certain category of objects to a specified content, then the data processing result is a generated image, and the generated image has the specified content. Alternatively, if the original prompt text is used to describe the generation of a paragraph of text based on the target image, then the data processing result is the text generated based on the target image and the original prompt text; or, if the original prompt text is used to describe the generation of a video based on the target image, then the data processing result is a video generated based on the target image and the original prompt text.
[0039] In order to retain the general capabilities of the pre-trained model, the image encoder includes: a pre-training image encoder and a fine-tuning image encoder. The pre-trained image encoder is trained using a large number of existing training samples in various fields and has general capabilities. The fine-tuning image encoder is obtained by fine-tuning the pre-trained image encoder using training samples corresponding to specific tasks and has special capabilities for specific tasks.
[0040] After acquiring the target image, the target image is input into the fine-tuning image encoder and the pre-training image encoder respectively. The image features output by the fine-tuning image encoder are called first image features, and the image features output by the pre-training image encoder are called second image features.
[0041] After obtaining the first image feature and the second image feature, the two image features are integrated to obtain the target image feature. In this way, the target image feature combines the capabilities of the pre-trained image encoder and the fine-tuned image encoder and has better image expression capabilities.
[0042] After the above processing, the target image features of the corresponding image mode are obtained.
[0043] In addition to the above-mentioned image encoder, MLLM also includes a text encoder, LLM and decoder.
[0044] For the text mode, the target text features can be obtained by using the text encoder and the original prompt text.
[0045] Specifically, the original prompt text can be input into a text encoder, and the output is the target text feature. Alternatively, the original prompt text can be processed, and the processed text can be input into a text encoder, and the target text feature can be output. Alternatively, the text feature output by the text encoder can be further processed to obtain the target text feature. For the specific acquisition process of the target text feature, please refer to the subsequent embodiments.
[0046] After obtaining the target text features, the target text features can be used as target prompt features.
[0047] In the above, the prompt information (prompt) is the original prompt text as an example. In order to improve the data processing effect, the prompt information can also be multimodal, that is, the prompt information includes not only the original prompt text but also a prompt image. The feature corresponding to the prompt image is called the prompt image feature. Afterwards, the target prompt feature can be obtained based on the target text feature and the prompt image feature.
[0048] After obtaining the target image features and target prompt features, LLM can be used to encode them to obtain encoded features. For example, the target image features are mapped to the space where the target prompt features are located to obtain the mapped image features. After the mapped image features and the target prompt features are concatenated, they are used as the input of LLM. After being processed by LLM, the output features are called encoded features.
[0049] The encoded features are input into the decoder, and the output is the data processing result, such as target detection result or generation result.
[0050] In this embodiment, a pre-trained image encoder is used to obtain a first image feature, and a fine-tuned image encoder is used to obtain a second image feature. The first image feature and the second image feature are integrated to obtain a target image feature. In this way, the target image feature combines the general capabilities of the pre-trained image encoder and the special capabilities of the fine-tuned image encoder, and has better image expression capabilities, thereby improving data processing accuracy.
[0051] In order to better understand the embodiments of the present disclosure, application scenarios to which the embodiments of the present disclosure can be applied are described.
[0052] Figure 2 It is a schematic diagram of an application scenario for implementing the embodiment of the present disclosure. The scenario includes: a user terminal 201 and a server 202, and the user terminal 201 may include: a personal computer (PC), a mobile device (such as a mobile phone), a tablet computer, a laptop computer, a smart wearable device, etc. The server 202 may be a cloud server or a local server, and the user terminal 201 and the server 202 may communicate using a communication network, and the communication network may include, for example, a wired network and / or a wireless network.
[0053] An application (APP) is deployed on the user terminal, and a server corresponding to the APP is deployed on the server. The user inputs the target image and the original prompt text used to describe the requirements through the APP. The APP sends the target image and the original prompt text input by the user to the server. The server uses the pre-configured MLLM to process the target image and the original prompt text to obtain the data processing result. After that, the server can send the data processing result to the APP, which will be displayed to the user.
[0054] In addition, in some cases, the user can also input a prompt image. At this time, the server processes the target image, original prompt text and prompt image input by the user based on MLLM to obtain a data processing result.
[0055] Figure 3 It is a schematic diagram of data processing based on MLLM provided according to an embodiment of the present disclosure.
[0056] In this embodiment, the processing of the target image and the original prompt text based on MLLM is taken as an example.
[0057] like Figure 3 As shown, the MLLM includes: a fine-tuned image encoder 301, a pre-trained image encoder 302, a text encoder 303, a LLM 304 and a decoder 305.
[0058] The input of the fine-tuning image encoder 301 is the target image, and the output is the first image feature; the input of the pre-training image encoder 302 is the target image, and the output is the second image feature; the input of the text encoder 303 is the target prompt text, and the output is the initial text feature; the input of the LLM 304 is the processed multimodal feature, and the output is the encoded feature; Figure 3 In the example, multimodal features including target image features and target text features are used. In specific implementation, the target image features and target text features can be processed and then input into the LLM. For example, the target image features are mapped to the space where the target text features are located to obtain the mapped image features, and the mapped image features and the target text features are spliced and then input into the LLM. The input of the decoder 305 is the encoded features, and the output is the data processing result.
[0059] The target image and the original prompt text are known data, for example, input by a user.
[0060] For the original prompt text: based on the original prompt text, initial text features can be obtained; based on the initial text features, target text features can be obtained; based on the target text features, the target prompt features can be obtained.
[0061] For example, the original prompt text is output to the text encoder, and the output is the initial text features; or Figure 3 As shown, the original prompt text is processed by the prompt text optimization module to obtain the target prompt text, and the target prompt text is input into the text encoder, and the output is the initial text feature.
[0062] After obtaining the initial text features, the initial text features can be used as target text features; or, as Figure 3 As shown, additional text features can also be obtained, and the initial text features and the additional text features are added to obtain the target text features.
[0063] After obtaining the target text features, such as Figure 3 As shown, the target text feature can be used as the target prompt feature. Alternatively, the prompt image feature corresponding to the prompt image can also be obtained, for example, the prompt image is input by the user, and the prompt image is encoded by using a fine-tuned image encoder to obtain the prompt image feature; the target prompt feature is obtained according to the target text feature and the prompt image feature, for example, the target text feature and the prompt image feature are fused by using a preset feature fusion algorithm to obtain the target prompt feature.
[0064] In this embodiment, initial text features are obtained based on the original prompt text, target text features are obtained based on the initial text features, and target prompt features are obtained based on the target text features. The target prompt features can be obtained in a variety of ways to improve flexibility and accuracy.
[0065] Further, for the initial text features: the original prompt text can be input into the text encoder, and the output is the initial text features. Or,
[0066] In order to improve the accuracy of target detection, the prompt text optimization module can be used to optimize the original prompt text to obtain the target prompt text; the target prompt text is input into the text encoder, and the output is the initial text feature.
[0067] Specifically, the prompt text optimization module may include a pre-configured optimization template, and the original prompt text is converted into the target prompt text based on the optimization template.
[0068] The structure of the optimization template is, for example: [V] 1 [V] 2 ...[V] M [CLASS], where [V] 1 [V] 2 ...[V] M Indicates target attribute information, and [CLASS] indicates target class information.
[0069] Specifically, the target prompt text obtained based on the optimized template is: [one][red][big][apple], where [one], [red], and [big] are all attribute information, and [apple] is category information.
[0070] Since the original prompt text is input by the user, different users may express it differently. For example, some users input "an image of a big red apple", while some users input "an image, there is a big apple in this image, and this apple is red". Through the prompt text optimization module, the original prompt texts input by different users can be standardized using optimized templates, which can improve detection accuracy.
[0071] After obtaining the target prompt text, the target prompt text is input into the text encoder, and the output is the initial text feature. The text encoder can be obtained by fine-tuning the pre-trained text encoder using training samples of a specific task.
[0072] In this embodiment, the target prompt text is obtained by processing the original prompt text, and the target prompt text is encoded using a text encoder to obtain initial text features, which can improve the accuracy of the initial text features and further improve the accuracy of the data processing results.
[0073] For target text features: the initial text features can be used as target text features; or,
[0074] In order to further improve the text representation capability, in this embodiment, additional text features may be introduced; and target text features are obtained based on the initial text features and the additional text features.
[0075] like Figure 3 As shown, the additional LLM can be used to process the original prompt text to obtain additional text features.
[0076] Specifically, the server can pre-configure a prompt template for obtaining additional text features. The prompt template is a two-stage template, that is, it includes a first template and a second template. In the first stage, the first template is used to obtain the objects contained in the target category, and in the second stage, the second template is used to obtain the common embedding features of the above objects. The specific content of the first template and the second template is configurable. For example, the first template is "query the objects contained in the category [CLASS]", and the second template is "query the common embedding features of each object under the category [CLASS]", where [CLASS] is the target category, which is obtained from the original prompt text. For example, the above original prompt text is about "apple", then [CLASS] is specifically filled with "apple".
[0077] After obtaining the universal embedding features of each object under the target category through the additional LLM, these universal embedding features are averaged and the average features are used as additional text features. Then, the initial text features and the additional text features are added to obtain the target text features.
[0078] In this embodiment, by introducing additional text features, the text expression ability can be improved and the data processing accuracy can be improved.
[0079] Regarding the target prompt feature: the target text feature can be used as the target prompt feature; or, the prompt image is encoded using a fine-tuned image encoder to obtain the prompt image feature; and the target prompt feature is obtained based on the target text feature and the prompt image feature.
[0080] In this embodiment, multimodal prompt information can be obtained through the prompt image, which can improve the accuracy of data processing.
[0081] For the target image: a first image feature and a second image feature are obtained respectively by fine-tuning an image encoder and a pre-trained image encoder, and an integration module is used to integrate the first image feature and the second image feature to obtain a target image feature.
[0082] Specifically, the text encoder can be used to encode the category text corresponding to the preset candidate category to obtain the candidate category feature; based on the first image feature and the candidate category feature, the zero-sample prediction feature is obtained; based on the zero-sample prediction feature, the first image feature and the second image feature are integrated to obtain the target image feature.
[0083] In this embodiment, the first image feature and the second image feature are integrated based on the zero-sample prediction feature, which can improve the accuracy of the target image feature and further improve the accuracy of data processing.
[0084] Specifically, the text encoder may be obtained by fine-tuning a pre-trained text encoder using training samples of a specific task. For this reason, the text encoder may be referred to as a fine-tuned text encoder.
[0085] The fine-tuned image encoder and the fine-tuned text encoder can be obtained by fine-tuning the CLIP image encoder and the CLIP text encoder.
[0086] CLIP stands for Contrastive Language-Image Pretraining. CLIP uses contrastive learning to understand the relationship between paired text and images. Its goal is to minimize the difference between the corresponding image encoding and text encoding.
[0087] After the contrastive learning process, the CLIP image encoder and the CLIP text encoder can be obtained. After that, the CLIP image encoder and the CLIP text encoder are fine-tuned to obtain a fine-tuned image encoder and a fine-tuned text encoder, and the obtained fine-tuned image encoder is used as the fine-tuned image encoder in the MLLM, and the obtained fine-tuned text encoder is used as the text encoder in the MLLM.
[0088] During the inference process, users can preset multiple candidate categories, such as the candidate categories for animals include: cats, dogs, etc. After that, each candidate category is converted into a category text using a preset template, such as the category text is "a cat". Then, the category text corresponding to each candidate category is encoded using the text encoder in MLLM to obtain the candidate category features corresponding to each candidate category. Assuming that there are N (N is a positive integer) candidate categories, N groups of candidate category features can be obtained.
[0089] After obtaining the candidate category features, the first image features and the candidate category features are used to obtain zero-sample prediction features.
[0090] The formula is:
[0091] p zs =f CLIP (CLIP text (P N )) T
[0092] Among them, p zs is the zero-shot prediction feature, f CLIP is the first image feature, CLIP text (P N ) is the candidate category feature, P N is the category text corresponding to the candidate category, and the superscript T represents the transposition operation.
[0093] The pre-trained image encoder in MLLM can be a pre-trained image encoder obtained by a pre-training method different from CLIP, specifically, a DINO image encoder. The DINO image encoder is an image encoder based on self-distillation learning, and its underlying architecture is a Vision Transformer (ViT) model.
[0094] Taking the DINO image encoder as an example, the pre-trained image encoder in MLLM is used to encode the target image using the DINO image encoder to obtain the second image feature f DINO Indicates that, based on this, the zero-sample prediction feature p is obtained zs After that, the zero-sample prediction feature is used to predict the first image feature f CLIP and the second image feature fDINO to integrate.
[0095] Specifically, the first image feature can be adapted to obtain a first prediction feature; the second image feature can be adapted to obtain a second prediction feature; based on the zero-sample prediction feature and the first prediction feature, a first weight can be obtained, and based on the zero-sample prediction feature and the second prediction feature, a second weight can be obtained; based on the first weight and the second weight, the first prediction feature and the second prediction feature are weighted summed to obtain a fused prediction feature; the zero-sample prediction feature and the fused prediction feature are added to obtain the target image feature.
[0096] In this embodiment, the target image feature is obtained by adding the zero-sample prediction feature and the fused prediction feature. The fused prediction feature is obtained by weighted summing the first prediction feature and the second prediction feature, which can improve the expression ability of the target image feature and enhance the data processing effect.
[0097] The formula is:
[0098] p en =p zs +w 1 p CLIP +w 2 p DINO
[0099] Among them, P en is the target image feature, p zs is the zero-shot prediction feature, p CLIP is the first predictive feature, p DINO is the second predictive feature, w 1 is the first weight, w 2 is the second weight, (w 1 p CLIP +w 2 p DINO ) is the fusion prediction feature.
[0100] The above adaptation process can be implemented through a cache model.
[0101] The cache model stores key-value pairs (KV), usually multiple pairs of KV, where the image feature of each support sample is K, and the label encoding of the corresponding support sample is V. The label encoding can be a one-hot encoding.
[0102] During adaptation, the similarity between the image features of the query sample and each K is calculated, and the predicted features are calculated using the similarity and the corresponding V.
[0103] In this embodiment, the first image feature and the second image feature are adapted respectively. To this end, two cache models are required, which are called the first cache model and the second cache model respectively. The image features of the query sample are the first image feature and the second image feature respectively. In this way, the first image feature is adapted by the first cache model to obtain the first prediction feature; the second image feature is adapted by the second cache model to obtain the second prediction feature. The first cache model is generated based on the first support sample and its label, and the second cache model is generated based on the second support sample and its label. The first support sample and the second support sample are preset and can be the same or different.
[0104] In this embodiment, the image features are adapted by the cache model to obtain the prediction features, which can eliminate the need for special training adapters, reduce learning costs, and improve processing efficiency.
[0105] Figure 4 It is a schematic diagram of a cache model provided according to an embodiment of the present disclosure.
[0106] like Figure 4 As shown, the fine-tuned image encoder is represented by CLIP image encoder, the pre-trained image encoder is represented by DINO image encoder, and the first image feature is represented by f CLIP The second image feature is represented by f DINO Indicates that the first prediction feature is p CLIP The second prediction feature is represented by p DINO Indicates that the image feature of the first support sample is represented by F CLIP Indicates that the label encoding of the first support sample is L CLIP Indicated by, the image feature of the second support sample is represented by F DINO Indicates that the label encoding of the second support sample is L DINO The first image feature is adapted through the first cache model to obtain the first prediction feature, and the second image feature is adapted through the second cache model to obtain the second prediction feature.
[0107] The adaptation process is expressed as:
[0108]
[0109] Among them, p CLIP ,p DINO are the first prediction feature and the second prediction feature respectively; f CLIP ,f DINO are the first image feature and the second image feature respectively; F CLIP ,F DINO are the image features of the first support sample and the image features of the second support sample respectively; L CLIP ,L DINOare the label codes corresponding to the first support sample and the label codes corresponding to the second support sample respectively; is a mapping function, such as the exponential function; the superscript T is the transpose operation.
[0110] The first weight and the second weight mentioned above are obtained based on the zero-sample prediction feature and the first prediction feature and the second prediction feature.
[0111] Specifically, a first initial weight is calculated based on the first prediction feature and the zero-sample prediction feature, and the first initial weight is normalized to obtain a first weight; a second initial weight is calculated based on the second prediction feature and the zero-sample prediction feature, and the second initial weight is normalized to obtain a second weight.
[0112] The calculation formula is:
[0113] w 1 =softmax(w CLIP )
[0114] w 2 =softmax(w DINO )
[0115]
[0116] Among them, w 1 ,w 2 are the first weight and the second weight respectively; p CLIP ,p DINO are the first prediction feature and the second prediction feature respectively; p zs is the zero-sample prediction feature; the superscript T is the transposition operation; softmax() is a normalization function.
[0117] Through the above process, the target image features corresponding to the target image can be obtained.
[0118] After obtaining the target prompt feature and the target image feature, the target prompt feature and the target image feature are encoded based on LLM to obtain the encoded feature. For example, the target image feature is mapped to the space where the target prompt feature is located to obtain the mapped image feature, and the mapped image feature and the target prompt feature are concatenated as the input of LLM. After LLM processing, the output of LLM is the encoded feature; the encoded feature is input into the decoder, and the output is the data processing result.
[0119] In this embodiment, the target image features and the target prompt features are encoded through LLM to obtain the encoded features, and then the data processing results are obtained based on the encoded features. The excellent performance of LLM can be used to improve the data processing effect.
[0120] In combination with the above application scenarios, the present disclosure also provides the following embodiments.
[0121] Figure 5 is a schematic diagram according to the second embodiment of the present disclosure, which provides a data processing method based on MLLM. The MLLM includes: a fine-tuned image encoder, a pre-trained image encoder, a text encoder, an LLM and a decoder.
[0122] This embodiment takes the prompt information as the original prompt text as an example, and accordingly, the target text feature is used as the target prompt feature.
[0123] like Figure 5 As shown, the method of this embodiment includes:
[0124] 501. Encode the target image using a fine-tuned image encoder to obtain a first image feature.
[0125] 502. Use a pre-trained image encoder to encode the target image to obtain a second image feature.
[0126] For example, see Figure 3 , a fine-tuned image encoder is used to obtain the first image feature, and a pre-trained image encoder is used to obtain the second image feature.
[0127] 503. Integrate the first image feature and the second image feature to obtain a target image feature.
[0128] For example, see Figure 4 , the first cache model is used to adapt the first image feature to obtain the first prediction feature, and the second cache model is used to adapt the second image feature to obtain the second prediction feature.
[0129] Afterwards, the first weight is calculated using the acquired zero-sample prediction feature and the first prediction feature, the second weight is calculated using the zero-sample prediction feature and the second prediction feature, and the first weight and the second weight are used to perform weighted summation on the first prediction feature and the second prediction feature to obtain a fused prediction feature, and the fused prediction feature is then added to the zero-sample prediction feature to obtain the target image feature.
[0130] 504. Obtain target text features based on the original prompt text and the text encoder.
[0131] For example, see Figure 3 , optimize the original prompt text based on the optimization template to obtain the target prompt text, input the target prompt text into the text encoder, and the output is the initial text feature. Use the additional LLM to process the original prompt text to obtain the additional text feature, and then add the initial text feature and the additional text feature to obtain the target text feature.
[0132] 505. Use LLM to encode the target image features and the target text features to obtain encoding features.
[0133] For example, the target image features are mapped to the space where the target text features are located to obtain the mapped image features. The mapped image features and the target text features are concatenated as the input of the LLM. After LLM processing, the output of the LLM is the encoded features.
[0134] 506. Use a decoder to decode the encoding feature to obtain a data processing result.
[0135] Among them, the encoded features can be input into the decoder, and the output is the data processing result.
[0136] In this embodiment, since the first image feature corresponds to the fine-tuned image encoder and the second image feature corresponds to the pre-trained image encoder, the target image feature is obtained by integrating the first image feature and the second image feature. This allows the target image feature to integrate general capabilities and special capabilities, improve the image expression capability, and thus improve the accuracy of data processing. By optimizing the original prompt text to obtain the target prompt text, the accuracy of the data processing results can be further improved. By obtaining additional text features through additional LLMs and obtaining target text features based on the initial text features and the additional text features, the text expression capability can be improved, thereby further improving the accuracy of the data processing results.
[0137] The above example takes the prompt information being the original prompt text as an example. In some cases, the prompt information may also include a prompt image. For example, if the original prompt text is text for detecting a school gate, then the prompt image may be an image about the school gate, which is helpful for MLLM to detect the school gate in the target image.
[0138] Figure 6 is a schematic diagram according to the third embodiment of the present disclosure, which provides a data processing method based on MLLM. The MLLM includes: a fine-tuned image encoder, a pre-trained image encoder, a text encoder, an LLM and a decoder.
[0139] This embodiment takes the prompt information including the original prompt text and the prompt image as an example, and accordingly, the target prompt feature is acquired based on the target text feature and the prompt image feature.
[0140] That is, the user not only inputs the original prompt text, but also inputs the prompt image, so that multimodal prompt information can be obtained. Processing based on the multimodal prompt information can improve the accuracy of data processing.
[0141] like Figure 6 As shown, the method of this embodiment includes:
[0142] 601. Encode the target image using a fine-tuned image encoder to obtain a first image feature.
[0143] 602. Use a pre-trained image encoder to encode the target image to obtain a second image feature.
[0144] 603. Integrate the first image feature and the second image feature to obtain a target image feature.
[0145] 604. Obtain target text features based on the original prompt text and the text encoder.
[0146] For the specific contents of 601 to 604, please refer to 501 to 504.
[0147] 605. Encode the prompt image using the fine-tuning image encoder to obtain prompt image features.
[0148] After obtaining the prompt image input by the user, the prompt image can be input into the fine-tuning image encoder, and the output is the prompt image feature.
[0149] 606. Obtain target prompt features based on the target text features and the prompt image features.
[0150] Among them, various related text feature and image feature fusion algorithms can be used to fuse the target text feature and the prompt image feature to obtain the target prompt feature.
[0151] For example, the MQ-Det model can be used to fuse the target text features and the prompt image features to obtain the target prompt features.
[0152] The MQ-Det model is a large model of multi-modal queried open-world target detection (Multi-modal Queried Object Detect in the wild), which can integrate multi-modal prompt information.
[0153] 607. Use LLM to encode the target image feature and the target prompt feature to obtain an encoded feature.
[0154] For example, the target image features are mapped to the space where the target prompt features are located to obtain the mapped image features. The mapped image features and the target prompt features are concatenated as the input of the LLM. After LLM processing, the output of the LLM is the encoded features.
[0155] 608. Use a decoder to decode the encoding feature to obtain a data processing result.
[0156] Among them, the encoded features can be input into the decoder, and the output is the data processing result.
[0157] In this embodiment, since the first image feature corresponds to a fine-tuned image encoder and the second image feature corresponds to a pre-trained image encoder, the target image feature is obtained by integrating the first image feature and the second image feature. This allows the target image feature to integrate general capabilities and special capabilities, improve image expression capabilities, and thus improve data processing accuracy; by optimizing the original prompt text to obtain the target prompt text, the accuracy of the data processing results can be further improved; by obtaining additional text features through an additional LLM and obtaining the target text features based on the initial text features and the additional text features, the text expression capabilities can be improved, thereby further improving the accuracy of the data processing results; by including a prompt image in the prompt information, more modal prompt information can be provided, further improving the accuracy of the data processing results.
[0158] Figure 7 is a schematic diagram according to the fourth embodiment of the present disclosure. This embodiment provides an MLLM training method, the MLLM comprising: a fine-tuning image encoder and a pre-training image encoder, the method comprising:
[0159] 701. Use the fine-tuning image encoder to encode the target image sample to obtain a first image feature.
[0160] 702. Use the pre-trained image encoder to encode the target image sample to obtain a second image feature.
[0161] 703. Integrate the first image feature and the second image feature to obtain a target image feature.
[0162] 704. Obtain a prediction result based on the target image feature and the target prompt feature.
[0163] 705. Construct a loss function based on the prediction result, adjust the model parameters of the fine-tuning image encoder based on the loss function, and fix the model parameters of the pre-trained image encoder.
[0164] Among them, the target prompt feature is obtained based on the original prompt text sample.
[0165] After obtaining the prediction result, a loss function is constructed according to the prediction result and the actual result, and the model parameters of the fine-tuning image encoder are adjusted using the loss function, while the model parameters of the pre-trained image encoder are fixed.
[0166] Among them, the target image samples, original prompt text samples and true results are pre-obtained training data for specific tasks.
[0167] In this embodiment, a pre-trained image encoder is used to obtain a first image feature, a fine-tuned image encoder is used to obtain a second image feature, and the first image feature and the second image feature are integrated to obtain a target image feature, so that the target image feature combines the general capabilities of the pre-trained image encoder and the special capabilities of the fine-tuned image encoder, and has better image expression capabilities, thereby improving the accuracy of MLLM. In addition, by adjusting the model parameters of the fine-tuned image encoder and keeping the model parameters of the pre-trained image encoder unchanged, the training efficiency can be improved.
[0168] In some embodiments, the MLLM further comprises: a text encoder;
[0169] The integrating the first image feature and the second image feature to obtain a target image feature includes:
[0170] Using the text encoder, encoding the category text corresponding to the preset candidate category to obtain the candidate category feature;
[0171] Acquire a zero-sample prediction feature based on the first image feature and the candidate category feature;
[0172] Based on the zero-sample prediction feature, the first image feature and the second image feature are integrated to obtain the target image feature.
[0173] In this embodiment, the first image feature and the second image feature are integrated based on the zero-sample prediction feature, which can improve the accuracy of the target image feature and further improve the accuracy of data processing.
[0174] In some embodiments, the integrating the first image feature and the second image feature based on the zero-sample prediction feature to obtain the target image feature includes:
[0175] Adapting the first image feature to obtain a first prediction feature;
[0176] Adapting the second image feature to obtain a second prediction feature;
[0177] Based on the zero-sample prediction feature and the first prediction feature, obtaining a first weight, and based on the zero-sample prediction feature and the second prediction feature, obtaining a second weight;
[0178] Based on the first weight and the second weight, performing a weighted summation on the first prediction feature and the second prediction feature to obtain a fused prediction feature;
[0179] The zero-sample prediction feature and the fusion prediction feature are added to obtain the target image feature.
[0180] In this embodiment, the target image feature is obtained by adding the zero-sample prediction feature and the fused prediction feature. The fused prediction feature is obtained by weighted summing the first prediction feature and the second prediction feature, which can improve the expression ability of the target image feature and enhance the data processing effect.
[0181] In some embodiments, adapting the first image feature to obtain the first prediction feature includes: using a pre-built first cache model to adapt the first image feature to obtain the first prediction feature, the first cache model being generated based on a first support sample and its label;
[0182] and / or,
[0183] The adapting the second image feature to obtain the second prediction feature includes: using a pre-built second cache model to adapt the second image feature to obtain the second prediction feature, wherein the second cache model is generated based on a second support sample and its label.
[0184] In this embodiment, the image features are adapted by the cache model to obtain the prediction features, which can eliminate the need for special training adapters, reduce learning costs, and improve processing efficiency.
[0185] In some embodiments, the step of obtaining target prompt features based on the original prompt text sample includes:
[0186] Based on the original prompt text sample, obtaining initial text features;
[0187] Based on the initial text features, obtaining target text features;
[0188] Based on the target text feature, the target prompt feature is obtained.
[0189] In this embodiment, initial text features are obtained based on the original prompt text sample, target text features are obtained based on the initial text features, and target prompt features are obtained based on the target text features. The target prompt features can be obtained in a variety of ways to improve flexibility and accuracy.
[0190] In some embodiments, the MLLM further comprises: a text encoder;
[0191] The obtaining of initial text features based on the original prompt text sample includes:
[0192] Using the text encoder to encode the original prompt text to obtain the initial text features; or,
[0193] Based on a preset optimization template, the original prompt text is processed to obtain a target prompt text; and the target prompt text is encoded using the text encoder to obtain the initial text features.
[0194] In this embodiment, the target prompt text is obtained by processing the original prompt text sample, and the target prompt text is encoded using a text encoder to obtain initial text features, which can improve the accuracy of the initial text features and further improve the accuracy of the data processing model.
[0195] In some embodiments, the step of obtaining target text features based on the initial text features includes:
[0196] Using the initial text features as the target text features; or,
[0197] The original prompt text is processed using an additional LLM to obtain additional text features; and the target text features are obtained based on the additional text features and the initial text features.
[0198] In this embodiment, by introducing additional text features, the text expression ability can be improved and the accuracy of the data processing model can be improved.
[0199] In some embodiments, obtaining the target prompt feature based on the target text feature includes:
[0200] Using the target text feature as the target prompt feature; or,
[0201] The prompt image sample is encoded using the fine-tuned image encoder to obtain a prompt image feature; and the target prompt feature is obtained based on the target text feature and the prompt image feature.
[0202] In this embodiment, multimodal prompt information can be obtained by prompting image samples, which can improve the accuracy of the data processing model.
[0203] In some embodiments, the method further comprises:
[0204] Based on the loss function, model parameters of the text encoder are adjusted.
[0205] In this embodiment, by adjusting the model parameters of the text encoder, the text encoder can be fine-tuned, the text encoder performance can be improved, and the data processing performance can be enhanced.
[0206] In some embodiments, the MLLM further comprises: an LLM and a decoder;
[0207] The obtaining of prediction results based on the target image features and the target prompt features includes:
[0208] Using the LLM, encoding the target image feature and the target prompt feature to obtain a coded feature;
[0209] The decoder is used to decode the encoded feature to obtain the prediction result.
[0210] In this embodiment, the target image features and the target prompt features are encoded through LLM to obtain the encoded features, and then the target detection results are obtained based on the encoded features. The excellent performance of LLM can be used to improve the effect of the data processing model.
[0211] In some embodiments, the method further comprises:
[0212] Based on the loss function, the model parameters of the decoder are adjusted, and the model parameters of the LLM are fixed.
[0213] In this embodiment, by adjusting the model parameters of the decoder, the decoder can be fine-tuned, the decoder performance can be improved, and the performance of the data processing model can be improved. In addition, keeping the model parameters of the LLM unchanged can reduce the amount of calculation and improve processing efficiency.
[0214] Figure 8 801 , a second image encoding module 802 , an integration module 803 , a first acquisition module 804 , and a second acquisition module 805 .
[0215] The first image encoding module 801 is used to encode the target image using a fine-tuned image encoder to obtain a first image feature; the second image encoding module 802 is used to encode the target image using a pre-trained image encoder to obtain a second image feature; the integration module 803 is used to integrate the first image feature and the second image feature to obtain a target image feature; the first acquisition module 804 is used to obtain a target prompt feature based on the original prompt text; the second acquisition module 805 is used to obtain a data processing result based on the target image feature and the target prompt feature.
[0216] In this embodiment, a pre-trained image encoder is used to obtain a first image feature, and a fine-tuned image encoder is used to obtain a second image feature. The first image feature and the second image feature are integrated to obtain a target image feature. In this way, the target image feature combines the general capabilities of the pre-trained image encoder and the special capabilities of the fine-tuned image encoder, and has better image expression capabilities, thereby improving data processing accuracy.
[0217] In some embodiments, the MLLM further comprises: a text encoder, and the integration module 803 is further used to:
[0218] Using the text encoder, encoding the category text corresponding to the preset candidate category to obtain the candidate category feature;
[0219] Acquire a zero-sample prediction feature based on the first image feature and the candidate category feature;
[0220] Based on the zero-sample prediction feature, the first image feature and the second image feature are integrated to obtain the target image feature.
[0221] In this embodiment, the first image feature and the second image feature are integrated based on the zero-sample prediction feature, which can improve the accuracy of the target image feature and further improve the accuracy of data processing.
[0222] In some embodiments, the integration module 803 is further configured to:
[0223] Adapting the first image feature to obtain a first prediction feature;
[0224] Adapting the second image feature to obtain a second prediction feature;
[0225] Based on the zero-sample prediction feature and the first prediction feature, obtaining a first weight, and based on the zero-sample prediction feature and the second prediction feature, obtaining a second weight;
[0226] Based on the first weight and the second weight, performing a weighted summation on the first prediction feature and the second prediction feature to obtain a fused prediction feature;
[0227] The zero-sample prediction feature and the fusion prediction feature are added to obtain the target image feature.
[0228] In this embodiment, the target image feature is obtained by adding the zero-sample prediction feature and the fused prediction feature. The fused prediction feature is obtained by weighted summing the first prediction feature and the second prediction feature, which can improve the expression ability of the target image feature and enhance the data processing effect.
[0229] In some embodiments, the integration module 803 is further configured to:
[0230] Adapt the first image feature using a pre-built first cache model to obtain the first prediction feature, wherein the first cache model is generated based on a first support sample and its label; and / or adapt the second image feature using a pre-built second cache model to obtain the second prediction feature, wherein the second cache model is generated based on a second support sample and its label.
[0231] In this embodiment, the image features are adapted by the cache model to obtain the prediction features, which can eliminate the need for special training adapters, reduce learning costs, and improve processing efficiency.
[0232] In some embodiments, the first acquisition module 804 is further configured to:
[0233] Based on the original prompt text, obtaining initial text features;
[0234] Based on the initial text features, obtaining target text features;
[0235] Based on the target text feature, the target prompt feature is obtained.
[0236] In this embodiment, initial text features are obtained based on the original prompt text, target text features are obtained based on the initial text features, and target prompt features are obtained based on the target text features. The target prompt features can be obtained in a variety of ways to improve flexibility and accuracy.
[0237] In some embodiments, the MLLM further includes: a text encoder, and the first acquisition module 804 is further used to:
[0238] Using the text encoder to encode the original prompt text to obtain the initial text features; or,
[0239] Based on a preset optimization template, the original prompt text is processed to obtain a target prompt text; and the target prompt text is encoded using the text encoder to obtain the initial text features.
[0240] In this embodiment, the target prompt text is obtained by processing the original prompt text, and the target prompt text is encoded using a text encoder to obtain initial text features, which can improve the accuracy of the initial text features and further improve the accuracy of the data processing results.
[0241] In some embodiments, the first acquisition module 804 is further configured to:
[0242] Using the initial text features as the target text features; or,
[0243] The original prompt text is processed using an additional LLM to obtain additional text features; and the target text features are obtained based on the additional text features and the initial text features.
[0244] In this embodiment, by introducing additional text features, the text expression ability can be improved and the data processing accuracy can be improved.
[0245] In some embodiments, the first acquisition module 804 is further configured to:
[0246] Using the target text feature as the target prompt feature; or,
[0247] The prompt image is encoded using the fine-tuned image encoder to obtain prompt image features; and the target prompt features are obtained based on the target text features and the prompt image features.
[0248] In this embodiment, the user not only inputs the original prompt text, but also inputs the prompt image, so that multimodal prompt information can be obtained. Processing based on the multimodal prompt information can improve the accuracy of data processing.
[0249] In some embodiments, the MLLM further includes: an LLM and a decoder, and the second acquisition module 805 is further used for:
[0250] Using the LLM, encoding the target image feature and the target prompt feature to obtain a coded feature;
[0251] The decoder is used to decode the encoding feature to obtain the data processing result.
[0252] In this embodiment, the target image features and the target prompt features are encoded through LLM to obtain the encoded features, and then the data processing results are obtained based on the encoded features. The excellent performance of LLM can be used to improve the data processing effect.
[0253] Fig. 9 1 is a schematic diagram according to the sixth embodiment of the present disclosure. This embodiment provides an MLLM training device, the MLLM includes: a fine-tuning image encoder and a pre-training image encoder, the device includes: a first image encoding module 901, a second image encoding module 902, an integration module 903, an acquisition module 904 and an adjustment module 905.
[0254] The first image encoding module 901 is used to use the fine-tuning image encoder to encode the target image sample to obtain the first image feature; the second image encoding module 902 is used to use the pre-trained image encoder to encode the target image sample to obtain the second image feature; the integration module 903 is used to integrate the first image feature and the second image feature to obtain the target image feature; the acquisition module 904 is used to obtain the prediction result according to the target image feature and the target prompt feature; the adjustment module 905 is used to construct a loss function according to the prediction result, and adjust the model parameters of the fine-tuning image encoder according to the loss function, and fix the model parameters of the pre-trained image encoder.
[0255] In this embodiment, a pre-trained image encoder is used to obtain a first image feature, a fine-tuned image encoder is used to obtain a second image feature, and the first image feature and the second image feature are integrated to obtain a target image feature, so that the target image feature combines the general capabilities of the pre-trained image encoder and the special capabilities of the fine-tuned image encoder, and has better image expression capabilities, thereby improving the accuracy of MLLM. In addition, by adjusting the model parameters of the fine-tuned image encoder and keeping the model parameters of the pre-trained image encoder unchanged, the training efficiency can be improved.
[0256] In some embodiments, the MLLM further comprises: a text encoder;
[0257] The integration module 903 is further used for:
[0258] Using the text encoder, encoding the category text corresponding to the preset candidate category to obtain the candidate category feature;
[0259] Acquire a zero-sample prediction feature based on the first image feature and the candidate category feature;
[0260] Based on the zero-sample prediction feature, the first image feature and the second image feature are integrated to obtain the target image feature.
[0261] In this embodiment, the first image feature and the second image feature are integrated based on the zero-sample prediction feature, which can improve the accuracy of the target image feature and further improve the accuracy of data processing.
[0262] In some embodiments, the integration module 903 is further used to:
[0263] Adapting the first image feature to obtain a first prediction feature;
[0264] Adapting the second image feature to obtain a second prediction feature;
[0265] Based on the zero-sample prediction feature and the first prediction feature, obtaining a first weight, and based on the zero-sample prediction feature and the second prediction feature, obtaining a second weight;
[0266] Based on the first weight and the second weight, performing a weighted summation on the first prediction feature and the second prediction feature to obtain a fused prediction feature;
[0267] The zero-sample prediction feature and the fusion prediction feature are added to obtain the target image feature.
[0268] In this embodiment, the target image feature is obtained by adding the zero-sample prediction feature and the fused prediction feature. The fused prediction feature is obtained by weighted summing the first prediction feature and the second prediction feature, which can improve the expression ability of the target image feature and enhance the data processing effect.
[0269] In some embodiments, the integration module 903 is further used to:
[0270] Adapt the first image feature using a pre-built first cache model to obtain the first prediction feature, wherein the first cache model is generated based on a first support sample and its label; and / or adapt the second image feature using a pre-built second cache model to obtain the second prediction feature, wherein the second cache model is generated based on a second support sample and its label.
[0271] In this embodiment, the image features are adapted by the cache model to obtain the prediction features, which can eliminate the need for special training adapters, reduce learning costs, and improve processing efficiency.
[0272] In some embodiments, the acquisition module 904 is further configured to:
[0273] Based on the original prompt text sample, obtain initial text features;
[0274] Based on the initial text features, obtaining target text features;
[0275] Based on the target text feature, the target prompt feature is obtained.
[0276] In this embodiment, initial text features are obtained based on the original prompt text sample, target text features are obtained based on the initial text features, and target prompt features are obtained based on the target text features. The target prompt features can be obtained in a variety of ways to improve flexibility and accuracy.
[0277] In some embodiments, the MLLM further comprises: a text encoder;
[0278] The acquisition module 904 is further used for:
[0279] Using the text encoder to encode the original prompt text to obtain the initial text features; or,
[0280] Based on a preset optimization template, the original prompt text is processed to obtain a target prompt text; and the target prompt text is encoded using the text encoder to obtain the initial text features.
[0281] In this embodiment, the target prompt text is obtained by processing the original prompt text sample, and the target prompt text is encoded using a text encoder to obtain initial text features, which can improve the accuracy of the initial text features and further improve the accuracy of the data processing model.
[0282] In some embodiments, the acquisition module 904 is further configured to:
[0283] Using the initial text features as the target text features; or,
[0284] The original prompt text is processed using an additional LLM to obtain additional text features; and the target text features are obtained based on the additional text features and the initial text features.
[0285] In this embodiment, by introducing additional text features, the text expression ability can be improved and the accuracy of the data processing model can be improved.
[0286] In some embodiments, the acquisition module 904 is further configured to:
[0287] Using the target text feature as the target prompt feature; or,
[0288] The prompt image is encoded using the fine-tuned image encoder to obtain prompt image features; and the target prompt features are obtained based on the target text features and the prompt image features.
[0289] In this embodiment, multimodal prompt information can be obtained by prompting image samples, which can improve the accuracy of the data processing model.
[0290] In some embodiments, the adjustment module 905 is further used to: adjust the model parameters of the text encoder based on the loss function.
[0291] In this embodiment, by adjusting the model parameters of the text encoder, the text encoder can be fine-tuned, the text encoder performance can be improved, and the data processing performance can be enhanced.
[0292] In some embodiments, the MLLM further comprises: an LLM and a decoder;
[0293] The acquisition module 904 is further used for:
[0294] Using the LLM, encoding the target image feature and the target prompt feature to obtain a coded feature;
[0295] The decoder is used to decode the encoded feature to obtain the prediction result.
[0296] In this embodiment, the target image features and the target prompt features are encoded through LLM to obtain the encoded features, and then the target detection results are obtained based on the encoded features. The excellent performance of LLM can be used to improve the effect of the data processing model.
[0297] In some embodiments, the adjustment module 905 is further used to: adjust the model parameters of the decoder based on the loss function, and fix the model parameters of the LLM.
[0298] In this embodiment, by adjusting the model parameters of the decoder, the decoder can be fine-tuned, the decoder performance can be improved, and the performance of the data processing model can be improved. In addition, keeping the model parameters of the LLM unchanged can reduce the amount of calculation and improve processing efficiency.
[0299] It can be understood that in the embodiments of the present disclosure, the same or similar contents in different embodiments can be referenced to each other.
[0300] It can be understood that the “first”, “second”, etc. in the embodiments of the present disclosure are only used for distinction and do not indicate the degree of importance, time sequence, etc.
[0301] It is understandable that, unless otherwise specified, the order of the steps in the process indicates that the timing relationship between these steps is not limited.
[0302] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.
[0303] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.
[0304] Fig.10A schematic block diagram of an example electronic device 1000 that can be used to implement an embodiment of the present disclosure is shown. The electronic device 1000 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.
[0305] like Fig.10 As shown, the electronic device 1000 includes a computing unit 1001, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1002 or a computer program loaded from a storage unit 1008 into a random access memory (RAM) 1003. In the RAM 1003, various programs and data required for the operation of the electronic device 1000 can also be stored. The computing unit 1001, the ROM 1002, and the RAM 1003 are connected to each other via a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.
[0306] Multiple components in the electronic device 1000 are connected to the I / O interface 1005, including: an input unit 1006, such as a keyboard, a mouse, etc.; an output unit 1007, such as various types of displays, speakers, etc.; a storage unit 1008, such as a disk, an optical disk, etc.; and a communication unit 1009, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 1009 allows the electronic device 1000 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.
[0307] The computing unit 1001 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 1001 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 1001 performs the various methods and processes described above, such as a data processing method or an MLLM training method based on MLLM. For example, in some embodiments, a data processing method or an MLLM training method based on MLLM may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 1008. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 1000 via ROM 1002 and / or communication unit 1007. When the computer program is loaded into RAM 1003 and executed by the computing unit 1001, one or more steps of the data processing method or the MLLM training method based on MLLM described above may be performed. Alternatively, in other embodiments, the computing unit 1001 may be configured to execute the MLLM-based data processing method or the MLLM training method in any other appropriate manner (eg, by means of firmware).
[0308] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.
[0309] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable information recommendation device, so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine as a stand-alone software package and partially on a remote machine, or entirely on a remote machine or server.
[0310] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
[0311] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).
[0312] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.
[0313] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services ("Virtual Private Server", or "VPS" for short). The server may also be a server of a distributed system, or a server combined with a blockchain.
[0314] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.
[0315] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.
Claims
1. A data processing method based on MLLM, the MLLM comprising: Fine-tuning an image encoder and pre-training an image encoder, the method comprising: Using the fine-tuned image encoder, encoding the target image to obtain a first image feature; Using the pre-trained image encoder, encode the target image to obtain a second image feature; Integrating the first image feature and the second image feature to obtain a target image feature; Obtain target prompt features based on the original prompt text; Based on the target image features and the target prompt features, a data processing result is obtained.
2. The method according to claim 1, wherein: The MLLM also includes: a text encoder; The integrating the first image feature and the second image feature to obtain a target image feature includes: Using the text encoder, encoding the category text corresponding to the preset candidate category to obtain the candidate category feature; Acquire a zero-sample prediction feature based on the first image feature and the candidate category feature; Based on the zero-sample prediction feature, the first image feature and the second image feature are integrated to obtain the target image feature.
3. The method according to claim 2, wherein: The integrating the first image feature and the second image feature based on the zero-sample prediction feature to obtain the target image feature includes: Adapting the first image feature to obtain a first prediction feature; Adapting the second image feature to obtain a second prediction feature; Based on the zero-sample prediction feature and the first prediction feature, obtaining a first weight, and based on the zero-sample prediction feature and the second prediction feature, obtaining a second weight; Based on the first weight and the second weight, performing a weighted summation on the first prediction feature and the second prediction feature to obtain a fused prediction feature; The zero-sample prediction feature and the fusion prediction feature are added to obtain the target image feature.
4. The method according to claim 3, wherein: Adapting the first image feature to obtain a first prediction feature includes: using a pre-built first cache model to adapt the first image feature to obtain the first prediction feature, wherein the first cache model is generated based on a first support sample and its label; and / or, The adapting the second image feature to obtain the second prediction feature includes: using a pre-built second cache model to adapt the second image feature to obtain the second prediction feature, wherein the second cache model is generated based on a second support sample and its label.
5. The method according to claim 1, wherein: The step of acquiring the target prompt feature based on the original prompt text includes: Based on the original prompt text, obtaining initial text features; Based on the initial text features, obtaining target text features; Based on the target text feature, the target prompt feature is obtained.
6. The method according to claim 5, wherein: The MLLM also includes: a text encoder; The obtaining of initial text features based on the original prompt text includes: Using the text encoder to encode the original prompt text to obtain the initial text features; or, Based on a preset optimization template, the original prompt text is processed to obtain a target prompt text; and the target prompt text is encoded using the text encoder to obtain the initial text features.
7. The method according to claim 5, wherein: The step of obtaining target text features based on the initial text features includes: Using the initial text features as the target text features; or, The original prompt text is processed using an additional LLM to obtain additional text features; and the target text features are obtained based on the additional text features and the initial text features.
8. The method according to claim 5, wherein: The step of acquiring the target prompt feature based on the target text feature includes: Using the target text feature as the target prompt feature; or, The prompt image is encoded using the fine-tuned image encoder to obtain prompt image features; and the target prompt features are obtained based on the target text features and the prompt image features.
9. The method according to claim 1, wherein: The MLLM also includes: an LLM and a decoder; The obtaining of data processing results based on the target image features and the target prompt features includes: Using the LLM, encoding the target image feature and the target prompt feature to obtain a coded feature; The decoder is used to decode the encoding feature to obtain the data processing result.
10. A MLLM training method, the MLLM comprising: Fine-tuning an image encoder and pre-training an image encoder, the method comprising: Using the fine-tuned image encoder, encoding the target image sample to obtain a first image feature; Using the pre-trained image encoder, encode the target image sample to obtain a second image feature; Integrating the first image feature and the second image feature to obtain a target image feature; Acquire a prediction result based on the target image feature and the target prompt feature; A loss function is constructed based on the prediction result, and model parameters of the fine-tuning image encoder are adjusted based on the loss function, while model parameters of the pre-training image encoder are fixed.
11. The method according to claim 10, wherein: The integrating the first image feature and the second image feature to obtain the target image feature includes: using the text encoder included in the MLLM to encode the category text corresponding to the preset candidate category to obtain the candidate category feature; obtaining the zero-sample prediction feature based on the first image feature and the candidate category feature; integrating the first image feature and the second image feature based on the zero-sample prediction feature to obtain the target image feature; and / or, The method further includes: obtaining initial text features based on the original prompt text sample; obtaining target text features based on the initial text features; obtaining target prompt features based on the target text features; and / or, The obtaining of prediction results based on the target image features and the target prompt features includes: using the LLM included in the MLLM to encode the target image features and the target prompt features to obtain encoded features; and using the decoder included in the MLLM to decode the encoded features to obtain the prediction results.
12. The method according to claim 11, wherein: The integrating the first image feature and the second image feature based on the zero-sample prediction feature to obtain the target image feature includes: adapting the first image feature to obtain a first prediction feature; adapting the second image feature to obtain a second prediction feature; obtaining a first weight based on the zero-sample prediction feature and the first prediction feature, and obtaining a second weight based on the zero-sample prediction feature and the second prediction feature; performing weighted summation of the first prediction feature and the second prediction feature based on the first weight and the second weight to obtain a fused prediction feature; adding the zero-sample prediction feature and the fused prediction feature to obtain the target image feature; and / or, The obtaining of the initial text features based on the original prompt text sample includes: using the text encoder to encode the original prompt text to obtain the initial text features; or, based on a preset optimization template, processing the original prompt text to obtain a target prompt text; and, using the text encoder to encode the target prompt text to obtain the initial text features; and / or, The acquiring of the target text features based on the initial text features comprises: using the initial text features as the target text features; or, processing the original prompt text with an additional LLM to obtain additional text features; and acquiring the target text features based on the additional text features and the initial text features; and / or, The step of obtaining the target prompt feature based on the target text feature includes: using the target text feature as the target prompt feature; or encoding the prompt image using the fine-tuning image encoder to obtain a prompt image feature; and obtaining the target prompt feature based on the target text feature and the prompt image feature; and / or, The method further comprises: adjusting a model parameter of the text encoder based on the loss function; and / or, The method further includes: adjusting a model parameter of the decoder based on the loss function, and fixing a model parameter of the LLM.
13. The method according to claim 12, wherein: Adapting the first image feature to obtain a first prediction feature includes: using a pre-built first cache model to adapt the first image feature to obtain the first prediction feature, wherein the first cache model is generated based on a first support sample and its label; and / or, The adapting the second image feature to obtain the second prediction feature includes: using a pre-built second cache model to adapt the second image feature to obtain the second prediction feature, wherein the second cache model is generated based on a second support sample and its label.
14. A data processing device based on MLLM, the MLLM comprising: Fine-tuning an image encoder and pre-training an image encoder, the apparatus comprising: A first image encoding module, used to encode the target image using a fine-tuned image encoder to obtain a first image feature; A second image encoding module, used to encode the target image using a pre-trained image encoder to obtain a second image feature; An integration module, used for integrating the first image feature and the second image feature to obtain a target image feature; A first acquisition module is used to acquire target prompt features according to the original prompt text; The second acquisition module is used to acquire data processing results according to the target image features and the target prompt features.
15. A MLLM training device, the MLLM comprising: Fine-tuning an image encoder and pre-training an image encoder, the apparatus comprising: A first image encoding module, configured to encode a target image sample using the fine-tuned image encoder to obtain a first image feature; A second image encoding module, used to encode the target image sample using the pre-trained image encoder to obtain a second image feature; An integration module, used for integrating the first image feature and the second image feature to obtain a target image feature; An acquisition module, used for acquiring a prediction result according to the target image features and the target prompt features; An adjustment module is used to construct a loss function according to the prediction result, and adjust the model parameters of the fine-tuning image encoder according to the loss function, and fix the model parameters of the pre-trained image encoder.
16. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 13.
17. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-13.
18. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 13.
