Data processing method and device based on MLLM and MLLM training method and device based on MLLM

By using pre-trained image encoder and candidate text encoder to feature extraction and processing of multimodal data in a multimodal large language model, the problem of insufficient data processing performance in the prior art is solved, and the multimodal is realized Efficient processing of data and accurate results.

CN118939971APending Publication Date: 2024-11-12BEIJING BAIDU NETCOM SCI & TECH CO LTD
View PDF 0 Cites 2 Cited by

Patent Information

Application Number
CN202411045306.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-31
Publication Date
2024-11-12

AI Technical Summary

Technical Problem

How to improve the performance of data processing based on multimodal large language model (MLLM), especially when processing multimodal data, it is difficult for the existing technology to effectively improve performance.

Method used

By using pre-trained image encoder and multiple candidate text encoders, the image and text are encoded separately by determining the type of target multimodal data, and the features are obtained, and data processing is performed based on these features, and LLM is used for encoding and decoding. Get the final result.

Benefits of technology

It improves data processing performance, realizes efficient processing of multimodal data, retains the general ability of pre-trained models, and is suitable for object detection and generation scenarios.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118939971A_ABST
    Figure CN118939971A_ABST
Patent Text Reader

Abstract

The invention provides an MLLM-based data processing method and device and an MLLM training method and device, and relates to the technical field of artificial intelligence, in particular to the technical field of large-model and multi-modal data processing and the like. The MLLM-based data processing method comprises the following steps: determining a target data type of target multi-modal data; the multi-modal data comprises a target image and a target prompt text; encoding the target image by adopting a pre-training image encoder in the MLLM to obtain a target image feature; determining a target text encoder corresponding to the target data type in a plurality of candidate text encoders in the MLLM; adopting the target text encoder to encode the target prompt text to obtain target text features; obtaining a target prompt feature based on the target text feature; and obtaining a data processing result based on the target image feature and the target prompt feature.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, specifically to the technical fields of large models, multimodal data processing, etc., and in particular to a data processing and MLLM training method and device based on MLLM. Background Art

[0002] Multimodal Large Language Model (MLLM) is a new research hotspot that has emerged in recent years. It further introduces multimodal information processing capabilities based on the powerful generalization and reasoning capabilities of the Large Language Model (LLM).

[0003] How to improve the data processing performance based on MLLM is a problem that needs to be solved. Summary of the invention

[0004] The present invention provides a method, device and equipment for data processing and MLLM training based on MLLM.

[0005] According to one aspect of the present disclosure, a data processing method based on MLLM is provided, wherein the MLLM includes: a pre-trained image encoder and a plurality of candidate text encoders, and the method includes: determining a target data type of target multimodal data; the multimodal data includes: a target image and a target prompt text; using the pre-trained image encoder to encode the target image to obtain target image features; determining a target text encoder corresponding to the target data type among the plurality of candidate text encoders; using the target text encoder to encode the target prompt text to obtain target text features; obtaining target prompt features based on the target text features; and obtaining data processing results based on the target image features and the target prompt features.

[0006] According to another aspect of the present disclosure, an MLLM training method is provided, the MLLM comprising: a pre-trained image encoder and a candidate text encoder, the method comprising: obtaining training data of the same data type as that of the candidate text encoder, the training data comprising: a target image sample, a target prompt text sample and a real result; using the pre-trained image encoder to encode the target image sample to obtain a target image feature; using the candidate text encoder to encode the target prompt text sample to obtain a target text feature; obtaining a target prompt feature based on the target text feature; obtaining a prediction result based on the target image feature and the target prompt feature; constructing a loss function based on the prediction result and the real result; adjusting the model parameters of the candidate text encoder based on the loss function, and fixing the model parameters of the pre-trained image encoder.

[0007] According to another aspect of the present disclosure, a data processing device based on MLLM is provided, wherein the MLLM includes: a pre-trained image encoder and multiple candidate text encoders, and the device includes: a first determination module, used to determine a target data type of target multimodal data; the multimodal data includes: a target image and a target prompt text; an image encoding module, used to encode the target image using the pre-trained image encoder to obtain target image features; a second determination module, used to determine a target text encoder corresponding to the target data type among the multiple candidate text encoders; a text encoding module, used to encode the target prompt text using the target text encoder to obtain target text features; a first acquisition module, used to acquire target prompt features based on the target text features; and a second acquisition module, used to acquire data processing results based on the target image features and the target prompt features.

[0008] According to another aspect of the present disclosure, an MLLM training device is provided, the MLLM comprising: a pre-trained image encoder and a candidate text encoder, the device comprising: a first acquisition module, used to acquire training data of the same data type as that of the candidate text encoder, the training data comprising: a target image sample, a target prompt text sample and a real result; an image encoding module, used to encode the target image sample using the pre-trained image encoder to obtain a target image feature; a text encoding module, used to encode the target prompt text sample using the candidate text encoder to obtain a target text feature; a second acquisition module, used to acquire a target prompt feature based on the target text feature; a third acquisition module, used to acquire a prediction result based on the target image feature and the target prompt feature; a construction module, used to construct a loss function based on the prediction result and the real result; an adjustment module, used to adjust the model parameters of the candidate text encoder based on the loss function, and fix the model parameters of the pre-trained image encoder.

[0009] According to another aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute any method as described in any of the above aspects.

[0010] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute any one of the methods according to any one of the above aspects.

[0011] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, wherein when the computer program is executed by a processor, the computer program implements any one of the methods described in any one of the above aspects.

[0012] The present disclosure can improve data processing performance.

[0013] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0015] Figure 1 is a schematic diagram according to a first embodiment of the present disclosure;

[0016] Figure 2 is a schematic diagram of an application scenario for implementing an embodiment of the present disclosure;

[0017] Figure 3 is a schematic diagram of data processing based on MLLM according to an embodiment of the present disclosure;

[0018] Figure 4 is a schematic diagram according to a second embodiment of the present disclosure;

[0019] Figure 5 is a schematic diagram according to a third embodiment of the present disclosure;

[0020] Figure 6 is a schematic diagram according to a fourth embodiment of the present disclosure;

[0021] Figure 7 is a schematic diagram according to a fifth embodiment of the present disclosure;

[0022] Figure 8 is a schematic diagram according to a sixth embodiment of the present disclosure;

[0023] Fig. 9 It is a schematic diagram of an electronic device used to implement the MLLM-based data processing method or MLLM training method of an embodiment of the present disclosure. DETAILED DESCRIPTION

[0024] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0025] In order to improve the data processing performance based on MLLM, the present disclosure provides the following embodiments.

[0026] Figure 1 is a schematic diagram according to the first embodiment of the present disclosure. This embodiment provides a data processing method based on MLLM, wherein the MLLM comprises: a pre-trained image encoder and multiple candidate text encoders, and the method comprises:

[0027] 101. Determine a target data type of target multimodal data; the multimodal data includes: a target image and a target prompt text.

[0028] 102. Use the pre-trained image encoder to encode the target image to obtain target image features.

[0029] 103. Determine a target text encoder corresponding to the target data type among the multiple candidate text encoders.

[0030] 104. Use the target text encoder to encode the target prompt text to obtain target text features.

[0031] 105. Obtain a target prompt feature based on the target text feature.

[0032] 106. Obtain a data processing result based on the target image feature and the target prompt feature.

[0033] Among them, MLLM introduces multimodal data on the basis of LLM. In the embodiment of the present disclosure, the multimodal data includes images and texts.

[0034] The multimodal data currently being processed is called target multimodal data, and the target multimodal data includes: a target image and a target prompt text.

[0035] Specifically, during data processing, the input of MLLM includes the target image and the target prompt text, and the output is the data processing result.

[0036] MLLM can be applied to target detection scenarios. In this scenario, the target image is the image to be detected, which includes one or more categories of objects. The target prompt text is used to describe the category of the object to be detected. The data processing result is the target detection result. For example, the target image is an image of a school, which includes a teaching building and a school gate. The target prompt text is a text similar to "identify the school gate". Then, after MLLM processing, the school gate in the target image can be obtained, such as marking the "school gate" with a rectangular frame.

[0037] MLLM can also be applied to other scenarios, such as generation scenarios. Specifically, if the target image contains one or more categories of objects, and the target prompt text is used to describe the modification of a certain category of objects to a specified content, then the data processing result is a generated image, and the generated image has the specified content. Alternatively, if the target prompt text is used to describe the generation of a paragraph of text based on the target image, then the data processing result is the text generated based on the target image and the target prompt text; or, if the target prompt text is used to describe the generation of a video based on the target image, then the data processing result is a video generated based on the target image and the target prompt text.

[0038] The training process of a model usually includes a pre-training process and a fine-tuning process. The corresponding models can be called pre-trained models and fine-tuned models, respectively. The pre-trained model is used for general tasks, and the fine-tuned model is obtained by adjusting the pre-trained model based on a specific task.

[0039] In order to retain the general capabilities of the pre-trained model, the image encoder uses a pre-training image encoder. The pre-trained image encoder is trained using a large number of existing training samples in various fields and has general capabilities.

[0040] After obtaining the target image, the target image is input into the pre-trained image encoder, and the output is the target image features.

[0041] After the above processing, the target image features of the corresponding image mode are obtained.

[0042] In addition to the above-mentioned image encoder, MLLM also includes a text encoder.

[0043] Multimodal data can be divided into multiple types. In order to improve data processing performance, the text encoder specifically includes multiple candidate text encoders. Each candidate text encoder is used to process text data of a data type, and the candidate text encoders corresponding to different data types are different.

[0044] The data type of the target multimodal data is called the target data type, and the candidate text encoder corresponding to the target data type is called the target text encoder. The correspondence between the data type and the candidate text encoder can be specifically configured, and the target text encoder corresponding to the target data type can be obtained based on the correspondence.

[0045] The target data type may be selected by the user, for example, the data processing system provides the user with a plurality of selectable data types, and the user selects one of the plurality of selectable data types as the target data type. Alternatively, the data processing system may analyze the target image and / or target prompt text input by the user to obtain the target data type.

[0046] After the target text encoder is determined, the target prompt text is encoded using the target text encoder to obtain the target text features. For example, the target prompt text is input into the target text encoder, and the output is the target text features.

[0047] After obtaining the target text features, the target text features can be used as target prompt features.

[0048] The above takes the prompt information (prompt) as the target prompt text as an example. In order to improve the data processing effect, the prompt information can also be multimodal, that is, the prompt information includes not only the above-mentioned target prompt text but also a prompt image. The feature corresponding to the prompt image is called the prompt image feature. Afterwards, the target prompt feature can be obtained based on the target text feature and the prompt image feature.

[0049] MLLM can also include LLM and decoder.

[0050] After obtaining the target image features and target prompt features, LLM can be used to encode them to obtain encoded features. For example, the target image features are mapped to the space where the target prompt features are located to obtain the mapped image features. After the mapped image features and the target prompt features are concatenated, they are used as the input of LLM. After being processed by LLM, the output features are called encoded features.

[0051] The encoded features are input into the decoder and the output is the data processing result.

[0052] In this embodiment, a target text encoder is determined from a plurality of candidate text encoders based on the target data type of the target multimodal data, and different candidate text encoders can be used to process target prompt texts of different data types, thereby realizing separate processing of multimodal data of different data types, which is conducive to improving data processing performance. In addition, using a pre-trained image encoder to encode the target image can retain the good performance of the pre-trained model and further improve data processing performance.

[0053] In order to better understand the embodiments of the present disclosure, application scenarios to which the embodiments of the present disclosure can be applied are described.

[0054] Figure 2 It is a schematic diagram of an application scenario for implementing the embodiment of the present disclosure. The scenario includes: a user terminal 201 and a server 202, and the user terminal 201 may include: a personal computer (PC), a mobile device (such as a mobile phone), a tablet computer, a laptop computer, a smart wearable device, etc. The server 202 may be a cloud server or a local server, and the user terminal 201 and the server 202 may communicate using a communication network, and the communication network may include, for example, a wired network and / or a wireless network.

[0055] The target application (Application, APP) is deployed on the user terminal, and the server corresponding to the APP is deployed on the server. The user enters the target image and the target prompt text used to describe the requirements through the APP. The APP sends the target image and target prompt text entered by the user to the server. The server uses the pre-configured MLLM to process the target image and target prompt text to obtain the data processing result. After that, the server can send the data processing result to the APP, which will be displayed to the user.

[0056] In addition, in some cases, the user can also input a prompt image. At this time, the server processes the target image, target prompt text and prompt image input by the user based on MLLM to obtain a data processing result.

[0057] Figure 3 It is a schematic diagram of data processing based on MLLM provided according to an embodiment of the present disclosure.

[0058] In this embodiment, the processing of the target image and the target prompt text based on MLLM is taken as an example.

[0059] like Figure 3 As shown, the MLLM includes: a pre-trained image encoder 301, a candidate text encoder 302, a LLM 303 and a decoder 304.

[0060] The input of the pre-trained image encoder 301 is the target image, and the output is the target image feature.

[0061] Multimodal data is divided into multiple data types. This embodiment takes three data types as examples, which are called general type, first special type and second special type respectively. There are three corresponding candidate text encoders 302. The first candidate text encoder is a pre-trained text encoder, the second candidate text encoder includes a text adapter and a pre-trained text encoder, and the third candidate text encoder is a fine-tuned text encoder.

[0062] During data processing, the target data type of the target multimodal data is determined, a target text encoder corresponding to the target data type is determined from a plurality of candidate text encoders, and then the target text encoder is used to encode the input target prompt text to output the target text features.

[0063] The input of LLM 303 is the processed multimodal features, and the output is the encoded features; Figure 3 In the figure, multimodal features including target image features and target text features are taken as an example. In specific implementation, the target image features and target text features can be processed and then input into the LLM. For example, the target image features are mapped to the space where the target text features are located to obtain the mapped image features, and the mapped image features and the target text features are spliced ​​and then input into the LLM.

[0064] The input of the decoder 304 is the encoded feature, and the output is the data processing result.

[0065] In combination with the above application scenarios, the present disclosure also provides the following embodiments.

[0066] Figure 4 is a schematic diagram according to the second embodiment of the present disclosure, which provides a data processing method based on MLLM. The MLLM includes: a pre-trained image encoder, multiple candidate text encoders, LLM and a decoder.

[0067] This embodiment takes the prompt information as the original prompt text as an example.

[0068] like Figure 4 As shown, the method of this embodiment includes:

[0069] 401. Determine a target data type of target multimodal data; the multimodal data includes: a target image and a target prompt text.

[0070] 402. Use a pre-trained image encoder to encode the target image to obtain target image features.

[0071] 403. Determine a target text encoder corresponding to the target data type among a plurality of candidate text encoders.

[0072] Taking three data types as an example, the process of determining a target text encoder from a plurality of candidate text encoders may include:

[0073] If the target data type belongs to a preset first special type, determining that the target text encoder includes: a text adapter and a pre-trained text encoder; or,

[0074] If the target data type belongs to a preset second special type, determining the target text encoder includes: fine-tuning the text encoder; or,

[0075] If the target data type belongs to a preset general type, determining the target text encoder includes: a pre-trained text encoder;

[0076] The text adapter is obtained by fine-tuning the first training sample corresponding to the first special type;

[0077] The fine-tuned text encoder is obtained by fine-tuning the second training sample corresponding to the second special type;

[0078] The number of the first training samples is smaller than the number of the second training samples.

[0079] Specifically, taking three data types as an example, these three data types can be respectively referred to as the first special type, the second special type and the general type. The first special type and the second special type are types of specific task data required by users, such as multimodal data for target detection scenarios, and the number of training samples of the first special type is less than the number of training samples of the second special type. For example, for target detection scenarios, it can include target detection applied to autonomous driving scenarios, or target detection applied to cultural relics scenes, etc. Generally speaking, the number of training samples corresponding to autonomous driving scenarios is more than the number of training samples corresponding to cultural relics scenes. For this reason, if the multimodal data to be processed is multimodal data of cultural relics scenes, then the target data type corresponding to the multimodal data is the first special type; or, if the multimodal data to be processed is multimodal data of autonomous driving scenes, then the target data type corresponding to the multimodal data is the second special type. In specific implementation, the specific types included in the first special type and the second special type are configurable. The specific types included in the general type are also configurable, or other types other than the first special type and the second special type are defaulted to the general type.

[0080] Among them, the correspondence between the data type and the candidate text encoder can be pre-configured, and based on the correspondence, the candidate text encoder corresponding to the target data type is obtained as the target text encoder. For example, the first special type can be pre-configured to include type A, type B, etc., and the second special type can be pre-configured to include type C, type D, etc. Based on this, if the target data type is type A, the target text encoder includes a text adapter and a pre-trained text encoder; if the target data type is type C, the target text encoder is a fine-tuned text encoder.

[0081] In addition, it is assumed that data types other than the first special type and the second special type are defaulted to general types. Based on this, if the target data type is type E, if type E belongs neither to the first special type nor to the second special type, then the target data type at this time belongs to the general type, and the target text encoder is a pre-trained text encoder.

[0082] The pre-trained text encoder is trained using a large number of existing training samples in various fields and has general capabilities.

[0083] The pre-trained text encoder and the pre-trained image encoder may specifically be a CLIP text encoder and a CLIP image encoder, respectively.

[0084] CLIP stands for Contrastive Language-Image Pretraining. CLIP uses contrastive learning to understand the relationship between paired text and images. Its goal is to minimize the difference between the corresponding image encoding and text encoding.

[0085] After the contrastive learning process, the CLIP image encoder and the CLIP text encoder can be obtained. Afterwards, the CLIP image encoder and the CLIP text encoder are used as the pre-trained image encoder and the pre-trained text encoder in the MLLM, respectively.

[0086] The text adapter is a small neural network model whose parameters are adjustable during the training process. Specifically, the first training sample can be used for adjustment during training.

[0087] During the training process, the model parameters of the text adapter can be adjusted while keeping other model parameters (such as the model parameters of the pre-trained image encoder and the model parameters of the pre-trained text encoder) unchanged. In this way, since the text adapter is small and has fewer parameters, the amount of parameters that need to be adjusted can be reduced, thereby improving training efficiency.

[0088] The fine-tuned text encoder is obtained by fine-tuning the pre-trained text encoder using task-specific training data.

[0089] Model parameters usually include weight parameters w and bias parameters b. For fine-tuning the text encoder, the parameter adjusted during the training process can specifically be the bias parameter b, while the weight parameter w is fixed. This can reduce the amount of parameters that need to be adjusted and improve training efficiency.

[0090] In this embodiment, different target text encoders are used for different data types, so that multimodal data of different data types can be separated and processed, thereby improving data processing performance; in addition, a text adapter and a pre-trained text encoder are used for the first special type with a small sample size, a fine-tuning text encoder is used for the second special type with a large sample size, and a pre-trained text encoder is used for general types other than the first special type and the second special type. Therefore, a suitable target text encoder can be selected based on the characteristics of different data types, thereby improving data processing performance.

[0091] In addition, for the first special type, the MLLM may further include an image adapter, which is similar to the text adapter and is fine-tuned using the first training sample. At this time, the image adapter is used to encode the target image to obtain the initial image features; and the pre-trained image encoder is used to encode the initial image features to obtain the target image features.

[0092] Among them, similar to the text adapter, the image adapter is also a small neural network model, which is applied before the pre-trained image encoder. The parameters of the image adapter are adjustable during the training process. Specifically, the parameters can be adjusted using the first training sample during training.

[0093] Based on this, for the first special type of multimodal data, the target image in the multimodal data is input to the image adapter, the output of the image adapter is the initial image feature, the initial image feature is input to the pre-trained image encoder, and the output of the pre-trained image encoder is the target image feature. The target prompt text in the multimodal data is input to the text adapter, the output of the text adapter is the initial text feature, the initial text feature is input to the pre-trained text encoder, and the output of the pre-trained text encoder is the target text feature. Afterwards, the data processing result is obtained based on the target image feature and the target text feature.

[0094] In this embodiment, the target image of the first special type is processed by using an image adapter and a pre-trained image encoder, which can enhance the representation capability of the target image features and improve the accuracy of data processing.

[0095] 404. Use the target text encoder to encode the target prompt text to obtain target text features.

[0096] Among them, the target prompt text can be input into the target text encoder, and the output is the target text feature.

[0097] 405. Use LLM to encode the target image features and the target text features to obtain encoding features.

[0098] For example, the target image features are mapped to the space where the target text features are located to obtain the mapped image features. The mapped image features and the target text features are concatenated as the input of the LLM. After LLM processing, the output of the LLM is the encoded features.

[0099] 406. Use a decoder to decode the encoding feature to obtain a data processing result.

[0100] Among them, the encoded features can be input into the decoder, and the output is the data processing result.

[0101] In this embodiment, the target image features and the target prompt features are encoded through LLM to obtain the encoded features, and then the data processing results are obtained based on the encoded features. The excellent performance of LLM can be used to improve the data processing effect.

[0102] In this embodiment, the target text encoder is determined from a plurality of candidate text encoders based on the target data type of the target multimodal data, and different candidate text encoders can be used to process target prompt texts of different data types, thereby realizing separate processing of multimodal data of different data types, which is conducive to improving data processing performance. In addition, the target image is encoded using a pre-trained image encoder, which can retain the good performance of the pre-trained model and further improve data processing performance. In addition, using the target prompt text as prompt information can easily and efficiently obtain data processing results.

[0103] The above example takes the prompt information being the original prompt text as an example. In some cases, the prompt information may also include a prompt image. For example, if the original prompt text is text for detecting a school gate, then the prompt image may be an image about the school gate, which is helpful for MLLM to detect the school gate in the target image.

[0104] Figure 5 is a schematic diagram according to the third embodiment of the present disclosure, which provides a data processing method based on MLLM. The MLLM includes: a pre-trained image encoder, multiple candidate text encoders, LLM and a decoder.

[0105] This embodiment takes the prompt information including the original prompt text and the prompt image as an example.

[0106] like Figure 5 As shown, the method of this embodiment includes:

[0107] 501. Determine a target data type of target multimodal data; the multimodal data includes: a target image and a target prompt text.

[0108] 502. Use a pre-trained image encoder to encode the target image to obtain target image features.

[0109] 503. Determine a target text encoder corresponding to the target data type among multiple candidate text encoders.

[0110] 504. Use the target text encoder to encode the target prompt text to obtain target text features.

[0111] For the specific contents of 501 to 504, please refer to 401 to 404.

[0112] 505. Use the pre-trained image encoder to encode the prompt image to obtain prompt image features.

[0113] After obtaining the prompt image input by the user, the prompt image can be input into a pre-trained image encoder, and the output is the prompt image feature.

[0114] 506. Obtain target prompt features based on the target text features and the prompt image features.

[0115] Among them, various related text feature and image feature fusion algorithms can be used to fuse the target text feature and the prompt image feature to obtain the target prompt feature.

[0116] For example, the MQ-Det model can be used to fuse the target text features and the prompt image features to obtain the target prompt features.

[0117] The MQ-Det model is a large model of multi-modal queried open-world target detection (Multi-modal Queried Object Detect in the wild), which can integrate multi-modal prompt information.

[0118] 507. Use LLM to encode the target image feature and the target prompt feature to obtain a coded feature.

[0119] For example, the target image features are mapped to the space where the target prompt features are located to obtain the mapped image features. The mapped image features and the target prompt features are concatenated as the input of the LLM. After LLM processing, the output of the LLM is the encoded features.

[0120] 508. Use a decoder to decode the encoding feature to obtain a data processing result.

[0121] Among them, the encoded features can be input into the decoder, and the output is the data processing result.

[0122] In this embodiment, the target image features and the target prompt features are encoded through LLM to obtain the encoded features, and then the data processing results are obtained based on the encoded features. The excellent performance of LLM can be used to improve the data processing effect.

[0123] In this embodiment, a target text encoder is determined from a plurality of candidate text encoders based on the target data type of the target multimodal data, and different candidate text encoders can be used to process target prompt texts of different data types, thereby realizing separate processing of multimodal data of different data types, which is conducive to improving data processing performance. In addition, by using a pre-trained image encoder to encode the target image, the good performance of the pre-trained model can be retained, further improving data processing performance. In addition, by using the target prompt text as prompt information, data processing results can be obtained simply and efficiently. In addition, by including a prompt image in the prompt information, prompt information of more modalities can be provided, further improving the accuracy of the data processing results.

[0124] Figure 6is a schematic diagram according to the fourth embodiment of the present disclosure. This embodiment provides an MLLM training method, the MLLM comprising: a pre-trained image encoder and a candidate text encoder, the method comprising:

[0125] 601. Acquire training data of the same data type as that of the candidate text encoder, wherein the training data includes: a target image sample, a target prompt text sample, and a true result.

[0126] 602. Use the pre-trained image encoder to encode the target image sample to obtain target image features.

[0127] 603. Use the candidate text encoder to encode the target prompt text sample to obtain target text features.

[0128] 604. Obtain a target prompt feature based on the target text feature.

[0129] 605. Obtain a prediction result based on the target image feature and the target prompt feature.

[0130] 606. Construct a loss function based on the predicted result and the actual result.

[0131] 607. Based on the loss function, adjust the model parameters of the candidate text encoder and fix the model parameters of the pre-trained image encoder.

[0132] The target image sample, the original prompt text sample and the true result are pre-acquired training data, and the type of the training data is consistent with the type of the candidate text encoder.

[0133] After obtaining the target image sample and the target prompt text sample, the model training process is similar to the model application process for processing these data, except that the model output obtained in the model application process is called the data processing result, and the model output in the training process is called the prediction result. For example, the target image sample is input into the pre-trained image encoder, and the output is the target image feature. The target prompt text sample is input into the candidate text encoder, and the output is the target text feature. After that, the target prompt feature is obtained based on the target text feature, and the prediction result is obtained based on the target image feature and the target prompt feature. In the model training process, after obtaining the prediction result, a loss function is constructed based on the prediction result and the real result obtained in advance, and the model parameters are adjusted based on the loss function. The adjusted model parameters are specifically the model parameters of the candidate text encoder, while the model parameters of the pre-trained image encoder are fixed.

[0134] In this embodiment, based on the candidate text encoders with the same data type as the training data, a suitable candidate text encoder can be selected for inference based on the target data type during the inference phase to improve data processing performance. In addition, by adjusting the model parameters of the candidate text encoder and fixing the model parameters of the pre-trained image encoder, the amount of parameters required to be adjusted can be reduced, thereby improving training efficiency.

[0135] In some embodiments, the candidate text encoder includes: a text adapter and a pre-trained text encoder; adjusting the model parameters of the candidate text encoder based on the loss function includes: adjusting the model parameters of the text adapter based on the loss function, and fixing the model parameters of the pre-trained text encoder.

[0136] The pre-trained text encoder is trained using a large number of existing training samples in various fields and has general capabilities.

[0137] The pre-trained text encoder and the pre-trained image encoder may specifically be a CLIP text encoder and a CLIP image encoder, respectively.

[0138] CLIP stands for Contrastive Language-Image Pretraining. CLIP uses contrastive learning to understand the relationship between paired text and images. Its goal is to minimize the difference between the corresponding image encoding and text encoding.

[0139] After the contrastive learning process, the CLIP image encoder and the CLIP text encoder can be obtained. Afterwards, the CLIP image encoder and the CLIP text encoder are used as the pre-trained image encoder and the pre-trained text encoder in the MLLM, respectively.

[0140] The text adapter is a small neural network model whose parameters are adjustable during the training process. Specifically, the first training sample can be used for adjustment during training.

[0141] During training, the model parameters of the text adapter can be adjusted while keeping other model parameters (such as the model parameters of the pre-trained image encoder and the model parameters of the pre-trained text encoder) unchanged.

[0142] In this embodiment, since the text adapter is small and has fewer parameters, the number of parameters required to be adjusted can be reduced, thereby improving training efficiency.

[0143] In some embodiments, the MLLM further comprises: an image adapter;

[0144] The method further comprises:

[0145] Based on the loss function, model parameters of the image adapter are adjusted.

[0146] Among them, similar to the text adapter, the image adapter is also a small neural network model, which is applied before the pre-trained image encoder. The parameters of the image adapter are adjustable during the training process. Specifically, the parameters can be adjusted using the first training sample during training.

[0147] Specifically, during the training process, the target image sample in the first training sample is input into the image adapter, and the output is the initial image feature, and the initial image feature is input into the pre-trained image encoder, and the output is the target image feature; the target prompt text sample in the first training sample is input into the text adapter, and the output is the initial text feature, and the initial text feature is input into the pre-trained text encoder, and the output is the target text feature; then, the target prompt feature is obtained based on the target text feature, and the prediction result is obtained based on the target image feature and the target prompt feature, and a loss function is constructed based on the prediction result and the true result, and then the model parameters of the image adapter and the model parameters of the text adapter are adjusted according to the loss function, while the model parameters of the pre-trained image encoder and the model parameters of the pre-trained text encoder are fixed.

[0148] In this embodiment, the image adapter and the pre-trained image encoder are used for processing, which can enhance the representation capability of target image features and improve the accuracy of data processing.

[0149] In some embodiments, the candidate text encoder includes: a fine-tuning text encoder; adjusting the model parameters of the candidate text encoder based on the loss function includes: adjusting the bias parameters in the model parameters of the fine-tuning text encoder based on the loss function, and fixing the weight parameters in the model parameters.

[0150] Among them, the fine-tuned text encoder is obtained by fine-tuning the pre-trained text encoder using training data for a specific task.

[0151] Specifically, the training data for the specific task of training the fine-tuning text encoder is the second training sample. At this time, the target image sample in the second training sample is input into the pre-trained image encoder, and the output is the target image feature; the target prompt text sample in the second training sample is input into the fine-tuning text encoder, and the output is the target text feature; thereafter, the target prompt feature is obtained based on the target text feature, the prediction result is obtained based on the target image feature and the target prompt feature, a loss function is constructed based on the prediction result and the true result, and then the model parameters of the fine-tuning text encoder are adjusted according to the loss function, while the model parameters of the pre-trained image encoder are fixed.

[0152] Among them, the initial values ​​of the model parameters of the fine-tuning text encoder can be selected as the model parameters of the pre-trained text encoder, so that fine-tuning can be performed on the basis of the pre-trained text encoder to improve the training efficiency and the effect of the fine-tuning text encoder.

[0153] When adjusting the model parameters of the fine-tuning text encoder, the bias parameter b in the fine-tuning text encoder may be adjusted, while the weight parameter w in the fine-tuning text encoder is fixed.

[0154] In this embodiment, by adjusting the bias parameters and fixing the weight parameters, the amount of parameters that need to be adjusted can be reduced and the training efficiency can be improved.

[0155] In some embodiments, obtaining the target prompt feature based on the target text feature includes:

[0156] Using the target text feature as the target prompt feature; or,

[0157] The pre-trained image encoder is used to encode the prompt image sample to obtain the prompt image feature; and the target prompt feature is obtained based on the target text feature and the prompt image feature.

[0158] Specifically, when the prompt sample only includes the target prompt text, the target text feature is used as the target prompt feature.

[0159] When the prompt sample includes a target prompt text and a prompt image sample, the prompt image feature is also obtained based on the prompt image sample, such as inputting the prompt image sample into a pre-trained image encoder, and the output is the prompt image feature; then a preset fusion algorithm is used to fuse the target text feature and the prompt image feature to obtain the target prompt feature.

[0160] In this embodiment, multimodal prompt information can be obtained by prompting image samples, which can improve the accuracy of the target detection model.

[0161] In some embodiments, the MLLM further comprises: an LLM and a decoder;

[0162] The obtaining of prediction results based on the target image features and the target prompt features includes:

[0163] Using the LLM, encoding the target image feature and the target prompt feature to obtain a coded feature;

[0164] The decoder is used to decode the encoded feature to obtain the prediction result.

[0165] For example, the target image features are mapped to the space where the target text features are located to obtain the mapped image features. The mapped image features and the target text features are concatenated as the input of the LLM. After LLM processing, the output of the LLM is the encoded features. The encoded features are input into the decoder, and the output is the prediction result.

[0166] In this embodiment, the target image features and the target prompt features are encoded through LLM to obtain the encoded features, and then the target detection results are obtained based on the encoded features. The excellent performance of LLM can be used to improve the effect of the target detection model.

[0167] In some embodiments, the method further comprises:

[0168] Based on the loss function, the model parameters of the decoder are adjusted, and the model parameters of the LLM are fixed.

[0169] Among them, after building the loss function in combination with the previous steps, the model parameters of the candidate text encoder and the model parameters of the decoder are adjusted based on the loss function, while the model parameters of the pre-trained image encoder and the model parameters of the LLM are fixed.

[0170] In this embodiment, by adjusting the model parameters of the decoder, the decoder can be fine-tuned, the decoder performance can be improved, and the performance of the target detection model can be improved. In addition, keeping the model parameters of the LLM unchanged can reduce the amount of calculation and improve processing efficiency.

[0171] Figure 7 704 , a first acquisition module 705 , and a second acquisition module 706 .

[0172] The first determination module 701 is used to determine the target data type of the target multimodal data; the multimodal data includes: a target image and a target prompt text; the image encoding module 702 is used to use the pre-trained image encoder to encode the target image to obtain target image features; the second determination module 703 is used to determine the target text encoder corresponding to the target data type among the multiple candidate text encoders; the text encoding module 704 is used to use the target text encoder to encode the target prompt text to obtain target text features; the first acquisition module 705 is used to obtain target prompt features based on the target text features; the second acquisition module 706 is used to obtain data processing results based on the target image features and the target prompt features.

[0173] In this embodiment, a target text encoder is determined from a plurality of candidate text encoders based on the target data type of the target multimodal data, and different candidate text encoders can be used to process target prompt texts of different data types, thereby realizing separate processing of multimodal data of different data types, which is conducive to improving data processing performance. In addition, using a pre-trained image encoder to encode the target image can retain the good performance of the pre-trained model and further improve data processing performance.

[0174] In some embodiments, the second determining module 703 is further configured to:

[0175] If the target data type belongs to a preset first special type, determining that the target text encoder includes: a text adapter and a pre-trained text encoder; or,

[0176] If the target data type belongs to a preset second special type, determining the target text encoder includes: fine-tuning the text encoder; or,

[0177] If the target data type belongs to a preset general type, determining the target text encoder includes: a pre-trained text encoder;

[0178] The text adapter is obtained by fine-tuning the first training sample corresponding to the first special type;

[0179] The fine-tuned text encoder is obtained by fine-tuning the second training sample corresponding to the second special type;

[0180] The number of the first training samples is smaller than the number of the second training samples.

[0181] In this embodiment, different target text encoders are used for different data types, so that multimodal data of different data types can be separated and processed, thereby improving data processing performance.

[0182] In some embodiments, the text adapter is obtained by fixing the model parameters of the pre-trained text encoder and adjusting the model parameters of the text adapter; and / or,

[0183] The fine-tuning text encoder is obtained by fixing the weight parameters in the model parameters of the pre-trained text encoder and adjusting the bias parameters in the model parameters.

[0184] In this embodiment, by adjusting the model parameters of the text adapter or fine-tuning the bias parameters of the text encoder, the amount of parameters required to be adjusted can be reduced and the training efficiency can be improved.

[0185] In some embodiments, the MLLM further comprises: an image adapter;

[0186] The image encoder 702 is further configured to:

[0187] The target image is encoded to obtain the initial image features;

[0188] Using the pre-trained image encoder, encoding the initial image features to obtain the target image features;

[0189] The image adapter is obtained by fine-tuning the first training sample.

[0190] In this embodiment, the target image of the first special type is processed by using an image adapter and a pre-trained image encoder, which can enhance the representation capability of the target image features and improve the accuracy of data processing.

[0191] In some embodiments, the first acquisition module 705 is further used to:

[0192] Using the target text feature as the target prompt feature; or,

[0193] The pre-trained image encoder is used to encode the prompt image to obtain prompt image features; and the target prompt features are obtained based on the target text features and the prompt image features.

[0194] In this embodiment, by using the target text feature as the target prompt feature, the data processing result can be obtained simply and efficiently; or, by including a prompt image in the prompt information, more modal prompt information can be provided, further improving the accuracy of the data processing result.

[0195] In some embodiments, the MLLM further comprises: an LLM and a decoder;

[0196] The second acquisition module 706 is further used for:

[0197] Using the LLM, encoding the target image feature and the target prompt feature to obtain a coded feature;

[0198] The decoder is used to decode the encoding feature to obtain the data processing result.

[0199] In this embodiment, the target image features and the target prompt features are encoded through LLM to obtain the encoded features, and then the data processing results are obtained based on the encoded features. The excellent performance of LLM can be used to improve the data processing effect.

[0200] Figure 880 is a schematic diagram according to the sixth embodiment of the present disclosure. This embodiment provides an MLLM training device, the MLLM comprising: a pre-trained image encoder and a candidate text encoder, the device comprising: a first acquisition module 801, an image encoding module 802, a text encoding module 803, a second acquisition module 804, a third acquisition module 805, a construction module 806 and an adjustment module 807.

[0201] The first acquisition module 801 is used to acquire training data of the same data type as the candidate text encoder, and the training data includes: target image samples, target prompt text samples and real results; the image encoding module 802 is used to use the pre-trained image encoder to encode the target image samples to obtain target image features; the text encoding module 803 is used to use the candidate text encoder to encode the target prompt text samples to obtain target text features; the second acquisition module 804 is used to obtain target prompt features based on the target text features; the third acquisition module 805 is used to obtain prediction results based on the target image features and the target prompt features; the construction module 806 is used to construct a loss function based on the prediction results and the real results. The adjustment module 807 is used to adjust the model parameters of the candidate text encoder according to the loss function, and fix the model parameters of the pre-trained image encoder.

[0202] In this embodiment, based on the candidate text encoders with the same data type as the training data, a suitable candidate text encoder can be selected for inference based on the target data type during the inference phase to improve data processing performance. In addition, by adjusting the model parameters of the candidate text encoder and fixing the model parameters of the pre-trained image encoder, the amount of parameters required to be adjusted can be reduced, thereby improving training efficiency.

[0203] In some embodiments, the candidate text encoders include: a text adapter and a pre-trained text encoder;

[0204] The adjustment module 807 is further used for:

[0205] Based on the loss function, model parameters of the text adapter are adjusted, and model parameters of the pre-trained text encoder are fixed.

[0206] In this embodiment, since the text adapter is small and has fewer parameters, the number of parameters required to be adjusted can be reduced, thereby improving training efficiency.

[0207] In some embodiments, the MLLM further comprises: an image adapter;

[0208] The adjustment module 807 is also used for:

[0209] Based on the loss function, model parameters of the image adapter are adjusted.

[0210] In this embodiment, the image adapter and the pre-trained image encoder are used for processing, which can enhance the representation capability of target image features and improve the accuracy of data processing.

[0211] In some embodiments, the candidate text encoder includes: a fine-tuning text encoder;

[0212] The adjustment module 807 is further used for:

[0213] Based on the loss function, bias parameters in the model parameters of the fine-tuning text encoder are adjusted, and weight parameters in the model parameters are fixed.

[0214] In this embodiment, by adjusting the bias parameters and fixing the weight parameters, the amount of parameters that need to be adjusted can be reduced and the training efficiency can be improved.

[0215] In some embodiments, the second acquisition module 804 is further used to:

[0216] Using the target text feature as the target prompt feature; or,

[0217] The pre-trained image encoder is used to encode the prompt image sample to obtain the prompt image feature; and the target prompt feature is obtained based on the target text feature and the prompt image feature.

[0218] In this embodiment, multimodal prompt information can be obtained by prompting image samples, which can improve the accuracy of the target detection model.

[0219] In some embodiments, the MLLM further comprises: an LLM and a decoder;

[0220] The third acquisition module 805 is further used for:

[0221] Using the LLM, encoding the target image feature and the target prompt feature to obtain a coded feature;

[0222] The decoder is used to decode the encoded feature to obtain the prediction result.

[0223] In this embodiment, the target image features and the target prompt features are encoded through LLM to obtain the encoded features, and then the target detection results are obtained based on the encoded features. The excellent performance of LLM can be used to improve the effect of the target detection model.

[0224] In some embodiments, the adjustment module 807 is further configured to:

[0225] Based on the loss function, the model parameters of the decoder are adjusted, and the model parameters of the LLM are fixed.

[0226] In this embodiment, by adjusting the model parameters of the decoder, the decoder can be fine-tuned, the decoder performance can be improved, and the performance of the target detection model can be improved. In addition, keeping the model parameters of the LLM unchanged can reduce the amount of calculation and improve processing efficiency.

[0227] It can be understood that in the embodiments of the present disclosure, the same or similar contents in different embodiments can be referenced to each other.

[0228] It can be understood that the “first”, “second”, etc. in the embodiments of the present disclosure are only used for distinction and do not indicate the degree of importance, time sequence, etc.

[0229] It is understandable that, unless otherwise specified, the order of the steps in the process indicates that the timing relationship between these steps is not limited.

[0230] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0231] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0232] Fig. 9 A schematic block diagram of an example electronic device 900 that can be used to implement an embodiment of the present disclosure is shown. The electronic device 900 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0233] like Fig. 9 As shown, the electronic device 900 includes a computing unit 901, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 902 or a computer program loaded from a storage unit 908 into a random access memory (RAM) 903. In the RAM 903, various programs and data required for the operation of the electronic device 900 can also be stored. The computing unit 901, the ROM 902, and the RAM 903 are connected to each other via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0234] Multiple components in the electronic device 900 are connected to the I / O interface 905, including: an input unit 906, such as a keyboard, a mouse, etc.; an output unit 907, such as various types of displays, speakers, etc.; a storage unit 908, such as a disk, an optical disk, etc.; and a communication unit 909, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 909 allows the electronic device 900 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0235] The computing unit 901 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 901 performs the various methods and processes described above, such as a data processing method or an MLLM training method based on MLLM. For example, in some embodiments, a data processing method or an MLLM training method based on MLLM may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 908. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 900 via ROM 902 and / or a communication unit 909. When the computer program is loaded into RAM 903 and executed by the computing unit 901, one or more steps of the data processing method or the MLLM training method based on MLLM described above may be performed. Alternatively, in other embodiments, the computing unit 901 may be configured to execute the MLLM-based data processing method or the MLLM training method in any other appropriate manner (eg, by means of firmware).

[0236] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0237] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable information recommendation device, so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine as a stand-alone software package and partially on a remote machine, or entirely on a remote machine or server.

[0238] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0239] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0240] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0241] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services ("Virtual Private Server", or "VPS" for short). The server may also be a server of a distributed system, or a server combined with a blockchain.

[0242] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0243] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A data processing method based on MLLM, the MLLM comprising: Pre-training an image encoder and multiple candidate text encoders, the method comprising: Determine a target data type of target multimodal data; the multimodal data includes: a target image and a target prompt text; Using the pre-trained image encoder, encoding the target image to obtain target image features; Determine a target text encoder corresponding to the target data type among the plurality of candidate text encoders; Using the target text encoder, encoding the target prompt text to obtain target text features; Acquire a target prompt feature based on the target text feature; Based on the target image features and the target prompt features, a data processing result is obtained.

2. The method according to claim 1, wherein: Determining a target text encoder corresponding to the target data type among the plurality of candidate text encoders comprises: If the target data type belongs to a preset first special type, determining that the target text encoder includes: a text adapter and a pre-trained text encoder; or, If the target data type belongs to a preset second special type, determining the target text encoder includes: fine-tuning the text encoder; or, If the target data type belongs to a preset general type, determining the target text encoder includes: a pre-trained text encoder; The text adapter is obtained by fine-tuning the first training sample corresponding to the first special type; The fine-tuned text encoder is obtained by fine-tuning the second training sample corresponding to the second special type; The number of the first training samples is smaller than the number of the second training samples.

3. The method according to claim 2, wherein: The text adapter is obtained by fixing the model parameters of the pre-trained text encoder and adjusting the model parameters of the text adapter; and / or, The fine-tuning text encoder is obtained by fixing the weight parameters in the model parameters of the pre-trained text encoder and adjusting the bias parameters in the model parameters.

4. The method according to claim 2, wherein: The MLLM also includes: an image adapter; The method of encoding the target image using a pre-trained image encoder to obtain target image features includes: If the target data type belongs to the first dedicated type, encoding the target image using the image adapter to obtain initial image features; Using the pre-trained image encoder, encoding the initial image features to obtain the target image features; The image adapter is obtained by fine-tuning the first training sample.

5. The method according to claim 1, wherein: The step of acquiring a target prompt feature based on the target text feature includes: Using the target text feature as the target prompt feature; or, The pre-trained image encoder is used to encode the prompt image to obtain prompt image features; and the target prompt features are obtained based on the target text features and the prompt image features.

6. The method according to claim 1, wherein: The MLLM also includes: an LLM and a decoder; The obtaining of data processing results based on the target image features and the target prompt features includes: Using the LLM, encoding the target image feature and the target prompt feature to obtain a coded feature; The decoder is used to decode the encoding feature to obtain the data processing result.

7. A MLLM training method, the MLLM comprising: Pre-training an image encoder and a candidate text encoder, the method comprising: Acquire training data of the same data type as that of the candidate text encoder, the training data comprising: a target image sample, a target prompt text sample, and a true result; Using the pre-trained image encoder, encoding the target image sample to obtain target image features; Using the candidate text encoder, encoding the target prompt text sample to obtain target text features; Acquire a target prompt feature based on the target text feature; Based on the target image feature and the target prompt feature, obtaining a prediction result; Constructing a loss function based on the predicted result and the actual result; Based on the loss function, the model parameters of the candidate text encoder are adjusted, and the model parameters of the pre-trained image encoder are fixed.

8. The method according to claim 7, wherein: The candidate text encoders include: a text adapter and a pre-trained text encoder; The adjusting the model parameters of the candidate text encoder based on the loss function includes: Based on the loss function, model parameters of the text adapter are adjusted, and model parameters of the pre-trained text encoder are fixed.

9. The method according to claim 8, wherein: The MLLM also includes: an image adapter; The method further comprises: Based on the loss function, model parameters of the image adapter are adjusted.

10. The method according to claim 7, wherein: The candidate text encoder includes: a fine-tuning text encoder; The adjusting the model parameters of the candidate text encoder based on the loss function includes: Based on the loss function, bias parameters in the model parameters of the fine-tuning text encoder are adjusted, and weight parameters in the model parameters are fixed.

11. The method according to claim 7, wherein: The step of acquiring the target prompt feature based on the target text feature includes: Using the target text feature as the target prompt feature; or, The pre-trained image encoder is used to encode the prompt image sample to obtain the prompt image feature; and the target prompt feature is obtained based on the target text feature and the prompt image feature.

12. The method according to claim 7, wherein: The MLLM also includes: an LLM and a decoder; The obtaining of prediction results based on the target image features and the target prompt features includes: Using the LLM, encoding the target image feature and the target prompt feature to obtain a coded feature; The decoder is used to decode the encoded feature to obtain the prediction result.

13. The method according to claim 11, further comprising: Based on the loss function, the model parameters of the decoder are adjusted, and the model parameters of the LLM are fixed.

14. A data processing device based on MLLM, the MLLM comprising: Pre-trained image encoder and multiple candidate text encoders, the device comprises: A first determination module is used to determine a target data type of target multimodal data; the multimodal data includes: a target image and a target prompt text; An image encoding module, used to encode the target image using the pre-trained image encoder to obtain target image features; A second determination module is used to determine a target text encoder corresponding to the target data type among the multiple candidate text encoders; A text encoding module, used to encode the target prompt text using the target text encoder to obtain target text features; A first acquisition module, used for acquiring a target prompt feature according to the target text feature; The second acquisition module is used to acquire data processing results according to the target image features and the target prompt features.

15. The device according to claim 14, wherein: The second determining module is further used for: If the target data type belongs to a preset first special type, determining that the target text encoder includes: a text adapter and a pre-trained text encoder; or, If the target data type belongs to a preset second special type, determining the target text encoder includes: fine-tuning the text encoder; or, If the target data type belongs to a preset general type, determining the target text encoder includes: a pre-trained text encoder; The text adapter is obtained by fine-tuning the first training sample corresponding to the first special type; The fine-tuned text encoder is obtained by fine-tuning the second training sample corresponding to the second special type; The number of the first training samples is smaller than the number of the second training samples.

16. The device according to claim 15, wherein: The text adapter is obtained by fixing the model parameters of the pre-trained text encoder and adjusting the model parameters of the text adapter; and / or, The fine-tuning text encoder is obtained by fixing the weight parameters in the model parameters of the pre-trained text encoder and adjusting the bias parameters in the model parameters.

17. The device according to claim 15, wherein: The MLLM also includes: an image adapter; The image encoder is further configured to: The target image is encoded to obtain the initial image features; Using the pre-trained image encoder, encoding the initial image features to obtain the target image features; The image adapter is obtained by fine-tuning the first training sample.

18. The device according to claim 14, wherein: The first acquisition module is further used for: Using the target text feature as the target prompt feature; or, Using the pre-trained image encoder, encode the prompt image to obtain prompt image features; And, based on the target text feature and the prompt image feature, the target prompt feature is obtained.

19. The device according to claim 14, wherein: The MLLM also includes: an LLM and a decoder; The third acquisition module is further used for: Using the LLM, encoding the target image feature and the target prompt feature to obtain a coded feature; The decoder is used to decode the encoding feature to obtain the data processing result.

20. A MLLM training device, the MLLM comprising: A pre-trained image encoder and a candidate text encoder, the apparatus comprising: A first acquisition module is used to acquire training data of the same data type as the candidate text encoder, wherein the training data includes: a target image sample, a target prompt text sample and a real result; An image encoding module, used to encode the target image sample using the pre-trained image encoder to obtain target image features; A text encoding module, used to encode the target prompt text sample using the candidate text encoder to obtain target text features; A second acquisition module, used for acquiring a target prompt feature according to the target text feature; A third acquisition module, used for acquiring a prediction result according to the target image feature and the target prompt feature; A construction module, used to construct a loss function according to the predicted result and the actual result; An adjustment module is used to adjust the model parameters of the candidate text encoder according to the loss function and fix the model parameters of the pre-trained image encoder.

21. The device according to claim 20, wherein: The candidate text encoders include: a text adapter and a pre-trained text encoder; The adjustment module is further used for: Based on the loss function, model parameters of the text adapter are adjusted, and model parameters of the pre-trained text encoder are fixed.

22. The device according to claim 21, wherein The MLLM also includes: an image adapter; The adjustment module is also used for: Based on the loss function, model parameters of the image adapter are adjusted.

23. The device according to claim 20, wherein: The candidate text encoder includes: a fine-tuning text encoder; The adjustment module is further used for: Based on the loss function, bias parameters in the model parameters of the fine-tuning text encoder are adjusted, and weight parameters in the model parameters are fixed.

24. The device according to claim 20, wherein: The second acquisition module is further used for: Using the target text feature as the target prompt feature; or, Using the pre-trained image encoder, encode the prompt image sample to obtain the prompt image feature; And, based on the target text feature and the prompt image feature, the target prompt feature is obtained.

25. The device according to claim 20, wherein: The MLLM also includes: an LLM and a decoder; The second acquisition module is further used for: Using the LLM, encoding the target image feature and the target prompt feature to obtain a coded feature; The decoder is used to decode the encoded feature to obtain the prediction result.

26. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 13.

27. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-13.

28. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 13.

Citation Information

Cited By

  • Cross-document type retrieval method and system based on differentiated nested coding

    CN121579664A

  • A cross-document type retrieval method and system based on differential nested encoding

    CN121579664B