Data processing method, image-text data processing method, computing device and storage medium
By combining the first multimodal large model, the second multimodal large model, and the contrastive learning model, the problem of knowledge illusion in the processing of image and text data by multimodal large models is solved, and accurate image and text processing results are generated to meet the processing needs of real-world scenarios.
Patent Information
- Application Number
- CN202410585050.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-05-11
- Publication Date
- 2025-11-11
AI Technical Summary
Existing multimodal large models are prone to generating knowledge illusions during the processing of text and image data, producing information that is irrelevant to or incorrect to the input data, and thus failing to meet the processing needs of real-world scenarios.
A combined approach is adopted, which includes a first multimodal large model, a second multimodal large model, and a contrastive learning model. The contrastive learning model is used to process the first and second image-text processing results to generate accurate image-text processing results.
It effectively avoids the knowledge illusion problem of multimodal large models, ensures that the generated information corresponds to and is correct with the input data, and meets the needs of image and text data processing in real-world scenarios.
Smart Images

Figure CN120930037A_ABST
Abstract
Description
Technical Field
[0001] The embodiments in this specification relate to the field of machine learning technology, and in particular to a method for processing image and text data. Background Technology
[0002] With the continuous development of artificial intelligence technology and the widespread application of neural network models, many institutions, enterprises and other organizations can train multimodal large models according to the actual needs of image data processing in real-world scenarios, thereby obtaining multimodal large models for image and text data processing, which can then be used to process users' image and text data in real-world scenarios.
[0003] However, existing multimodal large models suffer from knowledge illusion during image and text data processing, causing them to generate information that is irrelevant or incorrect to the input image and text data. As a result, they cannot process the image and text data of users in real-world scenarios, and thus cannot meet the actual needs of image data processing in real-world scenarios. Summary of the Invention
[0004] In view of this, embodiments of this specification provide a method for processing text and image data. One or more embodiments of this specification simultaneously relate to a data processing method, a data processing model training method, a machine learning model training method, another data processing method, a text and image data processing apparatus, a data processing apparatus, a data processing model training apparatus, another data processing apparatus, a machine learning model training apparatus, a computing device, a computer-readable storage medium, and a computer program product, to address the technical deficiencies in the prior art where multimodal large models generate irrelevant or erroneous information from the input text and image data, thus failing to process user text and image data in real-world scenarios.
[0005] According to a first aspect of the embodiments of this specification, a method for processing graphic and textual data is provided, including:
[0006] The text and image data to be processed is determined, and the text and image data to be processed is input into the data processing model. The data processing model includes a second multimodal large model, a third multimodal large model, and a contrastive learning model. The first multimodal large model is determined based on the second multimodal large model.
[0007] The first multimodal large model is used to process the image and text data to be processed to obtain the first image and text processing result;
[0008] The second multimodal large model is used to process the image and text data to be processed to obtain the second image and text processing result;
[0009] The first and second image processing results are processed using the contrastive learning model to obtain the image processing result corresponding to the image data to be processed.
[0010] According to a second aspect of the embodiments of this specification, a graphic data processing apparatus is provided, comprising:
[0011] The data determination module is configured to determine the text and image data to be processed and input the text and image data to be processed into the data processing model, wherein the data processing model includes a first multimodal large model, a second multimodal large model and a contrastive learning model, and the first multimodal large model is determined based on the second multimodal large model;
[0012] The first data processing module is configured to process the image and text data to be processed using the first multimodal large model to obtain a first image and text processing result;
[0013] The second data processing module is configured to process the image and text data to be processed using the second multimodal large model to obtain the second image and text processing result;
[0014] The result determination module is configured to use the contrastive learning model to process the first image and text processing result and the second image and text processing result to obtain the image and text processing result corresponding to the image and text data to be processed.
[0015] According to a third aspect of the embodiments of this specification, a data processing method is provided, comprising:
[0016] The data to be processed is determined and input into the data processing model, wherein the data processing model includes a first data processing module, a second data processing module and a parameter learning module, and the first data processing module is determined based on the second data processing module;
[0017] The first data processing module is used to process the data to be processed to obtain a first data processing result;
[0018] The second data processing module is used to process the data to be processed to obtain a second data processing result;
[0019] The parameter learning module is used to process the first data processing result and the second data processing result to obtain the data processing result corresponding to the data to be processed.
[0020] According to a fourth aspect of the embodiments of this specification, a data processing apparatus is provided, comprising:
[0021] A data determination module is configured to determine the data to be processed and input the data to be processed into a data processing model, wherein the data processing model includes a first data processing module, a second data processing module and a parameter learning module, and the first data processing module is determined based on the second data processing module;
[0022] The first data processing module is configured to process the data to be processed using the first data processing module to obtain a first data processing result;
[0023] The second data processing module is configured to process the data to be processed using the second data processing module to obtain a second data processing result;
[0024] The result determination module is configured to use the parameter learning module to process the first data processing result and the second data processing result to obtain the data processing result corresponding to the data to be processed.
[0025] According to a fifth aspect of the embodiments of this specification, a machine learning model training method is provided, comprising:
[0026] The machine learning model to be trained is determined, as well as the first training data and the second training data associated with the target task, wherein the machine learning model to be trained includes a first data processing module, a second data processing module and a parameter learning module.
[0027] Based on the first training data, train the first data processing module to obtain the trained first data processing module.
[0028] The second training data is processed according to the first data processing module, the second data processing module, and the parameter learning module after training to obtain the processing result;
[0029] Based on the processing results, the model parameters of the first data processing module and the parameter learning module after training are adjusted to obtain the trained target model, wherein the target model is used to perform the target task.
[0030] According to a sixth aspect of the embodiments of this specification, a machine learning model training apparatus is provided, comprising:
[0031] The training data determination module is configured to determine the machine learning model to be trained, as well as the first training data and the second training data associated with the target task, wherein the machine learning model to be trained includes a first data processing module, a second data processing module, and a parameter learning module.
[0032] The first model training module is configured to train the first data processing module based on the first training data to obtain the trained first data processing module.
[0033] The processing result determination module is configured to process the second training data according to the first training data processing module, the second data processing module, and the parameter learning module after training, and obtain the processing result;
[0034] The second model training module is configured to adjust the model parameters of the trained first data processing module and the parameter learning module according to the processing results, so as to obtain the trained target model, wherein the target model is used to perform the target task.
[0035] According to a seventh aspect of the embodiments of this specification, a data processing model training method is provided, comprising:
[0036] The data processing model to be trained is determined, as well as the first image and text training data and the second image and text training data associated with the data processing task. The data processing model to be trained includes a first multimodal large model, a second multimodal large model and a contrastive learning model, and the first multimodal large model is determined based on the second multimodal large model.
[0037] Based on the first image and text training data, train the first multimodal large model to obtain the trained first multimodal large model;
[0038] Based on the trained first multimodal large model, the second multimodal large model, and the contrastive learning model, the second image and text training data is processed to obtain the processing result;
[0039] Based on the processing results, the model parameters of the trained first multimodal large model and the contrastive learning model are adjusted to obtain a trained data processing model, wherein the data processing model is used to perform the data processing task.
[0040] According to an eighth aspect of the embodiments of this specification, a data processing model training apparatus is provided, comprising:
[0041] The training data determination module is configured to determine the data processing model to be trained, as well as the first image and text training data and the second image and text training data associated with the data processing task. The data processing model to be trained includes a first multimodal large model, a second multimodal large model and a contrastive learning model, and the first multimodal large model is determined based on the second multimodal large model.
[0042] The first model training module is configured to train the first multimodal large model based on the first image and text training data, and obtain the trained first multimodal large model.
[0043] The processing result determination module is configured to process the second image and text training data based on the trained first multimodal large model, the second multimodal large model, and the contrastive learning model to obtain the processing result;
[0044] The second model training module is configured to adjust the model parameters of the trained first multimodal large model and the contrastive learning model based on the processing results, so as to obtain a trained data processing model, wherein the data processing model is used to perform the data processing task.
[0045] According to a ninth aspect of the embodiments of this specification, another data processing method is provided, including:
[0046] Determine the text and image data to be processed by the user;
[0047] The text and image data to be processed is input into a data processing model to obtain the data processing text corresponding to the text and image data to be processed. The data processing model includes a first multimodal large model, a second multimodal large model, and a contrastive learning model. The first multimodal large model is determined based on the second multimodal large model. The first multimodal large model is used to process the text and image data to be processed to obtain a first text and image processing result. The second multimodal large model is used to process the text and image data to be processed to obtain a second text and image processing result. The contrastive learning model is used to process the first text and image processing result and the second text and image processing result to obtain the data processing text corresponding to the text and image data to be processed.
[0048] According to a tenth aspect of the embodiments of this specification, another data processing apparatus is provided, comprising:
[0049] The data determination module is configured to determine the text and image data to be processed input by the user;
[0050] The data processing module is configured to input the text and image data to be processed into a data processing model to obtain the data processing text corresponding to the text and image data to be processed. The data processing model includes a first multimodal large model, a second multimodal large model, and a contrastive learning model. The first multimodal large model is determined based on the second multimodal large model. The first multimodal large model is used to process the text and image data to be processed to obtain a first text and image processing result. The second multimodal large model is used to process the text and image data to be processed to obtain a second text and image processing result. The contrastive learning model is used to process the first text and image processing result and the second text and image processing result to obtain the data processing text corresponding to the text and image data to be processed.
[0051] According to an eleventh aspect of the embodiments of this specification, a computing device is provided, comprising:
[0052] Memory and processor;
[0053] The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions. When the computer programs / instructions are executed by the processor, they implement the steps of the above-described image and text data processing method, data processing method, machine learning model training method, data processing model training method, or another data processing method.
[0054] According to an eighth aspect of the embodiments of this specification, a computer-readable storage medium is provided that stores a computer program / instructions, which, when executed by a processor, implement the steps of the above-described image and text data processing method, a data processing method, a machine learning model training method, a data processing model training method, or another data processing method.
[0055] According to a ninth aspect of the embodiments of this specification, a computer program product is provided, including a computer program / instructions that, when executed by a processor, implement the above-described image and text data processing method, a data processing method, a machine learning model training method, a data processing model training method, or another data processing method.
[0056] The image and text data processing method in one or more embodiments provided in this specification, after determining the image and text data to be processed, inputs it into a data processing model, uses a first multimodal large model and a second multimodal large model to process the image and text data to be processed, and obtains a first image and text processing result and a second image and text processing result. Then, a contrastive learning model is used to perform contrastive learning processing based on the first image and text processing result and the second image and text processing result, thereby obtaining an image and text processing result that accurately corresponds to the image and text data to be processed. This avoids the problem of knowledge illusion in multimodal large models, solves the problem of multimodal large models generating information that is irrelevant or incorrect to the input image and text data, and further solves the problem of not being able to process user image and text data in actual scenarios, thus meeting the actual needs of image data processing in real scenarios. Attached Figure Description
[0057] Figure 1 This is a schematic diagram illustrating the application of a data processing method provided in one embodiment of this specification;
[0058] Figure 2 This is a flowchart illustrating a data processing model training method provided in one embodiment of this specification;
[0059] Figure 3 This is a flowchart illustrating the processing procedure of a data processing model training method provided in one embodiment of this specification.
[0060] Figure 4 This is a flowchart illustrating a machine learning model training method provided in one embodiment of this specification;
[0061] Figure 5 This is a flowchart of another data processing method provided in one embodiment of this specification;
[0062] Figure 6 This is a flowchart illustrating a graphic data processing method provided in one embodiment of this specification;
[0063] Figure 7 This is a flowchart illustrating a data processing method provided in one embodiment of this specification;
[0064] Figure 8 This is a structural block diagram of a computing device provided in one embodiment of this specification. Detailed Implementation
[0065] Many specific details are set forth in the following description to provide a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar extensions without departing from the spirit of this specification. Therefore, this specification is not limited to the specific implementations disclosed below.
[0066] The terminology used in one or more embodiments of this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the one or more embodiments of this specification. The singular forms “a,” “described,” and “the” as used in one or more embodiments of this specification and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.
[0067] It should be understood that although the terms first, second, etc., may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first may also be referred to as second, and similarly, third may also be referred to as first, without departing from the scope of one or more embodiments of this specification. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."
[0068] Furthermore, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in one or more embodiments of this specification are all information and data authorized by the user or fully authorized by all parties. Moreover, the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.
[0069] In one or more embodiments of this specification, a large model refers to a deep learning model with a large number of model parameters, typically containing hundreds of millions, tens of billions, hundreds of billions, trillions, or even tens of trillions of model parameters. A large model can also be called a foundation model. It is pre-trained using large-scale unlabeled corpora to produce a pre-trained model with hundreds of millions of parameters. Such models can adapt to a wide range of downstream tasks and have good generalization ability. Examples include Large Language Models (LLMs) and multi-modal pre-training models.
[0070] In practical applications, large models only require a small number of samples to fine-tune the pre-trained model before they can be applied to different tasks. Large models can be widely used in fields such as Natural Language Processing (NLP) and Computer Vision. Specifically, they can be applied to computer vision tasks such as Visual Question Answering (VQA), Image Captioning (IC), and Image Generation, as well as NLP tasks such as text-based sentiment classification, text summarization, and machine translation. The main application scenarios for large models include digital assistants, intelligent robots, search, online education, office software, e-commerce, and intelligent design.
[0071] First, the terms and concepts used in one or more embodiments of this specification will be explained.
[0072] Large Vision-Language Models (LVLMs) are artificial intelligence models that combine visual and language processing capabilities. In other words, these LVLMs can simultaneously handle large-scale parameters of both text and images. They can understand and process both image and text information, enabling cross-modal learning and reasoning, and have wide applications in various fields, such as image description generation, visual question answering, visual reasoning, multimedia retrieval, and multimodal dialogue systems.
[0073] Hallucinations refer to situations where a model, when processing visual and linguistic information, may generate seemingly reasonable but actually inaccurate or unrealistic results. For example, when fusing visual and linguistic information, the model may produce inconsistent outputs, such as generating text descriptions that do not match the actual content of the input image, generating answers or descriptions that are inconsistent with the actual context, or generating information that is irrelevant to or incorrect about the given text and image inputs.
[0074] Large Language Models (LLMs) are large-scale models for solving text understanding problems.
[0075] The rapid development of LLMs in recent years marks a significant step towards achieving Artificial General Intelligence (AGI). LLMs have demonstrated impressive performance in the vision domain across various tasks, such as image annotation and localization. Despite their remarkable success, LLMs still face many challenges in practical applications. Among these challenges, the phenomenon of knowledge illusion is particularly critical, its frequent occurrence raising widespread concerns about the safety and stability of artificial intelligence systems. In the field of vision-language models, the term 'knowledge illusion' refers to a model generating information that is irrelevant or erroneous to the given text and image input.
[0076] To address the problem of knowledge illusion, this specification provides a solution that can mitigate the illusion by fine-tuning LVLMs on a fine-grained dataset, or by using an auxiliary network to identify the illusionary object or rewrite the text output by the LVLMs. However, this solution requires a large amount of additional model parameters or training data, resulting in low model training efficiency. Furthermore, large-scale vision-language models (LVLMs) often suffer from an issue where, when generating text responses, the model inappropriately relies on textual information, neglecting image content and producing illusionary text irrelevant to the image. To mitigate this language-biased illusion problem, this technical solution often requires introducing additional networks or extensive parameter tuning, which not only increases the model's complexity but also the demand for computational resources.
[0077] Based on this, this specification provides a data processing model training method. This specification also relates to a machine learning model training method, a data processing method, a data processing model training device, a data processing device, a machine learning model training device, a computing device, a computer-readable storage medium, and a computer program product, which will be described in detail in the following embodiments.
[0078] See Figure 1 , Figure 1 This diagram illustrates an application illustration of a data processing method according to an embodiment of this specification, based on... Figure 1As can be seen, the user sends image-text question data containing images and text to the server 104 via terminal 102. After receiving the image-text question data, the server 104 inputs the image-text question data into a data processing model. The data processing model processes the image-text question data as follows: First, it inputs the image and text sent by the user into a regular LVLMs model in the data processing model to obtain a regular prediction result (tokens sorted by time step); second, it inputs the image and text sent by the user into a biased image LVLMs model in the data processing model to obtain a biased image prediction result (tokens sorted by time step); finally, it inputs the regular prediction results and the biased image prediction results output by the two models into a contrastive learning network in the data processing model for comparison to obtain a similarity score; this similarity score guides the two models to predict the next token until the complete answer text is obtained. After obtaining the answer text through the data processing model, the server 104 sends the answer text to terminal 102 to display it to the user.
[0079] See Figure 2 , Figure 2 A flowchart of a data processing model training method according to an embodiment of this specification is shown, which specifically includes the following steps.
[0080] Step 202: Determine the data processing model to be trained, as well as the first and second image-text training data associated with the data processing task. The data processing model to be trained includes a first multimodal large model, a second multimodal large model, and a contrastive learning model, and the first multimodal large model is determined based on the second multimodal large model.
[0081] The data processing model can be understood as a model capable of processing input image and text data and outputting processed text corresponding to that data. The data processing model provided in this specification varies depending on the application scenario. When the data processing model training method provided in one or more embodiments of this specification is applied to an image-text question-and-answer scenario, the data processing model can be an image-text question-and-answer model. Correspondingly, the image and text data to be processed can be image-text question-and-answer data containing images and text, and the processed text can be the answer text for that image-text question-and-answer data. For example, the image-text question-and-answer data can be image-text question-and-answer data containing an image of a desk and the text question "What items are placed on the desk in the image?". Based on this, the image-text question-and-answer data can be input into the image-text question-and-answer model for question-and-answer processing, outputting the answer text "There is a computer, mouse, keyboard, and water cup on the desk." Thus, through this data processing model training method, a model that answers user-submitted image-text questions can be obtained, enabling responses to user-submitted image-text questions in real-world scenarios and meeting the need for image-text question-and-answer using a model in practical scenarios.
[0082] When the data processing model provided in one or more embodiments of this specification is applied in a user communication scenario, the data processing model can be a graphic communication model. Correspondingly, the graphic data to be processed can be graphic communication data containing images and text, and the data processing text can be communication text for the graphic communication data. For example, the graphic communication data can be graphic communication data containing a forest image and the text data "Is this scenery beautiful?". Based on this, the graphic communication data can be input into the graphic communication model for communication processing, and the communication text "The scenery of this forest is very beautiful" can be output. This enables communication with users in real-world scenarios and meets the needs of using models to communicate with users in real-world scenarios.
[0083] When the data processing model training method provided in one or more embodiments of this specification is applied to an image processing scenario, the data processing model can be an image processing model. Correspondingly, the image-text data to be processed can be image-text data containing the image to be processed and the image processing text. The image to be processed can be an image that requires image processing operations from the image processing model, and the image processing text can be text that guides the image processing model to perform image processing operations on the image to be processed. These image processing operations include, but are not limited to, image attribute information recognition operations (i.e., recognizing attribute information such as image size, storage space occupied by the image, image format, and image clarity of the image to be processed) and image text extraction operations (i.e., extracting text from the image to be processed). The processed text can be the text data obtained by the image processing model after performing image processing operations on the image to be processed based on the image processing text. For example, the processed text can be image clarity text obtained through image clarity recognition operation such as "resolution is 720p", image format text obtained through image format recognition operation such as "image format is JPG" or text data in the image obtained through image text extraction operation such as "ABCD". Based on this, the image processing model trained in the image processing scenario can determine the target image processing operation corresponding to the image to be processed from a variety of image processing operations based on the image processing text, and perform image processing on the image to be processed based on the target image processing operation to obtain the processed text. For example, inputting the image-text data containing the paper image and the image processing text "extract text from the image" into the image processing model, the image processing model can determine the image text extraction operation (i.e., the target image processing operation) from various image processing operations (image format recognition operation, image sharpness recognition operation, image text extraction operation, etc.) based on the image processing text, and extract the image text from the paper image based on the image text extraction operation to obtain the paper text data in the paper image; thus, by training this data processing model, an image processing model can be obtained, realizing the processing of image data in real-world scenarios, and meeting the need to use models to process image data in real-world scenarios.
[0084] It should be noted that the models trained by the data processing model training method in one or more of the above embodiments can all be large models. Furthermore, in one or more embodiments provided in this specification, the data processing model is described using a text-based question-and-answer model as an example. For explanations regarding data processing models that are text-based communication models or image processing models, please refer to the corresponding explanations in the embodiments using text-based question-and-answer models as an example, thus avoiding excessive detail.
[0085] The first image-text training data can be understood as the data used to train the first multimodal large model in the data processing model. This first image-text training data includes image-text training samples and sample labels. It should be noted that in one or more embodiments provided in this specification, the first image-text training data differs depending on the data processing model. When the data processing model is an image-text question-and-answer model, the image-text training samples can be image-text question-and-answer data containing images and text, such as an image-text question-and-answer data containing an image of a desk and the text question "What items are placed on the desk in the image?". Correspondingly, the sample label can be understood as the answer text corresponding to the image-text question-and-answer data, such as the answer text "There is a computer, mouse, keyboard, and water cup on the desk." When the data processing model is an image-text communication model, the image-text training samples can be image-text communication data containing images and text, and correspondingly, the sample label can be the communication text for the image-text communication data.
[0086] The second image and text training data can be understood as the training data used to train the multimodal large model and the contrastive learning model in the data processing model. This second image and text training data includes image and text training samples. It should be noted that, in one or more embodiments provided in this specification, the image and text training samples included in the first image and text training data and those included in the second image and text training data may be the same or different. For example, if the image and text training samples in the first image and text training data are different from those in the second image and text training data, when the data processing models are different, the second image and text training data... The training samples for the images and text also differ. When the data processing model is an image-text question-answering model, the training samples can be image-text question-answering data that includes both images and text, such as an image-text question-answering data containing an image of a piano and the text question "What items are in the image?". When the data processing model is an image-text communication model, the training samples can be image-text communication data that includes both images and text, such as an image-text communication data containing an image of a snow-capped mountain and the text "It's so cold in winter."
[0087] The second multimodal large model can be understood as a multimodal large model pre-trained with training data. In one or more embodiments provided in this specification, the second multimodal large model is obtained through training with a large amount of image and text training data, and the second multimodal large model has strong universality. However, when the second multimodal large model is applied to a specific scenario (such as an image and text question-and-answer scenario or a user communication scenario), the second multimodal large model will suffer from the problem of knowledge illusion, which will lead to the inability to accurately generate information that corresponds to the input data. For example, large visual-language models (LVLMs) usually have a problem that when generating text responses, the model will inappropriately rely on text information, causing it to ignore image content and thus generate illusory text that is irrelevant to the image. To address this problem, introducing additional networks or making a large number of parameter adjustments to the model not only increases the complexity of the model but also increases the demand for computing resources. Therefore, the data processing model training method provided in this specification can be used to process the model and avoid the problem of knowledge illusion.
[0088] The first multimodal large model can be understood as a multimodal large model obtained by fine-tuning the second multimodal large model. For example, the determination of the first multimodal large model based on the second multimodal large model can be understood as obtaining the first multimodal large model by fine-tuning the parameters of the second multimodal large model. In one or more embodiments provided in this specification, during the image and text data processing process of the data processing model, the participation of the first multimodal large model enables the data processing model to accurately generate information that corresponds to and is correct with the input data, avoiding the problem of knowledge illusion. It should be noted that obtaining the first multimodal large model by fine-tuning the parameters of the second multimodal large model can be understood as fine-tuning the attention weight matrix within θ of the second multimodal large model to obtain the first multimodal large model. The specific implementation method can be: by introducing an amplification coefficient to enhance the attention of the multimodal large model to the image token, thereby strengthening the focus of the multimodal large model on visual information. This subtle adjustment encourages the multimodal large model to consider image information more when generating responses (i.e., data processing text), rather than overly relying on previously generated text or input text information. Furthermore, in one or more embodiments provided in this specification, the first multimodal large model (which may be referred to as the image-biased multimodal large model θ^) shares weight parameters with the original model θ (i.e., the second multimodal large model). This means that no additional parameters need to be loaded during the inference phase, thus saving memory usage and simplifying the model's computational requirements. This approach not only reduces reliance on additional training data but also avoids significant changes to the model structure, lowering implementation costs while maintaining model performance.
[0089] Taking the application of the data processing model training method provided in this specification in the knowledge illusion elimination scenario of multimodal large models as an example, this data processing model training method is explained. The first multimodal large model, determined based on the second multimodal large model, can be understood as the first multimodal large model being obtained based on the second multimodal large model. In one or more embodiments provided in this specification, LVLMs (second multimodal large models) typically use a visual image v and text t as input to generate text output. This process first inputs the image v into a visual encoder, then encodes it in the feature space to generate a set of visual tokens. These visual tokens are then concatenated with the input text as input to the language model in the LVLMs. Within this framework, LVLMs generate responses y = {y1, y2, ..., yN} using an autoregressive formula, with a total of N time steps. Here, y is the text output by LVLMs (i.e., the data processing text); the probability distribution of generating token yi in the i-th time step is represented as pθ(yi|v, t, y<i) = Softmax(logitθ(yi|v, t, y<i)), where θ refers to the LVLMs model (i.e., the original model, the ordinary model), and y<i = {y1, y2, ..., yi-1} represents the responses generated in previous time steps.
[0090] Unlike the LVLMs described above, the data processing model training method in one or more embodiments of this specification does not directly use the probability distribution obtained by the above process for prediction. Instead, to reduce the knowledge illusion phenomenon, the prediction of the next token is obtained by comparing the ordinary prediction (the text output by the second multimodal large model) with the image-based prediction (the text output by the first multimodal large model). Specifically, in addition to model θ (the second multimodal large model), a new model θ^ (the first multimodal large model) is introduced. An efficient method can be used to construct the image-biased model θ^ for the new model θ^, which can be achieved by adjusting the attention weight matrix within the ordinary model θ.
[0091] Step 204: Based on the first image and text training data, train the first multimodal large model to obtain the trained first multimodal large model.
[0092] Specifically, in one or more embodiments provided in this specification, the training process for the data processing model is divided into two stages. The second stage can use the first image-text training data to train the first multimodal large model; the second stage can use the second image-text training data to train the trained first multimodal large model and the contrastive learning model. For example, in the first stage, image-text sample data can be obtained to train the image-biased model θ^ (i.e., the first multimodal large model) to obtain the trained image-biased model θ^. In the second stage, the ordinary model θ, the image-biased model θ^, and the contrastive learning model can be used to process the image-text sample data (e.g., answer text generation processing, communication text generation processing, etc.) to obtain processing results (e.g., answer text corresponding to image-text question-and-answer data, etc.). Then, based on this inference processing result, the parameters of the "image-biased model θ^" and the "contrastive learning network" are adjusted to obtain the trained model θ^ and the contrastive learning network. Based on this, the training method for the first multimodal large model includes:
[0093] The step of training the first multimodal large model based on the first image and text training data to obtain the trained first multimodal large model includes:
[0094] Determine the image and text training samples and sample labels included in the first image and text training data;
[0095] The image and text training samples are input into the first multimodal large model to obtain the first sample processing result;
[0096] The first multimodal large model is trained based on the first sample processing result and the sample label to obtain the trained first multimodal large model.
[0097] The first sample processing result can be understood as the data processing text output by the multimodal large model after processing the image and text training sample, corresponding to the image and text training sample. When the data processing model provided in this specification is different, the image and text training sample is different, and the corresponding first sample processing result is also different, which will not be elaborated in detail here; for example, the first sample processing result can be the answer text for the image and text question and answer data used as the training sample, the communication text for the image and text communication data used as the training sample, etc.
[0098] Specifically, the data processing model training method provided in one or more embodiments of this specification, during the training of the first multimodal large model using first image and text training data, can determine the image and text training samples and sample labels contained in the first image and text training data, and input the image and text training samples into the first multimodal large model for processing (e.g., answer text generation processing, communication text generation processing, etc.) to obtain the output first sample processing result; then, based on the first sample processing result and sample labels, the model parameters of the first multimodal large model are adjusted until the model training stopping condition is reached, to obtain the trained first multimodal large model; based on the trained first multimodal large model, the data processing model is effectively prompted to consider more image information when generating text responses, thereby effectively avoiding the knowledge illusion caused by relying on text; wherein, the model training stopping condition can be set according to the actual application scenario, for example, the model training stopping condition can be that the loss value reaches convergence, the number of model training rounds reaches a preset number of rounds, etc.
[0099] In one or more embodiments provided in this specification, inputting the image and text training samples into the first multimodal large model to obtain the first sample processing result includes:
[0100] The image and text training samples are input into the first multimodal large model, and the text processing unit in the first multimodal large model is used to encode the text samples in the image and text training samples to obtain the text sample encoding.
[0101] The image processing unit in the first multimodal large model is used to encode the image samples in the image and text training samples to obtain the image sample codes;
[0102] Using the decoding unit in the first multimodal large model and the sample weight enhancement parameters for the image samples, the text sample encoding and the image sample encoding are decoded to obtain the first sample processing result.
[0103] The text processing unit can be understood as a unit in the first multimodal large model used to encode text samples or text data. This text processing unit can be one or more network layers in the first multimodal large model, or it can be a sub-model within the first multimodal large model; for example, it can be a text encoder in LVLMs. Correspondingly, the text sample encoding can be understood as the encoded data obtained after encoding text samples or text data using the text processing unit. Similarly, the image processing unit can be understood as a unit in the first multimodal large model used to encode image samples or image data. This image processing unit can be one or more network layers in the first multimodal large model, or it can be a sub-model within the first multimodal large model; for example, it can be a visual encoder in LVLMs. Correspondingly, the image sample encoding can be understood as the encoded data obtained after encoding image samples or image data using the image processing unit.
[0104] The sample weight enhancement parameter for the image sample can be understood as a parameter for weight enhancement of the image sample encoding; specifically, this sample weight enhancement parameter is used to enhance the attention weight of the image sample encoding during the generation of the first sample processing result; this sample weight enhancement parameter can be understood as the amplification coefficient in the above embodiments; in one or more embodiments provided in this specification, attention weight can be adjusted by the sample weight enhancement parameter or the image weight enhancement parameter, specifically by increasing the amplification coefficient on the basis of the ordinary model θ to enhance the model's attention to the image token, thereby adjusting the attention weight matrix, reducing the model's over-reliance on text features, and enhancing the focus on visual information. In one or more embodiments provided in this specification, the process of constructing an image-biased model θ^ is achieved by adjusting the attention weight matrix in the ordinary model θ. In each attention weight matrix of the LVLMs language model, an amplification coefficient c (i.e., the sample weight enhancement parameter) is applied to the image token, thereby increasing the attention to visual information while reducing the interaction with text features. This sample weight enhancement parameter is a predefined value that is conditionally added to the result of the attention query-key product when the key corresponds to the image token. It should be noted that in the process of constructing the image-biased model (the first multimodal large model), the data processing model training method provided in this specification proposes and implements an image-biased model θ^ (the first multimodal large model). By enhancing the parameters with sample weights, the data processing model assigns higher weights to image information in the attention mechanism, making the text generated by the model more dependent on the input image rather than the previous text information, thereby preventing the illusion of language bias.
[0105] The decoding unit can be understood as a unit in the first multimodal large model used to decode text sample encoding and image sample encoding. The decoding unit can be one or more network layers in the first multimodal large model, or the decoding unit can be a sub-model in the first multimodal large model.
[0106] Specifically, the image and text training samples are input into the first multimodal large model. The text processing unit in the first multimodal large model encodes the text samples in the image and text training samples to obtain text sample codes. The image processing unit in the first multimodal large model encodes the image samples in the image and text training samples to obtain image sample codes. During the decoding process of the text sample codes and image sample codes by the decoding unit in the first multimodal large model, the weight of the image sample codes is enhanced by the sample weight enhancement parameter for image samples to obtain the first sample processing result of image enhancement (i.e., image weight enhancement). Based on this, the data processing model training method provided in this specification introduces the sample weight enhancement parameter to enhance the attention of the data processing model to image codes, thereby strengthening the data processing model's focus on visual information. This allows the data processing model to consider image information more, rather than relying excessively on previously generated text or input text information, thus avoiding the problem of knowledge illusion.
[0107] Following the previous example, the training samples are input into an image-biased LVLMs model for processing, and an amplification factor is applied to each image token to obtain the visually enhanced output text (i.e., the processing result of the first sample). The specific execution method is as follows: 1. Input the training samples into the LVLMs model. 2. Use the language model in the LVLMs model to process the text in the training samples to obtain text encoding. 3. Use the vision model in the LVLMs model to process the images in the training samples to obtain image encoding. 4. Input the text encoding and image encoding into the projection module in the LVLMs model for processing to obtain the visually enhanced output text. During the processing of text and image encodings in the projection module, the attention weights of the image encoding can be adjusted using the amplification factor to generate multiple text tokens, each corresponding to a time step; and these multiple text tokens constitute the visually enhanced output text. This projection module can be understood as a module used to predict the visually enhanced output text based on the text and image encodings; this projection module can be considered a decoding unit.
[0108] In one or more embodiments provided in this specification, the step of using the decoding unit in the first multimodal large model and the sample weight enhancement parameters for the image samples to decode the text sample encoding and the image sample encoding to obtain the first sample processing result includes:
[0109] Using the decoding unit in the first multimodal large model, the query parameters corresponding to the text sample encoding and the image sample encoding are calculated through an attention mechanism, and the key parameters corresponding to the text sample encoding and the image sample encoding are also calculated.
[0110] The query parameter and the key parameter are multiplied to obtain a parameter product. The image key parameter corresponding to the image sample encoding is determined from the key parameter, and the image parameter product corresponding to the image key parameter is determined from the parameter product.
[0111] The product of the sample weight enhancement parameter for the image sample and the image parameter is added together to obtain the image weight enhancement parameter product;
[0112] The text sample encoding and the image sample encoding are decoded based on the product of the image weight enhancement parameters to obtain the first sample processing result.
[0113] The image weight enhancement parameter product may include the image parameter product obtained by adding the weight enhancement parameter to the image parameter product of multiple parameter products, as well as other parameter products other than the image parameter product obtained by the addition.
[0114] Following the previous example, the prediction module processes text sample encoding and image sample encoding as follows: It calculates the query (i.e., query parameters) and key (i.e., key parameters) corresponding to the text sample encoding and image sample encoding using an attention mechanism; then, it multiplies the query and key to obtain the attention query-key product (i.e., parameter product). When a key corresponds to an image sample encoding, that is, when the nth key token corresponds to an image token, it determines the attention query-key product (i.e., image parameter product) corresponding to that key; it adds the magnification factor c for the image sample to the attention query-key product (i.e., image parameter product) to obtain the image-enhanced attention query-key product (i.e., image weight enhancement parameter product); and it decodes the text sample encoding and image sample encoding based on the image-enhanced attention query-key product to obtain the output text of the first multimodal large model.
[0115] Specifically, firstly, when generating the text at the i-th time step, the token at the i-th time step of the LVLMs model is represented as... in It refers to the token used to input the image v; This refers to other components (input text, previously generated response, padding token). Next, each element in the attention weight matrix WI is calculated using the following formula (1) to obtain the visual enhancement encoding (i.e., the visually enhanced output text).
[0116]
[0117]
[0118] In formula (1) above, Q l and K l They refer to T respectively l The derived query and key tokens, M represents the causal mask (also known as autoregressive encoding), and D represents the feature dimension. Represents the nth key token Corresponding to the image token, ∈ refers to the hyperparameter, and softmax refers to the normalized exponential function.
[0119] Formula (1) above can be interpreted as adding the amplification factor C to the result of the attention query-key product. Since in the attention mechanism, Query represents the target to be focused on (i.e., image encoding), and Key is used to measure the correlation between Query and other vectors (model output); therefore, by adding the amplification factor C to the result of the attention query-key product, the weight of image encoding is increased, resulting in visually enhanced encoding. This method encourages the attention mechanism to focus more on image information, thus making the constructed model θ^ a multimodal large model biased towards images. In other words, the data processing model training method provided in this specification, through refined adjustments to the decoding process of the multimodal large model, effectively prompts the multimodal large model to consider image information more when generating text responses, thereby effectively avoiding the illusion of relying on text and reducing the need for additional parameters or data. This strategy overcomes the limitations in dealing with the illusion problem, improving the relevance and accuracy of LVLMs' generated responses while also enhancing operational efficiency and practicality. It should be noted that the first multimodal large model θ^ and the second multimodal large model θ can share the same weight parameters, thereby avoiding additional memory usage when loading other parameters during the inference phase.
[0120] Step 206: Based on the trained first multimodal large model, the second multimodal large model, and the contrastive learning model, process the second image and text training data to obtain the processing result.
[0121] The processing result can be understood as the result obtained after processing the second image and text training data using the trained first multimodal large model, second multimodal large model, and contrastive learning model. This processing result can be used to adjust the model parameters of the trained first multimodal large model and contrastive learning model. In one or more embodiments provided in this specification, the processing result can be a loss value or a loss function. It should be noted that in one or more embodiments provided in this specification, the data processing model may differ in different practical application scenarios. Correspondingly, the loss value or loss function for this data processing model can be set according to the actual application scenario, and no specific limitations are imposed here.
[0122] In one or more embodiments provided in this specification, the step of processing the second image-text training data based on the trained first multimodal large model, the second multimodal large model, and the contrastive learning model to obtain the processing result includes steps one and two:
[0123] Step 1: Input the second image and text training data into the trained first multimodal large model to obtain the second sample processing result, and input the second image and text training data into the second multimodal large model to obtain the third sample processing result.
[0124] The first multimodal large model provided in one or more embodiments of this specification has the same processing steps for the second image and text training data as the processing steps for the image and text training samples in the above one or more embodiments. Therefore, the step of inputting the second image and text training data into the trained first multimodal large model to obtain the second sample processing result can be found in the description of the step of inputting the image and text training samples into the first multimodal large model to obtain the first sample processing result in the above one or more embodiments, and will not be repeated here.
[0125] Specifically, the data processing model training method provided in this specification allows for the input of second-level image and text training data into the trained first multimodal large model during the second-stage training process. This input yields a second sample processing result. The second-level image and text training data is then input into the second multimodal large model, where its text processing unit, image processing unit, and decoding unit process the data to obtain a third sample processing result. This method effectively utilizes the second multimodal large model to retrain the trained first multimodal large model during the second-stage training process, thereby obtaining a data processing model capable of accurately generating correct information related to the input image and text data. This avoids the problem of knowledge illusion in the trained multimodal large model.
[0126] Following the previous example, the training samples (i.e., the second image-text training data) are input into the image augmentation LVLMs model (the first multimodal large model). The language model, vision model, and projection module in the image augmentation LVLMs model are used for image augmentation processing. During the image augmentation process, the attention weights of the image encoding are adjusted using a magnification factor, thereby obtaining the image-augmented answer text corresponding to the training samples (i.e., the second sample processing result).
[0127] The training samples are input into the LVLMs model (the second multimodal large model). The language model, vision model, and projection module in the LVLMs model are used for processing to obtain the answer text corresponding to the training samples (i.e., the third sample processing result).
[0128] In one or more embodiments provided in this specification, the step of inputting the second image and text training data into the second multimodal large model to obtain the third sample processing result includes:
[0129] The second image and text training data is input into the second multimodal large model. The text processing unit in the second multimodal large model is used to encode the text samples in the second image and text training data to obtain the text sample encoding.
[0130] Using the image processing unit in the second multimodal large model, the image samples in the second image and text training data are encoded to obtain image sample codes;
[0131] The text sample encoding and the image sample encoding are decoded using the decoding unit in the second multimodal large model to obtain the third sample processing result.
[0132] The text processing unit can be understood as a unit in the second multimodal large model used to encode text samples. This text processing unit can be one or more network layers in the second multimodal large model, or it can be a sub-model within the second multimodal large model; for example, it can be a text encoder in LVLMs. Correspondingly, the text sample encoding can be understood as the encoded data obtained after encoding the text sample using the text processing unit in the second multimodal large model.
[0133] Here, the image processing unit can be understood as a unit in the second multimodal large model used for encoding image samples; the image processing unit can be one or more network layers in the second multimodal large model, or the image processing unit can be a sub-model in the second multimodal large model; for example, the image processing unit can be a visual encoder in LVLMs. Correspondingly, the image sample encoding can be understood as the encoded data obtained after encoding the image sample using the image processing unit in the second multimodal large model.
[0134] The decoding unit can be understood as a unit in the second multimodal large model used to decode text sample encoding and image sample encoding. The decoding unit can be one or more network layers in the second multimodal large model, or the decoding unit can be a sub-model in the second multimodal large model.
[0135] Continuing with the previous example, the training samples (the second text-image training data) are input into the LVLMs model (the second multimodal large model). The language model within the LVLMs model processes the text in the training samples to obtain text encoding. The vision model within the LVLMs model processes the images in the training samples to obtain image encoding. The text encoding and image encoding are then input into the projection module of the LVLMs model for decoding to obtain the answer text corresponding to the training samples (i.e., the third sample processing result). This facilitates the accurate generation of the processing results for training the data processing model based on the third sample processing result, thereby improving the performance of the data processing model.
[0136] Step 2: Input the processing results of the second sample and the processing results of the third sample into the contrastive learning model to obtain the processing results.
[0137] The processing result can be a loss function, which is used to adjust the model parameters of the data processing model.
[0138] In one or more embodiments provided in this specification, inputting the second sample processing result and the third sample processing result into the contrastive learning model to obtain the processing result includes:
[0139] The second sample processing result and the third sample processing result are input into the contrastive learning model, and the encoding unit in the contrastive learning model is used to encode the second sample processing result and the third sample processing result to obtain the second sample processing result code and the third sample processing result code.
[0140] The contrastive learning model is used to process the encoding of the second sample processing result and the encoding of the third sample processing result to obtain the encoding processing result;
[0141] The processing result is obtained based on the encoding processing result.
[0142] In this context, the encoding unit can be understood as a unit in the contrastive learning model used to encode the processing results of the second and third samples. This encoding unit can be one or more network layers in the contrastive learning model, or it can be a sub-model within the model. For example, the encoding unit can be the encoder in the contrastive learning model.
[0143] Following the previous example, after obtaining the corresponding answer texts through the two models, the contrastive learning model needs to be used to calculate the similarity between the two answer texts. Specifically, the two answer texts are input into the contrastive learning network, and the encoding network layer in the contrastive learning network encodes the two answer texts, obtaining two answer text codes (i.e., the code for the second sample processing result and the code for the third sample processing result). Then, the contrastive learning network calculates the similarity result between the two answer text codes, and calculates the loss function based on the similarity result. Subsequently, the parameters of the data processing model are adjusted based on the loss function.
[0144] The specific implementation of similarity calculation can be as follows: 1. Input the ordinary encoding (result of the third sample processing) and the visually enhanced encoding (result of the second sample processing) into the encoder of the contrastive learning network for encoding processing to obtain the encoded data corresponding to the ordinary encoding and the encoded data corresponding to the visually enhanced encoding; 2. Input the two encoded data into the project head (which can be composed of multiple Dense Layers) to compress the two encoded data into a latent space representation, allowing the contrastive learning model to learn the high-order features of the image; 3. Calculate the similarity between the two encoded data (e.g., cosine similarity) and send this similarity to the two models for prediction of the next token; 4. Through the above steps, obtain the data processing text (e.g., a detailed description of objects in the sample image) output by the two models (image-biased LVLMs model and ordinary LVLMs model) for the training data; 5. Calculate the similarity between the two data processing texts and calculate the loss value based on the similarity; 6. Adjust the parameters of the image-biased LVLMs model and the contrastive learning network using this loss value until the model training stopping condition is met.
[0145] It's important to note that, unlike the theoretically balanced prediction using image v and text {t, y < i}, the image-biased LVLMs model θ^ (i.e., the first multimodal large model) relies more heavily on the information contained in image v and amplifies the importance of this information. Predictions are obtained using θ^. Then, the probability (similarity) of the next token can be calculated using the following formula (2):
[0146]
[0147]
[0148] Wherein, in the formula It is a visually enhanced multimodal large model predictive answer text token, logit. θ (y i |v,t,y<i) are the answer text tokens predicted by the multimodal large model, p(y i |y<i) can be understood as the similarity between two answer text tokens.
[0149] The motivation for introducing the θ^ model and contrastive decoding method in the data processing model training method of this specification is based on the hallucination characteristics observed in LVLMs. It has been determined that over-reliance on the language prior of the language model is a significant factor contributing to the hallucination in LVLMs; specifically, it has been observed that LVLMs sometimes ignore the image input, treating the prediction of the next token as a pure text continuation task. This approach may lead to the generation of hallucination responses that are unrelated to the input image and are therefore text-biased. Based on these observations, the data processing model training method in this specification proposes a hypothesis that, at each time step, predictions using a model θ^ that pays more attention to information about the input image will result in the token with the highest probability of rising from θ to θ^ in the candidate pool being more likely to be the correct token without text-biased hallucinations.
[0150] In one or more embodiments provided in this specification, processing the encoding of the second sample processing result and the encoding of the third sample processing result using the contrastive learning model to obtain the encoding processing result includes:
[0151] The similarity between the second sample processing result encoding and the third sample processing result encoding is calculated using the contrastive learning model to obtain similarity results;
[0152] Obtaining the processing result based on the encoding processing result includes:
[0153] A loss function is calculated based on the similarity results, and the loss function is used as the processing result.
[0154] Following the previous example, the encoding network layer in the contrastive learning network encodes the two answer texts, obtaining two answer text encodings (the encoding of the second sample processing result and the encoding of the third sample processing result). Then, the contrastive learning network calculates the similarity between the two answer text encodings and uses this similarity result to calculate a loss function, which is then used as the processing result. This facilitates subsequent parameter adjustments to the trained first multimodal large model and the contrastive learning model based on this processing result, resulting in a data processing model capable of accurately generating relevant and correct information related to the input image and text data.
[0155] In one or more embodiments provided in this specification, the step of inputting the second image-text training data into the trained first multimodal large model to obtain a second sample processing result, and inputting the second image-text training data into the second multimodal large model to obtain a third sample processing result, includes:
[0156] The second image and text training data is input into the trained first multimodal large model to obtain the second sample processing result data, and the second image and text training data is input into the second multimodal large model to obtain the third sample processing result data;
[0157] The contrastive learning model is used to process the second sample processing result data and the third sample processing result data to obtain the data processing result;
[0158] Using the trained first multimodal large model, the second sample processing result is obtained based on the second sample processing result data and the data processing result; and using the second multimodal large model, the third sample processing result is obtained based on the third sample processing result data and the data processing result.
[0159] The sample processing result data can be understood as the data constituting the sample processing result. In one or more embodiments provided in this specification, the sample processing result can be data processing text for the training sample data, and the data processing text can be composed of multiple text tokens (also called character tokens, tokens, or tokens). Furthermore, these multiple text tokens are generated one after another by the multimodal large model in a time sequence; based on this, the sample processing result data can be understood as one or more text tokens constituting the data processing text; correspondingly, the first sample processing result data can be a text token generated by the first multimodal large model, and the second sample processing result data can be a text token generated by the second multimodal large model.
[0160] The data processing result can be understood as the similarity between the data from the second sample processing result and the data from the third sample processing result.
[0161] Continuing with the previous example, the multimodal large model needs to generate multiple text tokens during the generation of the corresponding answer text. During the reasoning process for these text tokens, text tokens from two different time steps can be input into a contrastive learning network for similarity calculation. The calculated similarity then guides the generation of the next text token. Specifically, ordinary text tokens (the result of the third sample processing) and visually enhanced large tokens (the result of the second sample processing) can be input into the encoder of the contrastive learning network for encoding, obtaining the encoded data corresponding to the ordinary text token and the visually enhanced text token. These two encoded data are then input into a projection head (which can consist of multiple DenseLayers), compressing them into a latent space representation. This allows the contrastive learning model to learn the high-order features of the image. Based on this latent space representation, the similarity between the two encoded data (e.g., cosine similarity) is calculated, and the calculated similarity is sent to two multimodal large models for predicting the next token. The process continues until all text tokens are deduced, thus obtaining the answer text. Based on this, the similarity between the text tokens output by the two models is used to guide the prediction of the next text token, thereby improving the accuracy of the prediction results (data processing text) of the two models.
[0162] In one or more embodiments provided in this specification, the step of inputting the second image-text training data into the trained first multimodal large model to obtain second sample processing result data, and inputting the second image-text training data into the second multimodal large model to obtain third sample processing result data, includes:
[0163] The second image and text training data is input into the trained first multimodal large model, and the second image and text training data is input into the second multimodal large model;
[0164] The trained first multimodal large model is used to process the second image and text training data to obtain the initial second sample processing result data;
[0165] The second multimodal large model is used to process the second image and text training data to obtain the initial third sample processing result data;
[0166] The initial second sample processing result data can be understood as any one of the multiple text tokens calculated by the first multimodal large model, except for the last text token, and the multiple text tokens are sorted by time steps.
[0167] The initial third sample processing result data can be understood as any one of the multiple text tokens calculated by the second multimodal large model, except for the last text token, and the multiple answer text tokens are sorted by time steps.
[0168] Continuing with the previous example, the multimodal large model needs to generate multiple text tokens during the generation of the corresponding answer text. In the process of inferring the text token, image and text sample data is input into both the first and second multimodal large models. The first multimodal large model uses the image and text sample data to perform answer text inference, obtaining the answer text token corresponding to the answer text. The second multimodal large model uses the same image and text sample data to perform answer text inference, obtaining the answer text token corresponding to the answer text. The time steps for the two answer text tokens are consistent.
[0169] In one or more embodiments provided in this specification, processing the second sample processing result data and the third sample processing result data using the contrastive learning model to obtain data processing results includes:
[0170] The initial second sample processing result data and the initial third sample processing result data are input into the contrastive learning model to obtain the data similarity between the initial second sample processing result data and the initial third sample processing result data.
[0171] Following the previous example, the two answer text tokens are input into the contrastive learning network, which calculates the similarity between the two answer text tokens and sends the similarity to the first multimodal large model and the second multimodal large model.
[0172] In one or more embodiments provided in this specification, obtaining the second sample processing result using the trained first multimodal large model based on the second sample processing result data and the data processing result, and obtaining the third sample processing result using the second multimodal large model based on the third sample processing result data and the data processing result, includes:
[0173] The data similarity is input into the trained first multimodal large model and the second multimodal large model;
[0174] Using the trained first multimodal large model, the second image and text training data is processed based on the data similarity to obtain the target second sample processing result data, and the second sample processing result is obtained based on the initial second sample processing result data and the target second sample processing result data.
[0175] Using the second multimodal large model, the second image and text training data is processed based on the data similarity to obtain the target third sample processing result data, and the third sample processing result is obtained based on the initial third sample processing result data and the target third sample processing result data.
[0176] The target second sample processing result data can be understood as the next second sample processing result data corresponding to the initial second sample processing result data; the target third sample processing result data can be understood as the next third sample processing result data corresponding to the initial third sample processing result data.
[0177] Following the previous example, after receiving the similarity score, the first and second multimodal large models use this similarity to continue reasoning about the answer text for the image-text sample data, obtaining the next answer text token (i.e., the target second sample processing result data and the target third sample processing result data). After the first and second multimodal large models complete the reasoning of all answer text tokens (i.e., the initial second sample processing result data, the target second sample processing result data, the initial third sample processing result data, and the target third sample processing result data), the answer text corresponding to the image-text training sample can be obtained. Based on this, in one or more of the above embodiments, the similarity between the text tokens output by the two models is used to guide the prediction of the next text token, thereby improving the accuracy of the prediction results (data processing text) of the two models and obtaining accurate prediction results.
[0178] Step 208: Based on the processing results, adjust the model parameters of the trained first multimodal large model and the contrastive learning model to obtain a trained data processing model, wherein the data processing model is used to perform the data processing task.
[0179] Here, the data processing task can be understood as a task that requires processing by a data processing model. The data processing task will vary depending on the data processing model. For example, the data processing task can be a text-image question-and-answer task, which contains text-image question-and-answer data; the data processing task can be a text-image communication task, which contains text-image communication data; the data processing task can be an image processing task, which contains text-image data to be processed.
[0180] Following the previous example, the similarity between the two language prediction results is calculated, and the loss function is calculated based on the similarity. The model parameters of the LVLMs model biased towards the image and the contrastive learning network are adjusted using the loss function until the model training stops, and the trained data processing model is obtained.
[0181] The data processing model training method provided in one or more embodiments of this specification trains a data processing model including a first multimodal large model, a second multimodal large model, and a contrastive learning model. During model training, the first multimodal large model is first trained in a first phase. Then, based on the trained first multimodal large model, second multimodal large model, and contrastive learning model, the processing results obtained from processing the second image and text training data are processed. The model parameters of the trained first multimodal large model and contrastive learning model are then adjusted again. Based on the second multimodal large model and contrastive learning model, the trained first multimodal large model is trained again. This results in a data processing model that can accurately generate correct information related to the input image and text data, avoiding the problem of knowledge illusion in the trained multimodal large model and solving the problem of the trained multimodal large model generating irrelevant or incorrect information related to the input image and text data. Furthermore, it solves the problem of being unable to process user image and text data in real-world scenarios, meeting the actual needs for image data processing in real-world scenarios.
[0182] The following is in conjunction with the appendix Figure 3 Taking the application of the data processing model training method provided in this specification in a knowledge illusion elimination scenario based on image bias coding for multimodal large models as an example, the data processing model training method will be further explained. Figure 3 The present specification illustrates a flowchart of a data processing model training method according to an embodiment, which includes the following steps.
[0183] Step 302: Determine the training sample data, and use the training sample data to train the multimodal large model and the visually enhanced multimodal large model.
[0184] The training sample data could be an image of a desk, along with the question text describing the details in the image: "The image includes a desk with a computer, a laptop, a mouse, and a..."
[0185] It should be noted that the visually enhanced multimodal large model is a pre-trained multimodal large model. In the data processing model training method provided in this manual, the training of the visually enhanced multimodal large model can be divided into two stages.
[0186] The first phase of training includes the following steps:
[0187] 1. Obtain a pre-trained multimodal large model (e.g., LVLMs model).
[0188] 2. Collect training data for LVLMs models.
[0189] Considering the knowledge illusion problem in LVLMs models, this approach re-collects training data (including sample images and the corresponding real descriptive text of the sample images as sample labels), and then fine-tunes the LVLMs model accordingly to obtain a visually enhanced LVLMs model.
[0190] 3. Determine the magnification factor C corresponding to the image token.
[0191] The amplification factor can be a predefined value.
[0192] 4. Input the training samples into the visually enhanced LVLMs model for answer text reasoning, and apply a magnification factor to each image token to obtain the visually enhanced answer text token.
[0193] The specific execution method for this step is as follows:
[0194] First, the training samples are input into the LVLMs model; the language model in the LVLMs model is used to process the text in the training samples to obtain text encoding; the vision model in the LVLMs model is used to process the images in the training samples to obtain image encoding.
[0195] Secondly, when generating the text for the i-th time step, the token for the i-th time step of the LVLM is represented as... in It refers to the token used to input the image v; This refers to other components (input text, previously generated responses, filler tokens).
[0196] Based on this, text encoding and image encoding are input into the encoding module (projection) of the LVLMs model. This encoding module can determine the query (i.e., query token) and key (i.e., key token) corresponding to the text encoding output by the language model, and determine the query (i.e., query token) and key (i.e., key token) corresponding to the image encoding output by the visual model, thereby obtaining the query from T. l Derivative query Q l Key token K l .
[0197] The above two query Q l and key K l Multiply to obtain the attention query-key product.
[0198] From multiple keys K lThe image key token corresponding to the image token is determined, and the attention query-key product corresponding to the image key token is determined from all attention query-key products.
[0199] The magnification factor C is added to the attention query-key product corresponding to the image key token, thereby enhancing the weight of the image in the attention query-key product.
[0200] The method for calculating each element in the attention weight matrix WI to obtain the visually enhanced attention query-key product can be found in the content of the above formula (1).
[0201] Finally, by using the attention query-key product that enhances image weights, the text encoding output by the language model and the image encoding output by the visual model are decoded to obtain the visually enhanced answer text token.
[0202] 5. Combine the visually enhanced answer text tokens with the sample labels, calculate the loss function, and adjust the parameters of the visually enhanced multimodal large model based on the loss function to obtain the trained visually enhanced multimodal large model.
[0203] Step 304: Use the language model, image model and encoding model in the multimodal large model to process the training sample data and obtain the corresponding answer text.
[0204] One possible answer is: "There is also a keyboard on this table."
[0205] Step 306: Use the language model, image model and encoding model in the visually enhanced multimodal large model to process the training sample data and obtain the corresponding visually enhanced answer text.
[0206] One possible answer is: "There are green plants on this table."
[0207] Step 308: Input the two answer texts into the contrastive learning network to obtain the similarity between the two answer texts.
[0208] Specifically, the execution process of this step includes:
[0209] 1. Use the encoder in the contrastive learning network to encode the two answer texts to obtain the answer text encoding corresponding to the answer text and the visually enhanced answer text encoding corresponding to the visually enhanced answer text.
[0210] 2. Encode the two answer texts and input them into the projection head (which can consist of multiple Dense Layers). Compress the two answer text encodings into a latent space representation, allowing the contrastive learning network to learn the high-order features of the image.
[0211] 3. Based on this latent space representation, calculate the similarity (e.g., cosine similarity) between the text encodings of the two answers.
[0212] Step 310: Calculate the loss function based on the similarity, and use the loss function to adjust the parameters of the visually enhanced multimodal large model and the contrastive learning network to achieve the second stage of model training until the model training stops.
[0213] Furthermore, it should be noted that in the process of generating the corresponding answer text in the multimodal large model, multiple text tokens need to be generated. In the process of reasoning about the text token, two text tokens at the same time step can be input into the contrastive learning network for similarity calculation, and the generation of the next text token is guided based on the calculated similarity.
[0214] The specific execution steps are as follows:
[0215] 1. Input the answer text tokens output by the multimodal large model and the answer text tokens output by the visually enhanced multimodal large model into the contrastive learning network.
[0216] 2. The encoder in the contrastive learning network encodes the two answer text tokens to obtain the corresponding encoded data.
[0217] 3. Input the two encoded data into the projection head and compress the two encoded data into a latent space representation.
[0218] 4. Calculate the similarity between two encoded data based on the latent space representation.
[0219] The similarity can be calculated using the formula (2) above.
[0220] 3. The calculated similarity is sent to two multimodal large models for predicting the next answer text token. This process continues until all answer text tokens have been inferred, thus obtaining the answer text.
[0221] Based on the steps described above, the data processing model training method provided in this specification aims to address the knowledge illusion problem in large multimodal models. By combining image bias coding, it effectively eliminates output errors caused by text input to large models. Furthermore, by fine-tuning the attention weight matrix within model θ and introducing an amplification factor to enhance the model's attention to image tokens, it strengthens the model's focus on visual information. This subtle adjustment encourages the model to consider image information more when generating responses, rather than overly relying on previously generated text or input text information.
[0222] Furthermore, the image-biased model θ^ in this scheme shares weight parameters with the original model θ, meaning that no additional parameters need to be loaded during the inference phase, thus saving memory usage and simplifying the model's computational requirements. This approach not only reduces reliance on additional training data but also avoids significant changes to the model structure, lowering implementation costs while maintaining model performance.
[0223] This approach, through fine-tuning the model's decoding process, effectively encourages the model to consider image information more when generating text responses. This effectively avoids relying on textual illusions and reduces the need for additional parameters or data. This strategy overcomes the limitations of previous methods in handling illusion problems, enabling this approach to improve the relevance and accuracy of LVLMs' generated responses while also enhancing operational efficiency and practicality.
[0224] In summary, this solution achieves several key benefits, including attention weight adjustment, image-biased model construction, shared weight parameters, and the elimination of the need for additional data or training. Specifically, attention weight adjustment enhances the model's attention to image tokens by adjusting the attention weight matrix on top of the standard model θ, particularly by increasing the amplification factor. This reduces the model's over-reliance on text features and strengthens its focus on visual information. Image-biased model construction involves proposing and implementing an image-biased model θ^, which assigns higher weights to image information in the attention mechanism. This makes the generated text more dependent on the input image than on previous text information, thus preventing language bias illusions. Shared weight parameters allow the θ^ model to share weight parameters with the initial LVLM model θ. This effectively reduces illusions without increasing model size and complexity and avoids loading additional parameters during inference, saving memory resources. Finally, the solution eliminates the need for large amounts of additional training data or complex model structures, making it simpler and more efficient than methods requiring new networks or extensive parameter adjustments. These combined benefits make this solution more efficient and practical in addressing the illusion problem in LVLMs.
[0225] See Figure 4 , Figure 4 A flowchart of a machine learning model training method according to an embodiment of this specification is shown, which specifically includes the following steps.
[0226] Step 402: Determine the machine learning model to be trained, as well as the first training data and the second training data associated with the target task, wherein the machine learning model to be trained includes a first data processing module, a second data processing module, and a parameter learning module.
[0227] In this specification, a machine learning model can be understood as a model capable of processing a target task. The machine learning model training method provided in this specification is applied in different scenarios, and the machine learning model is also different. In one or more embodiments provided in this specification, the machine learning model can be the data processing model in the above-mentioned data processing model training method. Correspondingly, the first data processing module is a first multimodal large model, the second data processing module is a second multimodal large model, the parameter learning module is a contrastive learning model, and the target task can be a data processing task.
[0228] In one or more embodiments provided in this specification, when the machine learning model training method is applied to a video generation scenario, the machine learning model can be a video generation model, the first data processing module can be a first video generation network, used to generate a corresponding video based on the input prompt information; the second data processing module can be a second video generation network, used to generate a corresponding video based on the input prompt information; wherein, the prompt information can be a combination of any two types of data such as text prompt information, image prompt information, and video prompt information; the text prompt information can be prompts such as "generate a video of a kitten dancing" or "generate a video of a puppy walking on the moon," and the image prompt information can be an image of a kitten or an image of a puppy; after the text prompt information and / or image prompt information are input to the first video generation network and the second video generation network, the first video generation network and the second video generation network can generate a corresponding video based on the text prompt information and / or image prompt information, for example, a video of a kitten dancing or a video of a puppy walking on the moon. The image prompt can be an image of a piano from brand A, and the video prompt can be a video of a pianist playing a broken piano. After inputting the image and video prompts into a first video generation network and a second video generation network, the first and second video generation networks can automatically generate corresponding videos based on the image prompts, such as a video of a pianist playing a piano from brand A. The parameter learning module can be a video detection network used to perform quality checks on the videos generated by the first and second video generation networks, thereby using the higher-quality video generated by the two video generation networks as the output of the video generation model. The quality check operation includes, but is not limited to, video sharpness detection, video duration detection, and error frame detection in the video. Based on this, the video generation model automatically generates videos, meeting the user's needs for generating videos in practical application scenarios. Furthermore, the video detection network performs quality checks on the videos generated by the two video generation networks, thereby outputting a higher-quality video to the user.
[0229] The first training data can be understood as the data used to train the first data processing module, and the second training data can be understood as the data used to train both the first data processing module and the parameter learning module. Due to different machine learning models, the first and second training data will differ. When the machine learning model is a data processing model, the first training data is the first text / image training data, and the second training data is the second text / image training data. When the machine learning model is a video generation model, the first training data can be text prompts, image prompts, and / or video prompts used as training samples, as well as sample labels (i.e., videos used as labels). The second training data can be text prompts and image prompts used as training samples.
[0230] Step 404: Train the first data processing module based on the first training data to obtain the trained first data processing module.
[0231] In one or more embodiments provided in this specification, training the first data processing module based on the first training data to obtain the trained first data processing module includes:
[0232] Determine the training samples and sample labels included in the first training data;
[0233] The training samples are input into the first data processing module to obtain the first sample processing result;
[0234] The first data processing module is trained based on the first sample processing result and the sample label to obtain the trained first data processing module.
[0235] The first sample processing result can be the processing result of the first data processing module on the training sample. For example, the first sample processing result can be the generated video.
[0236] The result of the first sample processing can be understood as the first image and text training data in the above data processing model training method.
[0237] Following the previous example, during the training of the first video generation network, the first training data can be determined to include text prompts and image prompts as samples, and videos as labels. The text prompts and image prompts as samples are input into the first video generation network to generate corresponding videos (i.e., the first sample processing result). The loss value is calculated based on the video and the video as labels, and the model parameters of the first video generation network are adjusted based on the calculated loss value until the model training stops, thus obtaining the trained first video generation network.
[0238] In one or more embodiments provided in this specification, inputting the training samples into the first data processing module to obtain the first sample processing result includes:
[0239] The training samples are input into the first data processing module, and the first modality processing unit in the first data processing module is used to encode the first modality samples in the training samples to obtain the first modality sample code.
[0240] The second modality processing unit in the first data processing module is used to encode the second modality samples in the training samples to obtain the second modality sample codes.
[0241] Using the decoding unit in the first data processing module and the sample weight enhancement parameters for the second modal sample, the encoding of the first modal sample and the encoding of the second modal sample are decoded to obtain the first sample processing result.
[0242] The training samples can be multimodal training samples, which can include any two data items from audio, video, image, text, etc. For example, the multimodal training samples may include an incomplete video and a prompt text for the video, "Generate a subsequent video based on the incomplete video". Correspondingly, the first modal sample can be the incomplete video, and the second modal sample can be the prompt text for the video.
[0243] The first modality processing unit and the second modality processing unit can be understood as units that process the first modality samples and the second modality samples in the multimodal training samples. The first modality processing unit and the second modality processing unit can be one or more network layers or sub-models.
[0244] In one or more embodiments provided in this specification, the first modality processing unit may be the text processing unit in the above-described data processing model training method, and the second modality processing unit may be the image processing unit in the above-described data processing model training method; correspondingly, the first modality sample data is a text sample, and the second modality sample data is an image sample.
[0245] The sample weight enhancement parameter can be understood as a parameter that enhances the weight of the second modality sample encoding; specifically, this sample weight enhancement parameter is used to enhance the attention weight of the second modality sample encoding during the generation of the first sample processing result; thereby increasing the attention paid to the second modality sample encoding.
[0246] In one or more embodiments provided in this specification, the step of using the decoding unit in the first data processing module and the sample weight enhancement parameters for the second modality sample to decode the encoding of the first modality sample and the encoding of the second modality sample to obtain the first sample processing result includes:
[0247] Using the decoding unit in the first data processing module, the query parameters corresponding to the first modality sample code and the second modality sample code are calculated through an attention mechanism, and the key parameters corresponding to the first modality sample code and the second modality sample code are also calculated.
[0248] The query parameter and the key parameter are multiplied to obtain a parameter product. The second modality key parameter corresponding to the second modality sample code is determined from the key parameter, and the second modality parameter product corresponding to the second modality key parameter is determined from the parameter product.
[0249] The product of the sample weight enhancement parameter for the second modality sample and the second modality parameter is added together to obtain the second modality weight enhancement parameter product;
[0250] The first modality sample encoding and the second modality sample encoding are decoded based on the product of the second modality weight enhancement parameters to obtain the first sample processing result.
[0251] For an explanation of this embodiment, please refer to the corresponding or relevant steps in the data processing model training method described above.
[0252] Step 406: Process the second training data according to the first data processing module, the second data processing module and the parameter learning module after training to obtain the processing result;
[0253] In one or more embodiments provided in this specification, the step of processing the second training data according to the trained first data processing module, the second data processing module, and the parameter learning module to obtain a processing result includes:
[0254] The second training data is input into the first data processing module after training to obtain the second sample processing result, and the second training data is input into the second data processing module to obtain the third sample processing result.
[0255] The processing results of the second sample and the third sample are input into the parameter learning module to obtain the processing results.
[0256] Following the previous example, the second training data is input into the trained first video generation network to obtain the generated video, and the second training data is input into the second video generation network to obtain the generated video; the similarity between the two videos is calculated, and the loss function (i.e., the processing result) is determined based on the similarity.
[0257] In one or more embodiments provided in this specification, the step of inputting the second training data into the second data processing module to obtain the third sample processing result includes:
[0258] The second training sample is input into the second data processing module, and the first modality processing unit in the second data processing module is used to encode the first modality sample in the second training sample to obtain the first modality sample code.
[0259] The second modality processing unit in the second data processing module is used to encode the second modality samples in the second training samples to obtain the second modality sample codes.
[0260] The decoding unit in the second data processing module is used to decode the first modal sample encoding and the second modal sample encoding to obtain the third sample processing result.
[0261] In one or more embodiments provided in this specification, inputting the second sample processing result and the third sample processing result into the parameter learning module to obtain the processing result includes:
[0262] The second sample processing result and the third sample processing result are input into the parameter learning module, and the encoding unit in the parameter learning module is used to encode the second sample processing result and the third sample processing result to obtain the second sample processing result code and the third sample processing result code.
[0263] The parameter learning module is used to process the encoding of the second sample processing result and the encoding of the third sample processing result to obtain the encoding processing result;
[0264] The processing result is obtained based on the encoding processing result.
[0265] In one or more embodiments provided in this specification, the step of inputting the second training data into the first data processing module after training to obtain a second sample processing result, and inputting the second training data into the second data processing module to obtain a third sample processing result, includes:
[0266] The second training data is input into the first data processing module after training to obtain the second sample processing result data, and the second training data is input into the second data processing module to obtain the third sample processing result data.
[0267] The parameter learning module is used to process the second sample processing result data and the third sample processing result data to obtain the data processing result;
[0268] Using the first data processing module after training, the second sample processing result is obtained based on the second sample processing result data and the data processing result; and using the second data processing module, the third sample processing result is obtained based on the third sample processing result data and the data processing result.
[0269] Step 408: Based on the processing result, adjust the model parameters of the first data processing module and the parameter learning module after training to obtain the target model after training, wherein the target model is used to perform the target task.
[0270] In one or more embodiments provided in this specification, the machine learning model includes a first data processing module, a second data processing module, and a parameter learning module, which can be applied to computer hardware. For example, the first data processing module, the second data processing module, and / or the parameter learning module can be applied to the GPU or CPU of a computer. Furthermore, the data generated by the first data processing module, the second data processing module, and the parameter learning module during the processing of the target task can be stored in storage devices such as memory, video memory, or external storage. Through the cooperation between the first data processing module, the second data processing module, and the parameter learning module applied to the computer hardware, the target task can be completed quickly and efficiently, thereby saving computer resources. In addition, it should be noted that during the training process of the machine learning model, the first data processing module, the second data processing module, and the parameter learning module can also be trained on their respective computer hardware, thereby avoiding problems such as poor performance or incompatibility of the machine learning model caused by inconsistencies between the model training hardware environment and the model application hardware environment.
[0271] In one or more embodiments provided in this specification, the above-described data processing model training method can be an application of a machine learning model training method in a data processing scenario. Based on this, for the explanation of the execution steps and terms in the machine learning model training method, please refer to the corresponding explanations in the above-described data processing model training method, which will not be elaborated further here.
[0272] The machine learning model training method in one or more embodiments provided in this specification trains a machine learning model including a first data processing module, a second data processing module, and a parameter learning module. During model training, a first-stage training is first performed on the first data processing module. Then, based on the processed results obtained from processing second training data using the trained first data processing module, second data processing module, and parameter learning module, the model parameters of the trained first data processing module and parameter learning module are adjusted again. Thus, based on factors such as the second data processing module and parameter learning module, the trained first data processing module is trained again, resulting in a machine learning model with high performance and high efficiency in handling target tasks. This avoids the problem of poor performance in trained machine learning models that cannot efficiently handle target tasks, facilitating the processing of user-related target tasks.
[0273] The above one or more embodiments are illustrative schemes of a machine learning model training method. It should be noted that the technical solution of this machine learning model training method belongs to the same concept as the technical solution of the data processing model training method described above. Details not described in detail in the technical solution of the machine learning model training method can be found in the description of the technical solution of the data processing model training method described above.
[0274] See Figure 5 , Figure 5 A flowchart of another data processing method provided according to an embodiment of this specification is shown, which specifically includes the following steps.
[0275] Step 502: Determine the text and image data to be processed input by the user;
[0276] Step 504: Input the image and text data to be processed into the data processing model to obtain the data processing text corresponding to the image and text data to be processed. The data processing model includes a first multimodal large model, a second multimodal large model, and a contrastive learning model. The first multimodal large model is based on the second multimodal large model. The first multimodal large model is used to process the image and text data to be processed to obtain a first image and text processing result. The second multimodal large model is used to process the image and text data to be processed to obtain a second image and text processing result. The contrastive learning model is used to process the first image and text processing result and the second image and text processing result to obtain the data processing text corresponding to the image and text data to be processed.
[0277] The data processing model is obtained by training according to the above-described data processing model training method.
[0278] For explanations regarding the text and image data to be processed and the text for data processing, please refer to the corresponding explanations in the above-mentioned data processing model training methods, which will not be elaborated here.
[0279] For example, the steps for applying this data processing model could be:
[0280] 1. Obtain the questions (including pictures and text) sent by users.
[0281] 2. Input the images and text sent by the user into the trained data processing model to obtain the model's output.
[0282] The data processing model employs the following methods:
[0283] First, the images and text sent by the user are input into a regular LVLMs model in the data processing model to obtain a regular inference result (tokens sorted by time steps).
[0284] Secondly, the images and text sent by the user are input into the biased image LVLMs model in the data processing model to obtain the biased image inference results (tokens sorted by time steps).
[0285] It should be noted that when obtaining the token at time step i, the tokens output by the two models need to be input into the contrastive learning network in the data processing model for comparison to obtain the similarity.
[0286] Finally, this similarity is used to guide the two models in predicting the next token until the final prediction result (ordinary inference result, image inference result) is obtained.
[0287] Since the prediction results are guided by similarity, the final ordinary inference results and image inference results have a very high similarity. Therefore, the ordinary inference results or image inference results are used as the output of the multimodal large model to obtain the answer text for the image-text question.
[0288] 3. Display the answer text output by the data processing model to the user, thereby answering the user's questions.
[0289] Another data processing method provided in one or more embodiments of this specification can utilize a data processing model that accurately generates information related to and correct to the input graphic and text data to process the graphic and text data to be processed. This avoids the problem of multimodal large models generating information that is irrelevant or incorrect to the input graphic and text data, thereby solving the graphic and text data processing problem faced by users in real-world scenarios and further facilitating users in processing graphic and text data.
[0290] See Figure 6 , Figure 6 A flowchart of a graphic data processing method according to an embodiment of this specification is shown, which specifically includes the following steps.
[0291] Step 602: Determine the text and image data to be processed, and input the text and image data to be processed into the data processing model, wherein the data processing model includes a second multimodal large model, a third multimodal large model and a contrastive learning model, and the first multimodal large model is determined based on the second multimodal large model.
[0292] Step 604: Use the first multimodal large model to process the image and text data to be processed, and obtain the first image and text processing result.
[0293] In one or more embodiments provided in this specification, processing the text and image data to be processed using the first multimodal large model to obtain a first text and image processing result includes:
[0294] The text and image data to be processed is input into the first multimodal large model. The text processing unit in the first multimodal large model is used to encode the text data in the text and image data to be processed to obtain the text data encoding.
[0295] The image processing unit in the first multimodal large model is used to encode the image data in the image and text data to be processed to obtain the image data encoding;
[0296] Using the decoding unit in the first multimodal large model and the image weight enhancement parameters for the image data, the text data encoding and image data encoding are decoded to obtain the first image-text processing result.
[0297] The first image-text processing result can be understood as the data processing text output by the multimodal large model after processing the image-text data to be processed, which corresponds to the image-text data to be processed. When the data processing model provided in this specification is different, the image-text data to be processed is different, and the corresponding first image-text processing result is also different, which will not be elaborated in detail here; for example, the first image-text processing result can be the answer text for image-text question-and-answer data, the communication text for image-text communication data, etc.
[0298] The image data encoding can be understood as the encoded data obtained after encoding the image data using the image processing unit. Similarly, the text sample encoding can be understood as the encoded data obtained after encoding the text data using the text processing unit.
[0299] The text processing unit can be understood as the unit in the first multimodal large model that encodes text data; the image processing unit can be understood as the unit in the first multimodal large model that encodes image data; and the decoding unit can be understood as the unit in the first multimodal large model that decodes the encoded text data and the encoded image data.
[0300] The image weight enhancement parameter for the image data can be understood as a parameter for weight enhancement of the image data encoding; specifically, the image weight enhancement parameter is a parameter used to enhance the attention weight of the image encoding during the generation of the first image-text processing result; the image weight enhancement parameter can be understood as the magnification coefficient in the above embodiment.
[0301] It should be noted that, in one or more embodiments of this specification, the text processing unit, image processing unit, and decoding unit in the first multimodal large model can be used to process the text and image data to be processed, thereby obtaining the first text and image processing result. The steps of the text processing unit, image processing unit, and decoding unit processing the text and image data to be processed can be referred to the steps of processing text and image training samples using the text processing unit, image processing unit, and decoding unit in the first multimodal large model in the aforementioned data processing model training method. That is, in the aforementioned data processing model training method, the text and image training samples are input into the first multimodal large model; the text processing unit in the first multimodal large model encodes the text samples in the text and image training samples to obtain text sample codes; the image processing unit in the first multimodal large model encodes the image samples in the text and image training samples to obtain image sample codes; and the decoding unit in the first multimodal large model and the sample weight enhancement parameters for the image samples decode the text sample codes and the image sample codes to obtain the first sample processing result. Further details are omitted here.
[0302] In one or more embodiments of this specification, the steps of processing the image and text data to be processed using the first multimodal large model can be found in the embodiments of the above-described data processing model training method, specifically the steps of processing the image and text training samples using the first multimodal large model.
[0303] For example, the image and text data to be processed is input into an image-biased LVLMs model for processing, and an amplification factor is applied to each image token to obtain visually enhanced output text (i.e., the first image and text processing result). The specific execution method is as follows: 1. Input the image and text data to be processed into the LVLMs model. 2. Use the language model in the LVLMs model to process the text in the image and text data to obtain text encoding. 3. Use the vision model in the LVLMs model to process the images in the image and text data to obtain image encoding. 4. Input the text encoding and image encoding into the projection module in the LVLMs model for processing to obtain visually enhanced output text. During the projection module's processing of text encoding and image encoding, the attention weight of the image encoding can be adjusted using an amplification factor to generate multiple text tokens, each corresponding to a time step; and these multiple text tokens constitute the visually enhanced (i.e., image feature enhancement) output text. The prediction module can be understood as a module used to predict visually enhanced output text based on text encoding and image encoding. This prediction module can be a decoding unit.
[0304] In one or more embodiments provided in this specification, the step of using the decoding unit in the first multimodal large model and the image weight enhancement parameters for the image data to decode the text data encoding and the image data encoding to obtain the first image-text processing result includes:
[0305] Using the decoding unit in the first multimodal large model, the query parameters corresponding to the text data encoding and the image data encoding are calculated through an attention mechanism, and the key parameters corresponding to the text data encoding and the image data encoding are also calculated.
[0306] The query parameter and the key parameter are multiplied to obtain a parameter product. The image key parameter corresponding to the image data encoding is determined from the key parameter, and the image parameter product corresponding to the image key parameter is determined from the parameter product.
[0307] The image weight enhancement parameter for the image data is added to the image parameter to obtain the image weight enhancement parameter product;
[0308] The text data encoding and the image data encoding are decoded based on the product of the image weight enhancement parameters to obtain the first image-text processing result.
[0309] Regarding the method in this embodiment that uses the decoding unit in the first multimodal large model and the image weight enhancement parameters for the image data to decode the text data encoding and the image data encoding to obtain the first image-text processing result, please refer to the above-mentioned data processing model training method, which uses the decoding unit in the second multimodal large model and the sample weight enhancement parameters for the image samples to decode the text sample encoding and the image sample encoding to obtain the first sample processing result. Further details are omitted here.
[0310] Following the previous example, the prediction module processes text and image data encoding as follows: It calculates the queries (i.e., query parameters) and keys (i.e., key parameters) corresponding to the text and image data encodings using an attention mechanism; then, it multiplies the queries and keys to obtain the attention query-key product (i.e., parameter product). When a key corresponds to an image data encoding (i.e., the nth key token corresponds to an image token), it determines the attention query-key product (i.e., image parameter product) corresponding to that key; it adds the magnification factor c for the image data to this attention query-key product (i.e., image parameter product) to obtain the image enhancement attention query-key product (i.e., image weight enhancement parameter product); and it decodes the text and image data encodings based on the image enhancement attention query-key product to obtain the output text of the first multimodal large model.
[0311] The image data processing method provided in this specification effectively encourages the multimodal large model to consider image information more when generating text responses by finely adjusting the decoding process of the multimodal large model. This effectively avoids the illusion of relying on text and reduces the need for additional parameters or data.
[0312] Step 606: Use the second multimodal large model to process the image and text data to be processed to obtain the second image and text processing result.
[0313] In one or more embodiments provided in this specification, the step of processing the text and image data to be processed using the second multimodal large model to obtain a second text and image processing result includes:
[0314] The image and text data to be processed is input into the second multimodal large model. The text processing unit in the second multimodal large model is used to encode the text data in the image and text data to be processed to obtain the text data encoding.
[0315] The image processing unit in the second multimodal large model is used to encode the image data in the image and text data to be processed to obtain the image data encoding;
[0316] The text data encoding and image data encoding are decoded using the decoding unit in the second multimodal large model to obtain the second image and text processing result.
[0317] In one or more embodiments provided in this specification, the text processing unit, image processing unit, and decoding unit in the second multimodal large model can be used to process the image and text data to be processed, thereby obtaining the second image and text processing result. The steps of using the text processing unit, image processing unit, and decoding unit to process the image and text data to be processed can be found in the steps of processing the second image and text training data using the text processing unit, image processing unit, and decoding unit in the second multimodal large model in the aforementioned data processing model training method. That is, in the aforementioned data processing model training method, the second image and text training data is input into the second multimodal large model; the text processing unit in the second multimodal large model encodes the text samples in the second image and text training data to obtain text sample codes; the image processing unit in the second multimodal large model encodes the image samples in the second image and text training data to obtain image sample codes; and the decoding unit in the second multimodal large model decodes the text sample codes and the image sample codes to obtain the third sample processing result.
[0318] The text processing unit can be understood as the unit in the second multimodal large model used to encode text data; correspondingly, the text data encoding can be understood as the encoded data obtained after the text data is encoded by the text processing unit in the second multimodal large model.
[0319] The image processing unit can be understood as the unit in the second multimodal large model used to encode image data; correspondingly, the image data encoding can be understood as the encoded data obtained after the image data is encoded by the image processing unit in the second multimodal large model.
[0320] This decoding unit can be understood as the unit in the second multimodal large model used for decoding text data encoding and image data encoding.
[0321] Continuing with the previous example, the text and image data to be processed is input into the LVLMs model (the second multimodal large model). Using the language model within the LVLMs model, the text data is processed to obtain text encoding. Using the vision model within the LVLMs model, the image data is processed to obtain image encoding. The text and image encodings are then input into the projection module of the LVLMs model for decoding to obtain the corresponding answer text (i.e., the second text and image processing result). This facilitates the accurate generation of subsequent data processing results based on this second text and image processing result, improving the accuracy of the data processing results.
[0322] Step 608: Use the contrastive learning model to process the first image and text processing result and the second image and text processing result to obtain the image and text processing result corresponding to the image and text data to be processed.
[0323] The image and text processing result can be the result obtained by the data processing model after processing the image and text data to be processed, and the image and text processing result can be the processed text.
[0324] In one or more embodiments provided in this specification, the contrastive learning model is used to process the first image-text processing result and the second image-text processing result to obtain the image-text processing result corresponding to the image-text data to be processed, including steps one and two:
[0325] Step 1: Input the first image processing result and the second image processing result into the contrastive learning model to obtain the processing result.
[0326] The processing result can be understood as a similarity result that characterizes the similarity between the first image and text processing result and the second image and text processing result. For example, the processing result can be similarity or similarity probability.
[0327] In one or more embodiments provided in this specification, inputting the first image-text processing result and the second image-text processing result into the contrastive learning model to obtain the processing result includes:
[0328] The first image and text processing result and the second image and text processing result are input into the contrastive learning model. The encoding unit in the contrastive learning model is used to encode the first image and text processing result to obtain the first image and text processing result encoding, and the second image and text processing result is also encoded to obtain the second image and text processing result encoding.
[0329] Using the contrastive learning model, the encoding of the first image-text processing result and the encoding of the second image-text processing result are processed to obtain similarity results.
[0330] The encoding unit can be understood as the unit in the contrastive learning model used to encode the first and second image processing results.
[0331] The first image-text processing result can be the first image-text processed text data, and the second image-text processing result can be the second image-text processed text data.
[0332] The steps described above, which involve inputting the first and second image processing results into the contrastive learning model to obtain the processing results, can be found in the aforementioned data processing model training method. Specifically, this involves inputting the second and third sample processing results into the contrastive learning model, using the encoding unit in the contrastive learning model to encode the second and third sample processing results to obtain the second sample processing result encoding and the third sample processing result encoding; and then using the contrastive learning model to process the second and third sample processing result encodings to obtain the encoded processing results.
[0333] Following the previous example, after obtaining the corresponding answer texts (i.e., the first and second image-text processing results) through two models (the first and second multimodal large models), the contrastive learning model is used to calculate the similarity between the two answer texts. Specifically, the two answer texts are input into the contrastive learning network, and the encoding network layer in the contrastive learning network encodes the two answer texts to obtain the corresponding text codes (i.e., the code for the second image-text processing result and the code for the third sample image-text result). Then, the contrastive learning network calculates the similarity between the codes of the two answer texts.
[0334] The specific implementation of similarity calculation can be as follows: 1. Input the ordinary encoding (the result of the second image processing) and the visually enhanced encoding (the result of the first image processing) into the encoder in the contrastive learning network for encoding processing to obtain the encoded data corresponding to the ordinary encoding and the encoded data corresponding to the visually enhanced encoding; 2. Input the two encoded data into the projection head (which can be composed of multiple Dense Layers) to compress the two encoded data into a latent space representation, allowing the contrastive learning model to learn the high-order features of the image; 3. Calculate the similarity between the two encoded data (e.g., cosine similarity) and send this similarity to the two models for prediction of the next token.
[0335] Step 2: Based on the first multimodal large model, the second multimodal large model, and the processing result, obtain the image and text processing result corresponding to the image and text data to be processed.
[0336] In one or more embodiments provided in this specification, obtaining the image and text processing result corresponding to the image and text data to be processed based on the first multimodal large model, the second multimodal large model, and the processing result includes:
[0337] Using the first multimodal large model, based on the processing result and the first image and text processing result, a target first image and text processing result is obtained; and using the second multimodal large model, based on the processing result and the second image and text processing result, a target second image and text processing result is obtained.
[0338] Select the image processing result corresponding to the image data to be processed from the first image processing result and the second image processing result.
[0339] In one or more embodiments provided in this specification, the first or second image / text processing result can be understood as the data constituting the image / text processing result, or a part of the image / text processing result. In one or more embodiments provided in this specification, the image / text processing result can be image / text data processing text for the image / text data to be processed, and the image / text data processing text can be composed of multiple text tokens (also called character tokens, tokens, or tokens). Furthermore, these multiple text tokens are generated one after another by the multimodal large model according to time steps; based on this, the first or second image / text processing result can be understood as one or more text tokens constituting the image / text processing result; correspondingly, the first image / text processing result can be a text token generated by the first multimodal large model, and the second image / text processing result can be a text token generated by the second multimodal large model.
[0340] In one or more embodiments provided in this specification, the multimodal large model needs to generate multiple text tokens during the generation of the corresponding image-text processing results. During the reasoning process of these text tokens, the generation of the next text token (a newly added first image-text processing result or a newly added second image-text processing result) can be guided based on the currently generated text token (the second image-text processing result or the second image-text processing result), until all text tokens (i.e., the target first image-text processing result or the target second image-text processing result) are generated. Correspondingly, the target first image-text processing result can be understood as all text tokens generated by the first multimodal large model, and the target second image-text processing result can be understood as all text tokens generated by the second multimodal large model; a newly added first image-text processing result can be understood as another first image-text processing result generated based on the first image-text processing result, and a newly added second image-text processing result can be understood as another second image-text processing result generated based on the second image-text processing result.
[0341] Selecting the image processing result corresponding to the image data to be processed from the first target image processing result and the second target image processing result can be understood as selecting the first target image processing result as the image processing result corresponding to the image data to be processed from the first target image processing result and the second target image processing result. Alternatively, either the first target image processing result or the second target image processing result can be randomly selected as the image processing result corresponding to the image data to be processed. For example, when obtaining the token at the i-th time step, the tokens output by the two models need to be input into a contrastive learning network for comparison to obtain similarity; this similarity is used to guide the two models to predict the next token until the final prediction result (the first target image processing result and the second target image processing result) is obtained. Since the prediction result is guided by similarity, the final first target image processing result and the second target image processing result have a very high similarity. Therefore, either the second image processing result (ordinary inference result) or the first image processing result (image enhancement inference result) can be used as the output of the multimodal large model. Alternatively, the first image processing result (image enhancement inference result) can be directly used as the output of the multimodal large model.
[0342] In one or more embodiments provided in this specification, the processing result is a similarity result;
[0343] The step of obtaining a target first image-text processing result using the first multimodal large model, based on the processing result and the first image-text processing result, and obtaining a target second image-text processing result using the second multimodal large model, based on the processing result and the second image-text processing result, includes:
[0344] The similarity results are input into the first multimodal large model and the first multimodal large model;
[0345] Using the first multimodal large model, the image and text data to be processed is processed based on the similarity results to obtain a newly added first image and text processing result, and based on the first image and text processing result and the newly added first image and text processing result, a target first image and text processing result is obtained;
[0346] Using the second multimodal large model, the text and image data to be processed is processed based on the similarity results to obtain a new second text and image processing result, and based on the second text and image processing result and the new second text and image processing result, a target second text and image processing result is obtained.
[0347] Following the previous example, the multimodal large model needs to generate multiple text tokens during the generation of the corresponding answer text. During the reasoning process for this text token, two text tokens from the same time step (i.e., the first image-text processing result and the second image-text processing result) can be input into the contrastive learning network for similarity calculation. The calculated similarity then guides the generation of the next text token (i.e., adding the first image-text processing result and adding the second image-text processing result). Specifically, ordinary text tokens (the second image-text processing result) and visually enhanced text tokens (i.e., the first image-text processing result) can be input into the encoder of the contrastive learning network for encoding, obtaining the encoded data corresponding to the ordinary text token and the encoded data corresponding to the visually enhanced text token. These two encoded data are then input into the projection head (which can consist of multiple dense layers), compressing them into a latent space representation, allowing the contrastive learning model to learn the high-order features of the image. Based on this latent space representation, the similarity result (e.g., cosine similarity) between two encoded data is calculated. This similarity result is then sent to two multimodal large models for predicting the next token (i.e., the newly added first image-text processing result and the newly added second image-text processing result). This process continues until all text tokens (i.e., the target first image-text processing result and the target second image-text processing result) are inferred, thus obtaining the answer text. Based on this, the similarity between the text tokens output by the two models guides the prediction of the next text token, thereby improving the accuracy of the prediction results of both models.
[0348] In one or more embodiments provided in this specification, determining the image and text data to be processed includes:
[0349] Receive unprocessed image and text data sent by the client, wherein the unprocessed image and text data is sent by the user through the image and text data processing interface in the client;
[0350] The step of processing the first and second image-text processing results using the contrastive learning model to obtain the image-text processing result corresponding to the image-text data to be processed includes:
[0351] The image and text processing results are sent to the client so that the client can display the image and text processing results to the user through the image and text data processing interface.
[0352] The image and text data processing interface can be understood as an interface used for interaction with the client. Through this interface, the client and the server (the server to which the image and text data processing method is applied) can be instructed to perform the image and text data processing steps (i.e., the steps corresponding to one or more of the above embodiments) to obtain the image and text processing results. This image and text data processing interface can be an application interface, a webpage, etc.
[0353] The image and text data processing method in one or more embodiments provided in this specification, after determining the image and text data to be processed, inputs it into a data processing model, uses a first multimodal large model and a second multimodal large model to process the image and text data to be processed, obtains a first image and text processing result and a second image and text processing result, and then uses a contrastive learning model to perform contrastive learning processing based on the first image and text processing result and the second image and text processing result, thereby obtaining an image and text processing result that accurately corresponds to the image and text data to be processed. This avoids the problem of knowledge illusion in the trained multimodal large model, solves the problem of the multimodal large model generating information that is irrelevant or incorrect to the input image and text data, and further solves the problem of not being able to process the user's image and text data in actual scenarios, meeting the actual needs of image data processing in real scenarios.
[0354] The above is an illustrative scheme of a text and image data processing method according to this embodiment. It should be noted that the technical solution of this text and image data processing method belongs to the same concept as the technical solutions of the data processing model training method and the machine learning model training method described above. For details not described in detail in the technical solution of the text and image data processing method, please refer to the descriptions of the technical solutions of the data processing model training method and the machine learning model training method described above.
[0355] See Figure 7 , Figure 7 A flowchart of a data processing method according to an embodiment of this specification is shown, which specifically includes the following steps.
[0356] Step 702: Determine the data to be processed and input the data to be processed into the data processing model, wherein the data processing model includes a first data processing module, a second data processing module and a parameter learning module, and the first data processing module is determined based on the second data processing module.
[0357] Step 704: Use the first data processing module to process the data to be processed to obtain the first data processing result.
[0358] In one or more embodiments provided in this specification, the step of processing the data to be processed using the first data processing module to obtain a first data processing result includes:
[0359] The data to be processed is input into the first data processing module, and the first modal processing unit in the first data processing module is used to encode the first modal data in the data to be processed to obtain the first modal data encoding.
[0360] The second modal processing unit in the first data processing module is used to encode the second modal data in the data to be processed to obtain the second modal data encoding.
[0361] Using the decoding unit in the first data processing module and the modal weight enhancement parameters for the second modal data, the first modal data encoding and the second modal data encoding are decoded to obtain the first data processing result.
[0362] In one or more embodiments provided in this specification, the step of using the decoding unit in the first data processing module and the modality weight enhancement parameters for the second modality data to decode the first modality data encoding and the second modality data encoding to obtain the first data processing result includes:
[0363] Using the decoding unit in the first data processing module, the query parameters corresponding to the first modal data encoding and the second modal data encoding are calculated through an attention mechanism, and the key parameters corresponding to the first modal data encoding and the second modal data encoding are also calculated.
[0364] The query parameter and the key parameter are multiplied to obtain a parameter product. The second modal key parameter corresponding to the second modal data encoding is determined from the key parameter, and the second modal parameter product corresponding to the second modal key parameter is determined from the parameter product.
[0365] The modality weight enhancement parameter for the second modality data is added to the product of the second modality parameter to obtain the second modality weight enhancement parameter product;
[0366] The first modal data encoding and the second modal data encoding are decoded based on the product of the second modal weight enhancement parameters to obtain the first data processing result.
[0367] Step 706: Use the second data processing module to process the data to be processed to obtain the second data processing result.
[0368] In one or more embodiments provided in this specification, the step of processing the data to be processed using the second data processing module to obtain a second data processing result includes:
[0369] The data to be processed is input into the second data processing module, and the first modal processing unit in the second data processing module is used to encode the first modal data in the data to be processed to obtain the first modal data encoding.
[0370] The second modal processing unit in the second data processing module is used to encode the second modal data in the data to be processed to obtain the second modal data encoding.
[0371] The decoding unit in the second data processing module is used to decode the first modal data encoding and the second modal data encoding to obtain the second data processing result.
[0372] Step 708: Using the parameter learning module, process the first data processing result and the second data processing result to obtain the data processing result corresponding to the data to be processed.
[0373] In one or more embodiments provided in this specification, the step of using the parameter learning module to process the first data processing result and the second data processing result to obtain the data processing result corresponding to the data to be processed includes:
[0374] The first data processing result and the second data processing result are input into the parameter learning model to obtain the processing result;
[0375] Based on the first data processing module, the second data processing module, and the processing result, the data processing result corresponding to the data to be processed is obtained.
[0376] In one or more embodiments provided in this specification, inputting the first data processing result and the second data processing result into the parameter learning model to obtain the processing result includes:
[0377] The first data processing result and the second data processing result are input into the parameter learning module. The encoding unit in the parameter learning module is used to encode the first data processing result to obtain the first data processing result encoding, and the second data processing result is encoded to obtain the second data processing result encoding.
[0378] The parameter learning module is used to process the first data processing result encoding and the second data processing result encoding to obtain the processing result.
[0379] In one or more embodiments provided in this specification, obtaining the data processing result corresponding to the data to be processed based on the first data processing module, the second data processing module, and the processing result includes:
[0380] Using the first data processing module, a target first data processing result is obtained based on the processing result and the first data processing result; and using the second data processing module, a target second data processing result is obtained based on the processing result and the second data processing result.
[0381] Select the data processing result corresponding to the data to be processed from the first data processing result and the second data processing result.
[0382] In one or more embodiments provided in this specification, the step of obtaining a target first data processing result using the first data processing module based on the processing result and the first data processing result, and obtaining a target second data processing result using the second data processing module based on the processing result and the second data processing result, includes:
[0383] The processing result is input into the first data processing module and the first data processing module;
[0384] Using the first data processing module, the data to be processed is processed based on the processing result to obtain a new first data processing result, and based on the first data processing result and the new first data processing result, a target first data processing result is obtained;
[0385] Using the second data processing module, the data to be processed is processed based on the processing result to obtain a new second data processing result, and based on the second data processing result and the new second data processing result, a target second data processing result is obtained.
[0386] In one or more embodiments provided in this specification, determining the data to be processed includes:
[0387] Receive pending data sent by the client, wherein the pending data is sent by the user through the data processing interface in the client;
[0388] The step of using the parameter learning module to process the first data processing result and the second data processing result to obtain the data processing result corresponding to the data to be processed includes:
[0389] The data processing result is sent to the client so that the client can display the data processing result to the user through the data processing interface.
[0390] One data processing method provided in one or more embodiments of this specification, after determining the data to be processed, inputs it into a data processing model, processes the data to be processed using a first data processing module and a second data processing module to obtain a first data processing result and a second data processing result, and then uses a parameter learning model to perform parameter learning processing based on the first and second data processing results, thereby obtaining a data processing result that accurately corresponds to the data to be processed. This avoids the problem of knowledge illusion in the data processing model, solves the problem of the data processing model generating information that is irrelevant or incorrect to the input data, and further solves the problem of not being able to process user data in real-world scenarios, thus meeting the actual needs of data processing in real-world scenarios.
[0391] The above is an illustrative scheme of a data processing method according to this embodiment. It should be noted that the technical solution of this data processing method belongs to the same concept as the above-described image and text data processing method, data processing model training method, and machine learning model training method. For details not described in detail in the technical solution of the data processing method, please refer to the description of the above-described image and text data processing method, data processing model training method, and machine learning model training method.
[0392] Corresponding to the above method embodiments, this specification also provides an embodiment of a data processing model training apparatus, which includes:
[0393] The training data determination module is configured to determine the data processing model to be trained, as well as the first image and text training data and the second image and text training data associated with the data processing task. The data processing model to be trained includes a first multimodal large model, a second multimodal large model and a contrastive learning model, and the first multimodal large model is determined based on the second multimodal large model.
[0394] The first model training module is configured to train the first multimodal large model based on the first image and text training data, and obtain the trained first multimodal large model.
[0395] The processing result determination module is configured to process the second image and text training data based on the trained first multimodal large model, the second multimodal large model, and the contrastive learning model to obtain the processing result;
[0396] The second model training module is configured to adjust the model parameters of the trained first multimodal large model and the contrastive learning model based on the processing results, so as to obtain a trained data processing model, wherein the data processing model is used to perform the data processing task.
[0397] The data processing model training device provided in one or more embodiments of this specification trains a data processing model including a first multimodal large model, a second multimodal large model, and a contrastive learning model. During model training, the first multimodal large model undergoes a first-stage training. Then, based on the trained first multimodal large model, second multimodal large model, and contrastive learning model, the processing results obtained from processing the second image and text training data are processed. The model parameters of the trained first multimodal large model and contrastive learning model are adjusted again. Based on factors such as the second multimodal large model and contrastive learning model, the trained first multimodal large model is trained again, thereby obtaining a data processing model capable of accurately generating correct information related to the input image and text data. This avoids the problem of knowledge illusion in the trained multimodal large model and solves the problem of the trained multimodal large model generating irrelevant or incorrect information related to the input image and text data. Furthermore, it solves the problem of being unable to process user image and text data in real-world scenarios, meeting the actual needs for image data processing in real-world scenarios.
[0398] The above is an illustrative scheme of a data processing model training device according to this embodiment. It should be noted that the technical solution of this data processing model training device and the technical solution of the data processing model training method described above belong to the same concept. For details not described in detail in the technical solution of the data processing model training device, please refer to the description of the technical solution of the data processing model training method described above.
[0399] Corresponding to the above method embodiments, this specification also provides embodiments of a machine learning model training apparatus, which includes:
[0400] The training data determination module is configured to determine the machine learning model to be trained, as well as the first training data and the second training data associated with the target task, wherein the machine learning model to be trained includes a first data processing module, a second data processing module, and a parameter learning module.
[0401] The first model training module is configured to train the first data processing module based on the first training data to obtain the trained first data processing module.
[0402] The processing result determination module is configured to process the second training data according to the first training data processing module, the second data processing module, and the parameter learning module after training, and obtain the processing result;
[0403] The second model training module is configured to adjust the model parameters of the trained first data processing module and the parameter learning module according to the processing results, so as to obtain the trained target model, wherein the target model is used to perform the target task.
[0404] The machine learning model training apparatus in one or more embodiments provided in this specification trains a machine learning model including a first data processing module, a second data processing module, and a parameter learning module. During model training, a first-stage training is first performed on the first data processing module. Then, based on the processed results obtained from processing second training data using the trained first data processing module, second data processing module, and parameter learning module, the model parameters of the trained first data processing module and parameter learning module are adjusted again. Thus, based on factors such as the second data processing module and parameter learning module, the trained first data processing module is trained again, thereby obtaining a machine learning model with high performance and high efficiency in processing target tasks. This avoids the problem of poor performance in trained machine learning models that cannot efficiently process target tasks, facilitating the processing of user-related target tasks.
[0405] The above is an illustrative scheme of a machine learning model training device according to this embodiment. It should be noted that the technical solution of this machine learning model training device and the technical solution of the above-described machine learning model training method belong to the same concept. For details not described in detail in the technical solution of the machine learning model training device, please refer to the description of the technical solution of the above-described machine learning model training method.
[0406] Corresponding to the above method embodiments, this specification also provides another data processing apparatus embodiment, which includes:
[0407] The data determination module is configured to determine the text and image data to be processed input by the user;
[0408] The data processing module is configured to input the text and image data to be processed into a data processing model to obtain the data processing text corresponding to the text and image data to be processed. The data processing model includes a first multimodal large model, a second multimodal large model, and a contrastive learning model. The first multimodal large model is determined based on the second multimodal large model. The first multimodal large model is used to process the text and image data to be processed to obtain a first text and image processing result. The second multimodal large model is used to process the text and image data to be processed to obtain a second text and image processing result. The contrastive learning model is used to process the first text and image processing result and the second text and image processing result to obtain the data processing text corresponding to the text and image data to be processed.
[0409] Another data processing apparatus provided in one or more embodiments of this specification can process the image and text data to be processed using a data processing model that accurately generates information related to and correct to the input image and text data. This avoids the problem of multimodal large models generating information that is irrelevant or incorrect to the input image and text data, thereby solving the image and text data processing problem faced by users in real-world scenarios and further facilitating users in processing image and text data.
[0410] The above is an illustrative scheme of another data processing device according to this embodiment. It should be noted that the technical solution of this data processing device and the technical solution of the other data processing method described above belong to the same concept. For details not described in detail in the technical solution of the data processing device, please refer to the description of the technical solution of the other data processing method described above.
[0411] Corresponding to the above method embodiments, this specification also provides an embodiment of a graphic data processing apparatus, which includes:
[0412] The data determination module is configured to determine the text and image data to be processed and input the text and image data to be processed into the data processing model, wherein the data processing model includes a first multimodal large model, a second multimodal large model and a contrastive learning model, and the first multimodal large model is determined based on the second multimodal large model;
[0413] The first data processing module is configured to process the image and text data to be processed using the first multimodal large model to obtain a first image and text processing result;
[0414] The second data processing module is configured to process the image and text data to be processed using the second multimodal large model to obtain the second image and text processing result;
[0415] The result determination module is configured to use the contrastive learning model to process the first image and text processing result and the second image and text processing result to obtain the image and text processing result corresponding to the image and text data to be processed.
[0416] Optionally, the first data processing module is further configured to:
[0417] The text and image data to be processed is input into the first multimodal large model. The text processing unit in the first multimodal large model is used to encode the text data in the text and image data to be processed to obtain the text data encoding.
[0418] The image processing unit in the first multimodal large model is used to encode the image data in the image and text data to be processed to obtain the image data encoding;
[0419] Using the decoding unit in the first multimodal large model and the image weight enhancement parameters for the image data, the text data encoding and image data encoding are decoded to obtain the first image-text processing result.
[0420] Optionally, the first data processing module is further configured to:
[0421] Using the decoding unit in the first multimodal model, the query parameters corresponding to the text data encoding and the image data encoding are calculated through an attention mechanism, and the key parameters corresponding to the text data encoding and the image data encoding are also calculated.
[0422] The query parameter and the key parameter are multiplied to obtain a parameter product. The image key parameter corresponding to the image data encoding is determined from the key parameter, and the image parameter product corresponding to the image key parameter is determined from the parameter product.
[0423] The image weight enhancement parameter for the image data is multiplied by the image parameter to obtain the image weight enhancement parameter product;
[0424] The text data encoding and the image data encoding are decoded based on the product of the image weight enhancement parameters to obtain the first image-text processing result.
[0425] Optionally, the second data processing module is further configured to:
[0426] The image and text data to be processed is input into the second multimodal large model. The text processing unit in the second multimodal large model is used to encode the text data in the image and text data to be processed to obtain the text data encoding.
[0427] The image processing unit in the second multimodal large model is used to encode the image data in the image and text data to be processed to obtain the image data encoding;
[0428] The text data encoding and image data encoding are decoded using the decoding unit in the second multimodal large model to obtain the second image and text processing result.
[0429] Optionally, the result determination module is further configured to:
[0430] The first image processing result and the second image processing result are input into the contrastive learning model to obtain the processing result;
[0431] Based on the first multimodal large model, the second multimodal large model, and the processing result, the image and text processing result corresponding to the image and text data to be processed is obtained.
[0432] Optionally, the result determination module is further configured to:
[0433] The first image and text processing result and the second image and text processing result are input into the contrastive learning model. The encoding unit in the contrastive learning model is used to encode the first image and text processing result to obtain the first image and text processing result encoding, and the second image and text processing result is also encoded to obtain the second image and text processing result encoding.
[0434] Using the contrastive learning model, the encoding of the first image-text processing result and the encoding of the second image-text processing result are processed to obtain similarity results.
[0435] Optionally, the result determination module is further configured to:
[0436] Using the first multimodal large model, based on the processing result and the first image and text processing result, a target first image and text processing result is obtained; and using the second multimodal large model, based on the processing result and the second image and text processing result, a target second image and text processing result is obtained.
[0437] Select the image processing result corresponding to the image data to be processed from the first image processing result and the second image processing result.
[0438] Optionally, the processing result is a similarity result;
[0439] The result determination module is further configured to:
[0440] The similarity results are input into the first multimodal large model and the first multimodal large model;
[0441] Using the first multimodal large model, the image and text data to be processed is processed based on the similarity results to obtain a newly added first image and text processing result, and based on the first image and text processing result and the newly added first image and text processing result, a target first image and text processing result is obtained;
[0442] Using the second multimodal large model, the text and image data to be processed is processed based on the similarity results to obtain a new second text and image processing result, and based on the second text and image processing result and the new second text and image processing result, a target second text and image processing result is obtained.
[0443] Optionally, the data determination module is further configured to:
[0444] Receive unprocessed image and text data sent by the client, wherein the unprocessed image and text data is sent by the user through the image and text data processing interface in the client;
[0445] The image and text data processing device further includes a result sending module, configured as follows:
[0446] The image and text processing results are sent to the client so that the client can display the image and text processing results to the user through the image and text data processing interface.
[0447] The image and text data processing apparatus in one or more embodiments provided in this specification, after determining the image and text data to be processed, inputs it into a data processing model, processes the image and text data to be processed using a first multimodal large model and a second multimodal large model, obtains a first image and text processing result and a second image and text processing result, and then uses a contrastive learning model to perform contrastive learning processing based on the first image and text processing result and the second image and text processing result, thereby obtaining an image and text processing result that accurately corresponds to the image and text data to be processed. This avoids the problem of knowledge illusion in multimodal large models, solves the problem of multimodal large models generating information that is irrelevant or incorrect to the input image and text data, and further solves the problem of not being able to process user image and text data in actual scenarios, thus meeting the actual needs of image data processing in real scenarios.
[0448] The above is a schematic scheme of a text and image data processing device according to this embodiment. It should be noted that the technical solution of this text and image data processing device and the technical solution of the above-described text and image data processing method belong to the same concept. For details not described in detail in the technical solution of the text and image data processing device, please refer to the description of the technical solution of the above-described text and image data processing method.
[0449] Corresponding to the above method embodiments, this specification also provides a data processing apparatus embodiment, the apparatus comprising:
[0450] A data determination module is configured to determine the data to be processed and input the data to be processed into a data processing model, wherein the data processing model includes a first data processing module, a second data processing module and a parameter learning module, and the first data processing module is determined based on the second data processing module;
[0451] The first data processing module is configured to process the data to be processed using the first data processing module to obtain a first data processing result;
[0452] The second data processing module is configured to process the data to be processed using the second data processing module to obtain a second data processing result;
[0453] The result determination module is configured to use the parameter learning module to process the first data processing result and the second data processing result to obtain the data processing result corresponding to the data to be processed.
[0454] Optionally, the first data processing module is further configured to:
[0455] The data to be processed is input into the first data processing module, and the first modal processing unit in the first data processing module is used to encode the first modal data in the data to be processed to obtain the first modal data encoding.
[0456] The second modal processing unit in the first data processing module is used to encode the second modal data in the data to be processed to obtain the second modal data encoding.
[0457] Using the decoding unit in the first data processing module and the modal weight enhancement parameters for the second modal data, the first modal data encoding and the second modal data encoding are decoded to obtain the first data processing result.
[0458] Optionally, the first data processing module is further configured to:
[0459] Using the decoding unit in the first data processing module, the query parameters corresponding to the first modal data encoding and the second modal data encoding are calculated through an attention mechanism, and the key parameters corresponding to the first modal data encoding and the second modal data encoding are also calculated.
[0460] The query parameter and the key parameter are multiplied to obtain a parameter product. The second modal key parameter corresponding to the second modal data encoding is determined from the key parameter, and the second modal parameter product corresponding to the second modal key parameter is determined from the parameter product.
[0461] The modality weight enhancement parameter for the second modality data is added to the product of the second modality parameter to obtain the second modality weight enhancement parameter product;
[0462] The first modal data encoding and the second modal data encoding are decoded based on the product of the second modal weight enhancement parameters to obtain the first data processing result.
[0463] Optionally, the second data processing module is further configured to:
[0464] The data to be processed is input into the second data processing module, and the first modal processing unit in the second data processing module is used to encode the first modal data in the data to be processed to obtain the first modal data encoding.
[0465] The second modal processing unit in the second data processing module is used to encode the second modal data in the data to be processed to obtain the second modal data encoding.
[0466] The decoding unit in the second data processing module is used to decode the first modal data encoding and the second modal data encoding to obtain the second data processing result.
[0467] Optionally, the result determination module is further configured to:
[0468] The first data processing result and the second data processing result are input into the parameter learning model to obtain the processing result;
[0469] Based on the first data processing module, the second data processing module, and the processing result, the data processing result corresponding to the data to be processed is obtained.
[0470] Optionally, the result determination module is further configured to:
[0471] The first data processing result and the second data processing result are input into the parameter learning module. The encoding unit in the parameter learning module is used to encode the first data processing result to obtain the first data processing result encoding, and the second data processing result is encoded to obtain the second data processing result encoding.
[0472] The parameter learning module is used to process the first data processing result encoding and the second data processing result encoding to obtain the processing result.
[0473] Optionally, the result determination module is further configured to:
[0474] Using the first data processing module, a target first data processing result is obtained based on the processing result and the first data processing result; and using the second data processing module, a target second data processing result is obtained based on the processing result and the second data processing result.
[0475] Select the data processing result corresponding to the data to be processed from the first data processing result and the second data processing result.
[0476] Optionally, the result determination module is further configured to:
[0477] The processing result is input into the first data processing module and the first data processing module;
[0478] Using the first data processing module, the data to be processed is processed based on the processing result to obtain a new first data processing result, and based on the first data processing result and the new first data processing result, a target first data processing result is obtained;
[0479] Using the second data processing module, the data to be processed is processed based on the processing result to obtain a new second data processing result, and based on the second data processing result and the new second data processing result, a target second data processing result is obtained.
[0480] Optionally, the data determination module is further configured to:
[0481] Receive pending data sent by the client, wherein the pending data is sent by the user through the data processing interface in the client;
[0482] The image and text data processing device further includes a result sending module, configured as follows:
[0483] The data processing result is sent to the client so that the client can display the data processing result to the user through the data processing interface.
[0484] One or more embodiments of the data processing apparatus provided in this specification, after determining the data to be processed, inputs it into a data processing model, processes the data to be processed using the first data processing module and the second data processing module to obtain a first data processing result and a second data processing result, and then uses a parameter learning model to perform parameter learning processing based on the first data processing result and the second data processing result, thereby obtaining a data processing result that accurately corresponds to the data to be processed. This avoids the problem of knowledge illusion in the data processing model, solves the problem of the data processing model generating information that is irrelevant or incorrect to the input data, and further solves the problem of not being able to process user data in real-world scenarios, thus meeting the actual needs of data processing in real-world scenarios.
[0485] The above is an illustrative scheme of a data processing apparatus according to this embodiment. It should be noted that the technical solution of this data processing apparatus and the technical solution of the data processing method described above belong to the same concept. For details not described in detail in the technical solution of the data processing apparatus, please refer to the description of the technical solution of the data processing method described above.
[0486] Figure 8A structural block diagram of a computing device 800 according to one embodiment of this specification is shown. The components of the computing device 800 include, but are not limited to, a memory 810 and a processor 820. The processor 820 is connected to the memory 810 via a bus 830, and a database 850 is used to store data.
[0487] The computing device 800 also includes an access device 840, which enables the computing device 800 to communicate via one or more networks 860. Examples of these networks include Public Switched Telephone Network (PSTN), Local Area Network (LAN), Wide Area Network (WAN), Personal Area Network (PAN), or combinations of communication networks such as the Internet. The access device 840 may include one or more of any type of wired or wireless network interface (e.g., a network interface card (NIC)), such as an IEEE 802.11 Wireless Local Area Network (WLAN) wireless interface, a Wi-MAX (Worldwide Interoperability for Microwave Access) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, or a Near Field Communication (NFC) interface.
[0488] In one embodiment of this specification, the above-described components of the computing device 800 and Figure 8 Other components, not shown, can also be connected to each other, for example, via a bus. It should be understood that... Figure 8 The block diagram of the computing device shown is for illustrative purposes only and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.
[0489] The computing device 800 can be any type of stationary or mobile computing device, including mobile computers or mobile computing devices (e.g., tablet computers, personal digital assistants, laptop computers, notebook computers, netbooks, etc.), mobile phones (e.g., smartphones), wearable computing devices (e.g., smartwatches, smart glasses, etc.) or other types of mobile devices, or stationary computing devices such as desktop computers or personal computers (PCs). The computing device 800 can also be a mobile or stationary server.
[0490] The processor 820 is configured to execute the following computer-executable instructions, which, when executed by the processor, implement the steps of the aforementioned machine learning model training method, data processing model training method, or data processing method.
[0491] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on its differences from other embodiments. In particular, the computing device embodiments are basically similar to embodiments of a machine learning model training method, a data processing model training method, or a data processing method, so the description is relatively simple. Relevant parts can be referred to in the descriptions of embodiments of a machine learning model training method, a data processing model training method, or a data processing method.
[0492] An embodiment of this specification also provides a computer-readable storage medium storing a computer program / instructions that, when executed by a processor, implement the steps of the above-described image and text data processing method, data processing method, machine learning model training method, data processing model training method, or another data processing method.
[0493] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the computer-readable storage medium embodiments are relatively simple in description because they are substantially similar to embodiments of a text and image data processing method, a data processing method, a machine learning model training method, a data processing model training method, or another data processing method. Relevant parts can be referred to in the description of embodiments of a text and image data processing method, a data processing method, a machine learning model training method, a data processing model training method, or another data processing method.
[0494] An embodiment of this specification also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the above-described image and text data processing method, a data processing method, a machine learning model training method, a data processing model training method, or another data processing method.
[0495] The above is an illustrative scheme of a computer program product according to this embodiment. It should be noted that the technical solution of this computer program product belongs to the same concept as the aforementioned technical solutions of a text and image data processing method, a data processing method, a machine learning model training method, a data processing model training method, or another data processing method. Details not described in detail in the technical solution of the computer program product can be found in the descriptions of the aforementioned technical solutions of a text and image data processing method, a data processing method, a machine learning model training method, a data processing model training method, or another data processing method.
[0496] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.
[0497] The computer instructions include computer program code, which may be in the form of source code, object code, executable file, or certain intermediate forms. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium may be appropriately added or removed according to the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media may not include electrical carrier signals and telecommunication signals.
[0498] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments in this specification are not limited to the described order of actions, because according to the embodiments in this specification, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the embodiments in this specification.
[0499] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.
[0500] The preferred embodiments disclosed above are merely illustrative of this specification. The optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the embodiments described herein. These embodiments are selected and specifically described in this specification to better explain the principles and practical applications of the embodiments, thereby enabling those skilled in the art to better understand and utilize this specification. This specification is limited only by the claims and their full scope and equivalents.
Claims
1. A data processing method, comprising: The data to be processed is determined and input into the data processing model, wherein the data processing model includes a first data processing module, a second data processing module and a parameter learning module, and the first data processing module is determined based on the second data processing module; The first data processing module is used to process the data to be processed to obtain a first data processing result; The second data processing module is used to process the data to be processed to obtain a second data processing result; The parameter learning module is used to process the first data processing result and the second data processing result to obtain the data processing result corresponding to the data to be processed.
2. The data processing method according to claim 1, wherein processing the data to be processed using the first data processing module to obtain a first data processing result includes: The data to be processed is input into the first data processing module, and the first modal processing unit in the first data processing module is used to encode the first modal data in the data to be processed to obtain the first modal data encoding. The second modal processing unit in the first data processing module is used to encode the second modal data in the data to be processed to obtain the second modal data encoding. Using the decoding unit in the first data processing module and the modal weight enhancement parameters for the second modal data, the first modal data encoding and the second modal data encoding are decoded to obtain the first data processing result.
3. The data processing method according to claim 2, wherein the step of using the decoding unit in the first data processing module and the modality weight enhancement parameters for the second modality data to perform decoding processing on the encoding of the first modality data and the encoding of the second modality data to obtain the first data processing result includes: Using the decoding unit in the first data processing module, the query parameters corresponding to the first modal data encoding and the second modal data encoding are calculated through an attention mechanism, and the key parameters corresponding to the first modal data encoding and the second modal data encoding are also calculated. The query parameter and the key parameter are multiplied to obtain a parameter product. The second modal key parameter corresponding to the second modal data encoding is determined from the key parameter, and the second modal parameter product corresponding to the second modal key parameter is determined from the parameter product. The modality weight enhancement parameter for the second modality data is added to the product of the second modality parameter to obtain the second modality weight enhancement parameter product; The first modal data encoding and the second modal data encoding are decoded based on the product of the second modal weight enhancement parameters to obtain the first data processing result.
4. The data processing method according to claim 1, wherein processing the data to be processed using the second data processing module to obtain a second data processing result includes: The data to be processed is input into the second data processing module, and the first modal processing unit in the second data processing module is used to encode the first modal data in the data to be processed to obtain the first modal data encoding. The second modal processing unit in the second data processing module is used to encode the second modal data in the data to be processed to obtain the second modal data encoding. The decoding unit in the second data processing module is used to decode the first modal data encoding and the second modal data encoding to obtain the second data processing result.
5. The data processing method according to claim 1, wherein processing the first data processing result and the second data processing result using the parameter learning module to obtain the data processing result corresponding to the data to be processed includes: The first data processing result and the second data processing result are input into the parameter learning model to obtain the processing result; Based on the first data processing module, the second data processing module, and the processing result, the data processing result corresponding to the data to be processed is obtained.
6. The data processing method according to claim 5, wherein inputting the first data processing result and the second data processing result into the parameter learning model to obtain the processing result includes: The first data processing result and the second data processing result are input into the parameter learning module. The encoding unit in the parameter learning module is used to encode the first data processing result to obtain the first data processing result encoding, and the second data processing result is encoded to obtain the second data processing result encoding. The parameter learning module is used to process the first data processing result encoding and the second data processing result encoding to obtain the processing result.
7. The data processing method according to claim 5, wherein obtaining the data processing result corresponding to the data to be processed based on the first data processing module, the second data processing module, and the processing result includes: Using the first data processing module, a target first data processing result is obtained based on the processing result and the first data processing result; and using the second data processing module, a target second data processing result is obtained based on the processing result and the second data processing result. Select the data processing result corresponding to the data to be processed from the first data processing result and the second data processing result.
8. The data processing method according to claim 7, wherein the step of obtaining a target first data processing result using the first data processing module based on the processing result and the first data processing result, and obtaining a target second data processing result using the second data processing module based on the processing result and the second data processing result, comprises: The processing result is input into the first data processing module and the first data processing module; Using the first data processing module, the data to be processed is processed based on the processing result to obtain a new first data processing result, and based on the first data processing result and the new first data processing result, a target first data processing result is obtained; Using the second data processing module, the data to be processed is processed based on the processing result to obtain a new second data processing result, and based on the second data processing result and the new second data processing result, a target second data processing result is obtained.
9. The data processing method according to claim 1, wherein determining the data to be processed includes: Receive pending data sent by the client, wherein the pending data is sent by the user through the data processing interface in the client; The step of using the parameter learning module to process the first data processing result and the second data processing result to obtain the data processing result corresponding to the data to be processed includes: The data processing result is sent to the client so that the client can display the data processing result to the user through the data processing interface.
10. A method for processing graphic and textual data, comprising: The text and image data to be processed is determined, and the text and image data to be processed is input into the data processing model. The data processing model includes a first multimodal large model, a second multimodal large model, and a contrastive learning model. The first multimodal large model is determined based on the second multimodal large model. The first multimodal large model is used to process the image and text data to be processed to obtain the first image and text processing result; The second multimodal large model is used to process the image and text data to be processed to obtain the second image and text processing result; The first and second image processing results are processed using the contrastive learning model to obtain the image processing result corresponding to the image data to be processed.
11. The image and text data processing method according to claim 10, wherein processing the image and text data to be processed using the first multimodal large model to obtain a first image and text processing result includes: The text and image data to be processed is input into the first multimodal large model. The text processing unit in the first multimodal large model is used to encode the text data in the text and image data to be processed to obtain the text data encoding. The image processing unit in the first multimodal large model is used to encode the image data in the image and text data to be processed to obtain the image data encoding; Using the decoding unit in the first multimodal large model and the image weight enhancement parameters for the image data, the text data encoding and image data encoding are decoded to obtain the first image-text processing result.
12. The image and text data processing method according to claim 11, wherein the step of using the decoding unit in the first multimodal large model and the image weight enhancement parameters for the image data to perform decoding processing on the text data encoding and the image data encoding to obtain the first image and text processing result includes: Using the decoding unit in the first multimodal large model, the query parameters corresponding to the text data encoding and the image data encoding are calculated through an attention mechanism, and the key parameters corresponding to the text data encoding and the image data encoding are also calculated. The query parameter and the key parameter are multiplied to obtain a parameter product. The image key parameter corresponding to the image data encoding is determined from the key parameter, and the image parameter product corresponding to the image key parameter is determined from the parameter product. The image weight enhancement parameter for the image data is added to the image parameter to obtain the image weight enhancement parameter product; The text data encoding and the image data encoding are decoded based on the product of the image weight enhancement parameters to obtain the first image-text processing result.
13. The image and text data processing method according to claim 10, wherein processing the image and text data to be processed using the second multimodal large model to obtain a second image and text processing result includes: The image and text data to be processed is input into the second multimodal large model. The text processing unit in the second multimodal large model is used to encode the text data in the image and text data to be processed to obtain the text data encoding. The image processing unit in the second multimodal large model is used to encode the image data in the image and text data to be processed to obtain the image data encoding; The text data encoding and image data encoding are decoded using the decoding unit in the second multimodal large model to obtain the second image and text processing result.
14. The image and text data processing method according to claim 10, wherein the first image and text processing result and the second image and text processing result are processed using the contrastive learning model to obtain the image and text processing result corresponding to the image and text data to be processed, comprising: The first image processing result and the second image processing result are input into the contrastive learning model to obtain the processing result; Based on the first multimodal large model, the second multimodal large model, and the processing result, the image and text processing result corresponding to the image and text data to be processed is obtained.
15. The image and text data processing method according to claim 14, wherein inputting the first image and text processing result and the second image and text processing result into the contrastive learning model to obtain the processing result includes: The first image and text processing result and the second image and text processing result are input into the contrastive learning model. The encoding unit in the contrastive learning model is used to encode the first image and text processing result to obtain the first image and text processing result encoding, and the second image and text processing result is also encoded to obtain the second image and text processing result encoding. Using the contrastive learning model, the encoding of the first image-text processing result and the encoding of the second image-text processing result are processed to obtain similarity results.
16. The image and text data processing method according to claim 14, wherein obtaining the image and text processing result corresponding to the image and text data to be processed based on the first multimodal large model, the second multimodal large model, and the processing result includes: Using the first multimodal large model, based on the processing result and the first image and text processing result, a target first image and text processing result is obtained; and using the second multimodal large model, based on the processing result and the second image and text processing result, a target second image and text processing result is obtained. Select the image processing result corresponding to the image data to be processed from the first image processing result and the second image processing result.
17. The image and text data processing method according to claim 16, wherein the processing result is a similarity result; The step of obtaining a target first image-text processing result using the first multimodal large model, based on the processing result and the first image-text processing result, and obtaining a target second image-text processing result using the second multimodal large model, based on the processing result and the second image-text processing result, includes: The similarity results are input into the first multimodal large model and the first multimodal large model; Using the first multimodal large model, the image and text data to be processed is processed based on the similarity results to obtain a newly added first image and text processing result, and based on the first image and text processing result and the newly added first image and text processing result, a target first image and text processing result is obtained; Using the second multimodal large model, the text and image data to be processed is processed based on the similarity results to obtain a new second text and image processing result, and based on the second text and image processing result and the new second text and image processing result, a target second text and image processing result is obtained.
18. The image and text data processing method according to claim 10, wherein determining the image and text data to be processed includes: Receive unprocessed image and text data sent by the client, wherein the unprocessed image and text data is sent by the user through the image and text data processing interface in the client; The step of processing the first and second image-text processing results using the contrastive learning model to obtain the image-text processing result corresponding to the image-text data to be processed includes: The image and text processing results are sent to the client so that the client can display the image and text processing results to the user through the image and text data processing interface.
19. A method for training a machine learning model, comprising: The machine learning model to be trained is determined, as well as the first training data and the second training data associated with the target task, wherein the machine learning model to be trained includes a first data processing module, a second data processing module and a parameter learning module. Based on the first training data, train the first data processing module to obtain the trained first data processing module. The second training data is processed according to the first data processing module, the second data processing module, and the parameter learning module after training to obtain the processing result; Based on the processing results, the model parameters of the first data processing module and the parameter learning module after training are adjusted to obtain the trained target model, wherein the target model is used to perform the target task.
20. A data processing model training method, comprising: The data processing model to be trained is determined, as well as the first image and text training data and the second image and text training data associated with the data processing task. The data processing model to be trained includes a first multimodal large model, a second multimodal large model and a contrastive learning model, and the first multimodal large model is determined based on the second multimodal large model. Based on the first image and text training data, train the first multimodal large model to obtain the trained first multimodal large model; Based on the trained first multimodal large model, the second multimodal large model, and the contrastive learning model, the second image and text training data is processed to obtain the processing result; Based on the processing results, the model parameters of the trained first multimodal large model and the contrastive learning model are adjusted to obtain a trained data processing model, wherein the data processing model is used to perform the data processing task.
21. A data processing method, comprising: Determine the text and image data to be processed by the user; The text and image data to be processed is input into a data processing model to obtain the data processing text corresponding to the text and image data to be processed. The data processing model includes a first multimodal large model, a second multimodal large model, and a contrastive learning model. The first multimodal large model is determined based on the second multimodal large model. The first multimodal large model is used to process the text and image data to be processed to obtain a first text and image processing result. The second multimodal large model is used to process the text and image data to be processed to obtain a second text and image processing result. The contrastive learning model is used to process the first text and image processing result and the second text and image processing result to obtain the data processing text corresponding to the text and image data to be processed.
22. A computing device, comprising: Memory and processor; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions. When the computer programs / instructions are executed by the processor, they implement the steps of the data processing method according to any one of claims 1 to 9, the image and text data processing method according to any one of claims 10 to 18, the machine learning model training method according to claim 19, the data processing model training method according to claim 20, or the data processing method according to claim 21.
23. A computer-readable storage medium storing a computer program / instructions that, when executed by a processor, implement the steps of the data processing method of any one of claims 1 to 9, the image and text data processing method of any one of claims 10 to 18, the machine learning model training method of claim 19, the data processing model training method of claim 20, or the data processing method of claim 21.
24. A computer program product comprising a computer program / instructions that, when executed by a processor, implement the steps of the data processing method of any one of claims 1 to 9, the image and text data processing method of any one of claims 10 to 18, the machine learning model training method of claim 19, the data processing model training method of claim 20, or the data processing method of claim 21.
