Multi-modal model training method and device, image processing method and device and electronic equipment
Through step-by-step training method, the problem of insufficient image processing capabilities in the multimodal model of large language models is solved, and the image processing capabilities are improved while maintaining text processing capabilities, and the overall performance of the multimodal model is improved.
Patent Information
- Application Number
- CN202411046168.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-31
- Publication Date
- 2024-11-12
AI Technical Summary
The large language model lacks image processing capabilities, resulting in the limitation of the working performance of multimodal models, and it is impossible to effectively improve image processing capabilities while maintaining text processing capabilities.
Using step-by-step training method, when the weight parameters of the large language model are fixed, the image encoder and bridge components are first trained to reduce the feature spatial differences between the image encoding model and the language model, and then the entire multimodal The model is trained overall, so that the image encoder and the language model learn from each other, achieving smooth alignment of feature space.
In the multimodal model built using large language models, the powerful text processing capabilities of the language model are maintained, but also significantly improved the image processing capabilities and improved the working performance of the multimodal model.
Smart Images

Figure CN118940859A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of artificial intelligence technology, in particular to technical fields such as computer vision, deep learning, and large language models, and specifically to a multimodal model training method, image processing method, device, and electronic equipment. Background Art
[0002] In recent years, with the rapid development of artificial intelligence technology, the text processing capabilities of large language models have also made significant progress, which has led to the widespread application of large language models in the field of natural language processing. However, although large language models perform well in text processing, their image processing capabilities are relatively weak. Therefore, when using large language models to build multimodal models, it is difficult to ensure the working performance of multimodal models. Summary of the invention
[0003] The present disclosure provides a multimodal model training method, an image processing method, a device and an electronic device.
[0004] According to a first aspect of the present disclosure, a multimodal model training method is provided, comprising:
[0005] Obtain a multimodal model; wherein the multimodal model includes an image encoder, a bridge component, and a large language model connected in sequence;
[0006] Performing a training operation on the multimodal model to obtain a trained multimodal model; wherein the training operation includes:
[0007] When the weight parameters of the large language model in the multimodal model are fixed, the image encoder and the bridging component in the multimodal model are trained using the first image and text sample set to obtain a preliminarily trained multimodal model; and the preliminarily trained multimodal model is trained as a whole using the second image and text sample set to obtain a trained multimodal model.
[0008] According to a second aspect of the present disclosure, there is provided an image processing method, comprising:
[0009] Get the target image;
[0010] Get instructions;
[0011] Utilize the target model and obtain a text output result corresponding to the target image according to the instruction information; wherein the target model is a trained multimodal model obtained by the method provided by the first aspect.
[0012] According to a third aspect of the present disclosure, a multimodal model training device is provided, comprising:
[0013] A model acquisition unit, used to acquire a multimodal model; wherein the multimodal model includes an image encoder, a bridge component and a large language model connected in sequence;
[0014] A training operation unit is used to perform a training operation on the multimodal model to obtain a trained multimodal model; wherein the training operation includes:
[0015] When the weight parameters of the large language model in the multimodal model are fixed, the image encoder and the bridging component in the multimodal model are trained using the first image and text sample set to obtain a preliminarily trained multimodal model; and the preliminarily trained multimodal model is trained as a whole using the second image and text sample set to obtain a trained multimodal model.
[0016] According to a fourth aspect of the present disclosure, there is provided an image processing apparatus, comprising:
[0017] An image acquisition unit, used for acquiring a target image;
[0018] An indication acquisition unit, used for acquiring indication information;
[0019] An image processing unit is used to obtain a text output result corresponding to the target image by using the target model according to the instruction information; wherein the target model is a trained multimodal model obtained by the device provided by the third aspect.
[0020] According to a fifth aspect of the present disclosure, there is provided an electronic device, including:
[0021] at least one processor;
[0022] a memory communicatively coupled to the at least one processor;
[0023] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method provided by the embodiment of the present disclosure.
[0024] According to a sixth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the method provided according to an embodiment of the present disclosure.
[0025] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, which implements the method provided according to the embodiment of the present disclosure when executed by a processor.
[0026] The present disclosure can maintain the original powerful text processing capability of the large language model in a multimodal model constructed using a large language model, and effectively improve the image processing capability of the large language model, so that the multimodal model has excellent working performance.
[0027] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS
[0028] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.
[0029] Figure 1 A schematic structural block diagram of a multimodal model provided in an embodiment of the present disclosure;
[0030] Figure 2 A flowchart of a multimodal model training method provided in an embodiment of the present disclosure;
[0031] Figure 3 A complete flow chart of a multimodal model training method provided in an embodiment of the present disclosure;
[0032] Figure 4 A schematic diagram of a scenario of a multimodal model training method provided in an embodiment of the present disclosure;
[0033] Figure 5 A flowchart of an image processing method provided by an embodiment of the present disclosure;
[0034] Figure 6 A schematic diagram of a scene of an image processing method provided by an embodiment of the present disclosure;
[0035] Figure 7 A schematic structural block diagram of a multimodal model training device provided in an embodiment of the present disclosure;
[0036] Figure 8 A schematic structural block diagram of an image processing device provided in an embodiment of the present disclosure;
[0037] Fig. 9 A schematic structural block diagram of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION
[0038] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope of the present disclosure. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.
[0039] As described in the background technology, in recent years, with the rapid development of artificial intelligence technology, the text processing capability of large language models (LLMs) has also made significant progress, which has led to the widespread application of LLMs in the field of natural language processing. However, although LLMs perform well in text processing, their image processing capabilities are relatively weak. Therefore, when using LLMs to build multimodal models, it is difficult to ensure the working performance of multimodal models.
[0040] Specifically, please combine Figure 1 , the multimodal model constructed using LLM can also include an image encoder (ImageEncoder) and a bridge component. Among them, the image encoder is connected to the LLM through the bridge component. Here, the image encoder is pre-trained and has a general image encoding function. It is used to encode the input target image to obtain an image encoding result; the bridge component is used to perform feature conversion on the image encoding result to obtain a feature conversion result that can be understood by the LLM, that is, the bridge component is used to align the visual feature space of the image encoder to the text feature space of the LLM; the LLM is also pre-trained and has general language knowledge, world knowledge, professional knowledge in various fields, etc. It is used to process the feature conversion result according to the instruction information to obtain a text output result corresponding to the target image.
[0041] At present, after the multimodal model is constructed using LLM, the multimodal model can be trained as a whole to obtain a trained multimodal model. Specifically, after the multimodal model is constructed using LLM, the weight parameters of the image encoder, bridge component and LLM in the multimodal model can be made adjustable, and the multimodal model can be trained as a whole using a set of image and text samples to adjust the weight parameters of the image encoder, bridge component and LLM in the multimodal model to obtain a trained multimodal model.
[0042] However, the inventors have found that in the prior art, since the method of training the multimodal model as a whole is adopted without considering the large feature space difference between the image coding model and the LLM, the knowledge reserve of the LLM will be destroyed, and the LLM will lose its original powerful text processing ability, thereby damaging the working performance of the multimodal model. In other words, through the above method, it is impossible to maintain the original powerful text processing ability of the LLM and effectively improve the image processing ability of the LLM in the multimodal model constructed using the LLM.
[0043] In view of the above problems, the embodiments of the present disclosure provide a multimodal model training method, which can be applied to electronic devices. The electronic device can be a server or a terminal device. Here, the terminal device can be a workbench, a mainframe computer, a conventional computer (for example, a desktop computer, a laptop computer, a car computer, etc.), a personal digital processing or other similar computing device. Figure 2 The flowchart diagram shown illustrates a multimodal model training method provided by an embodiment of the present disclosure. It should be noted that although a logical order is shown in the flowchart diagram, in some cases, the steps shown or described may also be performed in other orders.
[0044] Step S201, obtaining a multimodal model.
[0045] The multimodal model includes an image encoder, a bridge component and an LLM which are connected in sequence.
[0046] Here, the image encoder is pre-trained and has a general image encoding function. It is used to encode the input image samples to obtain image encoding result samples; the bridging component is used to perform feature conversion on the image encoding result samples to obtain feature conversion result samples that can be understood by the LLM, that is, the bridging component is used to align the visual feature space of the image encoder to the text feature space of the LLM; the LLM is also pre-trained and has general language knowledge, world knowledge, professional knowledge in various fields, etc. It is used to process the feature conversion result samples according to the instruction information samples to obtain text output result samples corresponding to the image samples.
[0047] In the disclosed embodiment, the bridging component may be a multilayer perceptron (MLP), a vision resampler, etc.; the LLM may be an autoregressive generative model of a Transformer architecture.
[0048] Step S202: Perform a training operation on the multimodal model to obtain a trained multimodal model.
[0049] The training operation includes:
[0050] When the weight parameters of the LLM in the multimodal model are fixed, the image encoder and the bridging component in the multimodal model are trained using the first image and text sample set to obtain a preliminarily trained multimodal model; and the preliminarily trained multimodal model is trained as a whole using the second image and text sample set to obtain a trained multimodal model.
[0051] Among them, the fixed weight parameters of LLM can be understood as: making the weight parameters of LLM in an unadjustable state; the first image and text sample set can include multiple first image and text sample groups, and each first image and text sample group in the multiple first image and text sample groups can include a first image sample, and a first text sample related to the first image sample; the second image and text sample set can include multiple second image and text sample groups, and each second image and text sample group in the multiple second image and text sample groups can include a second image sample, and a second text sample related to the second image sample.
[0052] In addition, in the embodiments of the present disclosure, "when the weight parameters of the LLM in the multimodal model are fixed, the image encoder and the bridging component in the multimodal model are trained using the first image and text sample set to obtain a multimodal model that has been preliminarily trained", can be understood as: making the weight parameters of the image encoder and the bridging component in the multimodal model adjustable, and at the same time, making the weight parameters of the LLM in the multimodal model non-adjustable, and using the first image and text sample set to train the image encoder and the bridging component in the multimodal model to adjust the weight parameters of the image encoder and the bridging component in the multimodal model to obtain a multimodal model that has been preliminarily trained. “Using the second set of image and text samples to perform overall training on the preliminarily trained multimodal model to obtain a trained multimodal model” can be understood as: making the weight parameters of the image encoder, bridging component and LLM in the preliminarily trained multimodal model in an adjustable state, and using the second set of image and text samples to perform overall training on the preliminarily trained multimodal model (that is, training the image encoder, bridging component and LLM in the preliminarily trained multimodal model) to adjust the weight parameters of the image encoder, bridging component and LLM in the preliminarily trained multimodal model to obtain a trained multimodal model.
[0053] The multimodal model training method provided by the embodiment of the present disclosure can be used to train the multimodal model in a step-by-step training manner. More specifically, first, when the weight parameters of the LLM in the multimodal model are fixed, the image encoder and the bridging component in the multimodal model are trained to obtain a preliminarily trained multimodal model, so as to reduce the feature space difference between the image coding model and the LLM with the help of the bridging component in the preliminarily trained multimodal model, and at the same time, avoid the knowledge reserve of the LLM from being destroyed, so that the LLM loses its original powerful text processing ability; thereafter, the preliminarily trained multimodal model is trained as a whole (that is, the image encoder, the bridging component and the LLM in the preliminarily trained multimodal model are trained), so that the image encoder and the LLM in the preliminarily trained multimodal model can learn each other's knowledge reserve, so that the image coding model and the LLM can achieve a smooth feature space alignment. In this way, in the multimodal model constructed using LLM, the original powerful text processing ability of LLM can be maintained, and the image processing ability of LLM can be effectively improved, so that the multimodal model has excellent working performance.
[0054] In addition, as mentioned above, in the embodiments of the present disclosure, the first image and text sample set may include multiple first image and text sample groups, and the multiple first image and text sample groups may correspond one-to-one to multiple image processing tasks; and / or, the second image and text sample set may include multiple second image and text sample groups, and the multiple second image and text sample groups may correspond one-to-one to multiple image processing tasks.
[0055] The plurality of image processing tasks include at least one of an image description task, a text extraction task and an object detection task.
[0056] That is to say, in the embodiment of the present disclosure, for each first image and text sample group among multiple first image and text sample groups, when the first image and text sample group corresponds to an image description task, the first text sample in the first image and text sample group may be a content description text for the first image sample in the first image and text sample group; when the first image and text sample group corresponds to a text extraction task, the first text sample in the first image and text sample group may be a text extraction result for the first image sample in the first image and text sample group; when the first image and text sample group corresponds to a target detection task, the first text sample in the first image and text sample group may be a target detection result for the first image sample in the first image and text sample group. Similarly, in the embodiment of the present disclosure, for each second image and text sample group among the multiple second image and text sample groups, when the second image and text sample group corresponds to an image description task, the second text sample in the second image and text sample group may be a content description text for the second image sample in the second image and text sample group; when the second image and text sample group corresponds to a text extraction task, the second text sample in the second image and text sample group may be a text extraction result for the second image sample in the second image and text sample group; when the second image and text sample group corresponds to a target detection task, the second text sample in the second image and text sample group may be a target detection result for the second image sample in the second image and text sample group.
[0057] In this way, in the embodiment of the present disclosure, the diversification of training data (including the first image and text sample set and the second image and text sample set) can be ensured, so that the trained multimodal model can be applied to at least one of the image description task, text extraction task and target detection task, thereby expanding the scope of application of the trained multimodal model.
[0058] Further, in the embodiment of the present disclosure, in step S202, “when the weight parameters of the LLM in the multimodal model are fixed, using the first image and text sample set, training the image encoder and the bridge component in the multimodal model to obtain a preliminarily trained multimodal model” may include:
[0059] When the weight parameters of the LLM in the multimodal model are fixed, the first image sample in the first target sample group is input into the image encoder in the multimodal model, so as to encode the first image sample in the first target image-text sample group using the image encoder in the multimodal model to obtain a first image encoding result sample; wherein the first target sample group is any first image-text sample group among the plurality of first image-text sample groups;
[0060] Using a bridge component in a multimodal model, feature conversion is performed on the first image encoding result sample to obtain a first feature conversion result sample;
[0061] Using the LLM in the multimodal model, according to the first indication information sample, the first feature conversion result sample is processed to obtain a first text output result sample corresponding to the first image sample in the first target image-text sample group; wherein the first indication information sample is used to instruct the LLM in the multimodal model to perform at least one of the multiple image processing tasks;
[0062] Based on the first text sample and the first text output result sample in the first target sample group, the image encoder and the bridge component in the multimodal model are trained to obtain a preliminarily trained multimodal model.
[0063] In one example, an autoregressive loss function can be used to calculate a first loss value between a first text sample in a first target sample group and a first text output result sample, and based on the first loss value, the image encoder and the bridging component in the multimodal model are trained to adjust the weight parameters of the image encoder and the bridging component in the multimodal model to obtain a preliminarily trained multimodal model.
[0064] Similarly, in the embodiment of the present disclosure, "using the second image and text sample set to perform overall training on the preliminarily trained multimodal model to obtain a trained multimodal model" in step S202 may include:
[0065] Inputting the second image samples in the second target sample group into the image encoder in the preliminarily trained multimodal model, so as to encode the second image samples in the second target image-text sample group using the image encoder in the preliminarily trained multimodal model, and obtain a second image encoding result sample; wherein the second target sample group is any second image-text sample group among the plurality of second image-text sample groups;
[0066] Using a bridge component in the preliminarily trained multimodal model, perform feature conversion on the second image encoding result sample to obtain a second feature conversion result sample;
[0067] Using the LLM in the preliminarily trained multimodal model, according to the second indication information sample, the second feature conversion result sample is processed to obtain a second text output result sample corresponding to the second image sample in the second target image-text sample group; wherein the second indication information sample is used to instruct the LLM in the preliminarily trained multimodal model to perform at least one of the multiple image processing tasks;
[0068] Based on the second text samples and the second text output result samples in the second target sample group, the preliminarily trained multimodal model is trained as a whole to obtain a trained multimodal model.
[0069] In one example, an autoregressive loss function can be used to calculate a second loss value between a second text sample in a second target sample group and a second text output result sample, and based on the second loss value, the preliminarily trained multimodal model is trained as a whole to adjust the weight parameters of the image encoder, bridging component and LLM in the preliminarily trained multimodal model to obtain a trained multimodal model.
[0070] In some optional implementations, “performing a training operation on the multimodal model to obtain a trained multimodal model” may include:
[0071] The training operation for the multimodal model is executed n times in a loop to obtain a trained multimodal model.
[0072] Wherein, n≥2 and is a positive integer.
[0073] In addition, it should be noted that in the embodiment of the present disclosure, the data quality of the first image and text sample set used when performing the i+1th training operation for the multimodal model is better than the data quality of the first image and text sample set used when performing the i-th training operation for the multimodal model.
[0074] Wherein, 1≤i≤n-1, and i is a positive integer.
[0075] As mentioned above, in the embodiment of the present disclosure, the first image-text sample set may include multiple first image-text sample groups, and each of the multiple first image-text sample groups may include a first image sample, and a first text sample related to the first image sample. Based on this, in one example, for each first image-text sample group in the multiple first image-text sample groups, the data quality of the first image-text sample group can be characterized by the data quality evaluation result of the first text sample in the first image-text sample group, and the data quality evaluation result of the first text sample in the first image-text sample group can be determined by comprehensive evaluation from multiple evaluation dimensions. Among them, the multiple evaluation dimensions may include at least one of fluency, accuracy, task completion, and content relevance.
[0076] In one example, the first text sample included in each of the plurality of first text and image sample groups can be used as a target text sample, and the data quality evaluation result of the target text sample can be obtained by:
[0077] (1) Obtaining data quality assessment scores of the target text sample from multiple assessment dimensions respectively to obtain multiple data quality assessment scores.
[0078] As mentioned above, in the embodiment of the present disclosure, the multiple evaluation dimensions may include at least one of the following:
[0079] a. Fluency;
[0080] b. Accuracy;
[0081] c. Task completion;
[0082] d. Content relevance.
[0083] Among them, fluency is used to characterize the readability of the target text sample; accuracy is used to characterize whether there are factual errors in the target text sample; task completion is used to characterize the degree of conformity between the target text sample and the task requirements of the image processing task when the first image sample (hereinafter referred to as the target image sample for the convenience of description) belonging to the same first image and text sample group as the target text sample is used as the processing object to perform a certain image processing task, and the obtained task processing result (that is, the content description text, text extraction result or target detection result) is the target text sample; content relevance is used to characterize whether the target text sample is the accurate text content in the target image sample when the image processing task performed with the target image sample as the processing object is a text extraction task, and the obtained text extraction result is the target text sample.
[0084] In a specific example, for the three evaluation dimensions of fluency, accuracy and task completion, the Chat Generative Pre-trained Transformer (ChatGPT) model can be used to obtain the data quality evaluation score of the target text sample; for the evaluation dimension of content relevance, the Optical Character Recognition (OCR) model can be used to obtain the data quality evaluation score of the target text sample.
[0085] (2) Obtain multiple score weights corresponding to the multiple data quality assessment scores.
[0086] Among them, multiple score weights corresponding to multiple data quality assessment scores can be set according to actual application requirements, and the embodiments of the present disclosure do not limit this.
[0087] (3) Based on multiple data quality assessment scores and multiple score weights, a data quality assessment result of the target text sample is obtained.
[0088] In a specific example, for each data quality assessment score among multiple data quality assessment scores, the product of the data quality assessment score and the score weight corresponding to the data quality assessment score can be calculated as the value to be superimposed to obtain multiple scores to be superimposed, and the sum of the multiple scores to be superimposed is calculated as the data quality assessment result of the target text sample.
[0089] It should be noted that, in the embodiment of the present disclosure, for each first image and text sample group in multiple first image and text sample groups, if the data quality evaluation result of the first text sample in the first image and text sample group is greater than or equal to the preset first high-quality threshold, the first image and text sample group in the first image and text sample group can be confirmed as high-quality data; if the data quality evaluation result of the first text sample in the first image and text sample group is less than the first preset high-quality threshold, the first image and text sample group in the first image and text sample group can be confirmed as low-quality data. The first preset high-quality threshold can be set according to actual application requirements, and the embodiment of the present disclosure does not limit this.
[0090] It should also be noted that, in the embodiments of the present disclosure, “the data quality of the first image and text sample set used when performing the i+1th training operation on the multimodal model is better than the data quality of the first image and text sample set used when performing the i-th training operation on the multimodal model” can be understood as: a first number of first image and text sample groups in the first image and text sample set used when performing the i+1th training operation on the multimodal model are high-quality data, and a second number of first image and text sample groups in the first image and text sample set used when performing the i-th training operation on the multimodal model are high-quality data, and the first number is greater than the second number; or, a first proportion of first image and text sample groups in the first image and text sample set used when performing the i+1th training operation on the multimodal model are high-quality data, and a second proportion of first image and text sample groups in the first image and text sample set used when performing the i-th training operation on the multimodal model are high-quality data, and the first proportion is greater than the second proportion.
[0091] Through the above method, in the disclosed embodiment, the trained multimodal model can be obtained by cyclically executing the training operation for the multimodal model n times, and the data quality of the first image and text sample set used when executing the i+1th training operation for the multimodal model is better than the data quality of the first image and text sample set used when executing the i-th training operation for the multimodal model. In this way, on the one hand, the alternating alignment of the feature space of the image encoder and the LLM in the multimodal model can be achieved through the progressive division of the data quality of the first image and text sample set, and the alignment process is progressive (that is, first roughly aligning, and then transitioning to precise alignment), so it can be ensured that the feature space of the image encoder and the LLM in the multimodal model can achieve the final precise alignment; on the other hand, the multimodal model that has been preliminarily trained can be trained as a whole for many times, so that the image encoder and LLM in the preliminarily trained multimodal model can learn from each other's knowledge reserves, so that the image encoding model and the LLM can achieve a smooth feature space alignment, thereby further improving the working performance of the multimodal model.
[0092] In some optional implementations, the multimodal model training method may further include:
[0093] Obtaining a first candidate image sample;
[0094] Using the data production model, a first candidate text sample corresponding to the first candidate image sample is obtained; wherein the data production model is a trained multimodal model obtained when performing the i-th training operation on the multimodal model;
[0095] Based on the first candidate image sample and the first candidate text sample, a first image-text sample set used for performing the (i+1)th training operation on the multimodal model is constructed.
[0096] The first candidate image sample may be a network image.
[0097] In one example, the first candidate text sample corresponding to the first candidate image sample may be obtained in the following manner:
[0098] Inputting the first candidate image sample into the image encoder in the data production model, so as to encode the first candidate image sample using the image encoder in the data production model to obtain a first candidate image encoding result sample;
[0099] Using the bridge component in the data production model, perform feature conversion on the first candidate image encoding result sample to obtain a first candidate feature conversion result sample;
[0100] Using the LLM in the data production model, the first candidate feature conversion result sample is processed according to the candidate indication information sample to obtain a first candidate text sample corresponding to the first candidate image sample; wherein the candidate indication information sample is used to instruct the LLM in the data production model to perform at least one of multiple image processing tasks.
[0101] In the disclosed embodiment, after obtaining the first candidate image sample and the first candidate text sample, the first candidate image sample can be used as the first image sample, and at the same time, the first candidate text sample can be used as the first text sample, and the first image sample and the first text sample can be used to form a first image-text sample group to construct a first image-text sample set used for performing the i+1th training operation on the multimodal model.
[0102] Through the above method, in the embodiment of the present disclosure, the first candidate image sample can be obtained, and the data production model can be used to obtain the first candidate text sample corresponding to the first candidate image sample, and then based on the first candidate image sample and the first candidate text sample, the first image and text sample set used when performing the i+1th training operation on the multimodal model is constructed, and the data production model is the trained multimodal model obtained when performing the i-th training operation on the multimodal model. That is to say, in the embodiment of the present disclosure, the first image and text sample set used when performing the i+1th training operation on the multimodal model can be generated using the trained multimodal model obtained when performing the i-th training operation on the multimodal model. In this way, the difficulty of obtaining training data (for example, the first image and text sample set used when performing the i+1th training operation on the multimodal model) can be reduced, thereby improving the training efficiency of the multimodal model.
[0103] In some optional implementations, “constructing a first image-text sample set used for performing the (i+1)th training operation on the multimodal model based on the first candidate image sample and the first candidate text sample” may include:
[0104] Obtaining a data quality assessment result of the first candidate text sample;
[0105] Obtaining a data quality assessment threshold corresponding to the i+1th training operation for the multimodal model; wherein the data quality assessment threshold is positively correlated with the value of i+1;
[0106] When the data quality assessment result is greater than or equal to the data quality assessment threshold, a first image and text sample set used for performing the (i+1)th training operation on the multimodal model is constructed based on the first candidate image sample and the first candidate text sample.
[0107] In one example, the data quality evaluation result of the first candidate text sample can be obtained in the following manner:
[0108] (1) Obtaining data quality assessment scores of the first candidate text sample from multiple assessment dimensions respectively to obtain multiple data quality assessment scores.
[0109] As mentioned above, in the embodiment of the present disclosure, the multiple evaluation dimensions may include at least one of the following:
[0110] a. Fluency;
[0111] b. Accuracy;
[0112] c. Task completion;
[0113] d. Content relevance.
[0114] Among them, fluency is used to characterize the readability of the first candidate text sample; accuracy is used to characterize whether there are factual errors in the first candidate text sample; task completion is used to characterize the degree of conformity between the first candidate text sample and the task requirements of the image processing task when a certain image processing task is performed with the first candidate image sample as the processing object and the obtained task processing result (that is, content description text, text extraction result or target detection result) is the first candidate text sample; content relevance is used to characterize whether the first candidate text sample is the accurate text content in the first candidate image sample when the image processing task performed with the first candidate image sample as the processing object is a text extraction task and the obtained text extraction result is the first candidate text sample.
[0115] In a specific example, for the three evaluation dimensions of fluency, accuracy and task completion, ChatGPT can be used to obtain the data quality evaluation score of the first candidate text sample; for the evaluation dimension of content relevance, the OCR model can be used to obtain the data quality evaluation score of the first candidate text sample.
[0116] (2) Obtain multiple score weights corresponding to the multiple data quality assessment scores.
[0117] Among them, multiple score weights corresponding to multiple data quality assessment scores can be set according to actual application requirements, and the embodiments of the present disclosure do not limit this.
[0118] (3) Based on the multiple data quality assessment scores and the multiple score weights, a data quality assessment result of the first candidate text sample is obtained.
[0119] In a specific example, for each data quality assessment score among multiple data quality assessment scores, the product of the data quality assessment score and the score weight corresponding to the data quality assessment score can be calculated as the value to be superimposed to obtain multiple scores to be superimposed, and the sum of the multiple scores to be superimposed is calculated as the data quality assessment result of the first candidate text sample.
[0120] That is to say, in the embodiment of the present disclosure, the data quality assessment scores of the first candidate text sample can be obtained from multiple assessment dimensions respectively to obtain multiple data quality assessment scores, and multiple score weights corresponding to the multiple data quality assessment scores can be obtained, and the data quality assessment result of the first candidate text sample can be obtained based on the multiple data quality assessment scores and the multiple score weights. In this way, the comprehensiveness of the data quality assessment result can be ensured, thereby improving the reliability of the multimodal model training method.
[0121] In addition, in the embodiment of the present disclosure, "the data quality assessment threshold is positively correlated with the value of i+1" can be understood as: the larger the value of i+1, the larger the data quality assessment threshold; the smaller the value of i+1, the smaller the data quality assessment threshold.
[0122] In an embodiment of the present disclosure, after obtaining the data quality assessment result of the first candidate text sample and the data quality assessment threshold corresponding to the i+1th training operation for the multimodal model, the first candidate image sample can be used as the first image sample if the data quality assessment result is greater than or equal to the data quality assessment threshold. At the same time, the first candidate text sample can be used as the first text sample, and the first image sample and the first text sample can be used to form a first image-text sample group to construct a first image-text sample set used when performing the i+1th training operation for the multimodal model. If the data quality assessment result is less than the data quality assessment threshold, the first candidate image sample and the first candidate text sample are discarded.
[0123] Through the above method, in the embodiment of the present disclosure, the data quality evaluation result of the first candidate text sample can be obtained, and the data quality evaluation threshold corresponding to the i+1th training operation for the multimodal model can be obtained (the data quality evaluation threshold is positively correlated with the value of i+1), and when the data quality evaluation result is greater than or equal to the data quality evaluation threshold, the first image and text sample set used when performing the i+1th training operation for the multimodal model is constructed based on the first candidate image sample and the first candidate text sample. Since the data quality evaluation threshold is positively correlated with the value of i+1, it can be ensured that the data quality of the first image and text sample set used when performing the i+1th training operation for the multimodal model is better than the data quality of the first image and text sample set used when performing the i-th training operation for the multimodal model, that is, it can be ensured that the alternating alignment of the feature space of the image encoder and the LLM in the multimodal model is achieved through the progressive division of the data quality of the first image and text sample set.
[0124] In some optional implementations, the multimodal model training method may further include:
[0125] Obtaining a second candidate image sample;
[0126] In response to the labeling operation on the second candidate image sample, a second candidate text sample corresponding to the second candidate image sample is obtained;
[0127] Based on the second candidate image samples and the second candidate text samples, a second image-text sample set used for performing a training operation on the multimodal model is constructed.
[0128] The second candidate image sample may be a network image; the annotation operation may be performed by a professional data annotator, thus ensuring that the second candidate text sample has a higher data quality assessment result.
[0129] In one example, after obtaining the second candidate image sample and the second candidate text sample, the second candidate image sample can be used as the second image sample, and at the same time, the second candidate text sample can be used as the second text sample, and the second image sample and the second text sample can be used to form a second image-text sample group to construct a second image-text sample set used to perform training operations on the multimodal model.
[0130] In another example, after obtaining the second candidate image sample and the second candidate text sample, a data quality evaluation result of the second candidate text sample can be further obtained, and when the data quality evaluation result of the second candidate text sample is greater than or equal to a second preset high quality threshold, the second candidate image sample is used as the second image sample, and at the same time, the second candidate text sample is used as the second text sample, and the second image sample and the second text sample are used to form a second image-text sample group to construct a second image-text sample set used when performing training operations on the multimodal model; when the data quality evaluation result of the second candidate text sample is less than the second preset high quality threshold, the second candidate image sample and the second candidate text sample are discarded.
[0131] Among them, the data quality assessment result of the second candidate text sample can be determined by comprehensive assessment from multiple assessment dimensions. For details, please refer to the method adopted when obtaining the data quality assessment result of the first candidate text sample mentioned above, which will not be repeated here; the second preset high-quality threshold can be set according to actual application requirements, and the embodiment of the present disclosure does not limit this.
[0132] Through the above manner, in the embodiment of the present disclosure, a second candidate image sample can be obtained, and in response to the labeling operation on the second candidate image sample, a second candidate text sample corresponding to the second candidate image sample can be obtained, and then based on the second candidate image sample and the second candidate text sample, a second image and text sample set used for performing training operations on the multimodal model is constructed, thereby ensuring that the second image and text sample set belongs to a high-quality data set, so as to ensure that when the multimodal model that has been preliminarily trained is trained as a whole, so that the image encoder and LLM in the preliminarily trained multimodal model can learn from each other's knowledge reserves, the feature space of the image encoder and LLM in the multimodal model can be smoothly aligned, thereby further improving the working performance of the multimodal model.
[0133] The following will be combined Figure 3 , the integrity process of a multimodal model training method provided by an embodiment of the present disclosure is described.
[0134] Step S301, obtaining a multimodal model.
[0135] The multimodal model includes an image encoder, a bridge component and an LLM which are connected in sequence.
[0136] Here, the image encoder is pre-trained and has a general image encoding function. It is used to encode the input image samples to obtain image encoding result samples; the bridging component is used to perform feature conversion on the image encoding result samples to obtain feature conversion result samples that can be understood by the LLM, that is, the bridging component is used to align the visual feature space of the image encoder to the text feature space of the LLM; the LLM is also pre-trained and has general language knowledge, world knowledge, professional knowledge in various fields, etc. It is used to process the feature conversion result samples according to the instruction information samples to obtain text output result samples corresponding to the image samples.
[0137] In the disclosed embodiment, the bridging component may be an MLP, a visual resampler, etc.; the LLM may be an autoregressive generative model of a Transformer architecture.
[0138] Step S302: perform the i-th training operation on the multimodal model to obtain a trained multimodal model.
[0139] Among them, 1≤i≤n-1, and i is a positive integer; n≥2, and is a positive integer.
[0140] The training operation includes:
[0141] Step S302-1, when the weight parameters of the LLM in the multimodal model are fixed, the image encoder and the bridge component in the multimodal model are trained using the first image and text sample set to obtain a preliminarily trained multimodal model.
[0142] The fact that the weight parameters of the LLM are fixed can be understood as: making the weight parameters of the LLM in an unadjustable state; the first image-text sample set can include multiple first image-text sample groups (for example, hundreds of millions of first image-text sample groups), and each of the multiple first image-text sample groups can include a first image sample and a first text sample related to the first image sample. Here, the multiple first image-text sample groups can correspond one-to-one to multiple image processing tasks, and the multiple image processing tasks include at least one of an image description task, a text extraction task, and an object detection task.
[0143] That is to say, in the embodiment of the present disclosure, for each first image and text sample group among multiple first image and text sample groups, when the first image and text sample group corresponds to an image description task, the first text sample in the first image and text sample group may be a content description text for the first image sample in the first image and text sample group; when the first image and text sample group corresponds to a text extraction task, the first text sample in the first image and text sample group may be a text extraction result for the first image sample in the first image and text sample group; when the first image and text sample group corresponds to a target detection task, the first text sample in the first image and text sample group may be a target detection result for the first image sample in the first image and text sample group.
[0144] In addition, in the embodiments of the present disclosure, "when the weight parameters of the LLM in the multimodal model are fixed, the image encoder and the bridging component in the multimodal model are trained using the first image and text sample set to obtain a multimodal model that has been preliminarily trained", can be understood as: making the weight parameters of the image encoder and the bridging component in the multimodal model adjustable, and at the same time, making the weight parameters of the LLM in the multimodal model non-adjustable, and using the first image and text sample set to train the image encoder and the bridging component in the multimodal model to adjust the weight parameters of the image encoder and the bridging component in the multimodal model to obtain a multimodal model that has been preliminarily trained.
[0145] Further, in the embodiment of the present disclosure, step S302-1, that is, "when the weight parameters of the LLM in the multimodal model are fixed, using the first image and text sample set, the image encoder and the bridge component in the multimodal model are trained to obtain a preliminarily trained multimodal model" may include:
[0146] When the weight parameters of the LLM in the multimodal model are fixed, the first image sample in the first target sample group is input into the image encoder in the multimodal model, so as to encode the first image sample in the first target image-text sample group using the image encoder in the multimodal model to obtain a first image encoding result sample; wherein the first target sample group is any first image-text sample group among the plurality of first image-text sample groups;
[0147] Using a bridge component in a multimodal model, feature conversion is performed on the first image encoding result sample to obtain a first feature conversion result sample;
[0148] Using the LLM in the multimodal model, according to the first indication information sample, the first feature conversion result sample is processed to obtain a first text output result sample corresponding to the first image sample in the first target image-text sample group; wherein the first indication information sample is used to instruct the LLM in the multimodal model to perform at least one of the multiple image processing tasks;
[0149] Based on the first text sample and the first text output result sample in the first target sample group, the image encoder and the bridge component in the multimodal model are trained to obtain a preliminarily trained multimodal model.
[0150] In one example, an autoregressive loss function can be used to calculate a first loss value between a first text sample in a first target sample group and a first text output result sample, and based on the first loss value, the image encoder and the bridging component in the multimodal model are trained to adjust the weight parameters of the image encoder and the bridging component in the multimodal model to obtain a preliminarily trained multimodal model.
[0151] Step S302-2: Use the second image and text sample set to perform overall training on the preliminarily trained multimodal model to obtain a trained multimodal model.
[0152] The second image-text sample set may include multiple second image-text sample groups (e.g., tens of thousands of second image-text sample groups), and each of the multiple second image-text sample groups may include a second image sample and a second text sample related to the second image sample. Here, the multiple second image-text sample groups may correspond one-to-one to the multiple image processing tasks, and the multiple second image-text sample groups may correspond one-to-one to the multiple image processing tasks.
[0153] That is to say, in the embodiment of the present disclosure, for each second image and text sample group among the multiple second image and text sample groups, when the second image and text sample group corresponds to an image description task, the second text sample in the second image and text sample group may be a content description text for the second image sample in the second image and text sample group; when the second image and text sample group corresponds to a text extraction task, the second text sample in the second image and text sample group may be a text extraction result for the second image sample in the second image and text sample group; when the second image and text sample group corresponds to a target detection task, the second text sample in the second image and text sample group may be a target detection result for the second image sample in the second image and text sample group.
[0154] In one example, the second image and text sample set may be obtained in the following manner to ensure that the second image and text sample set belongs to a high-quality data set:
[0155] Obtaining a second candidate image sample;
[0156] In response to the labeling operation on the second candidate image sample, a second candidate text sample corresponding to the second candidate image sample is obtained;
[0157] Based on the second candidate image samples and the second candidate text samples, a second image-text sample set used for performing a training operation on the multimodal model is constructed.
[0158] The second candidate image sample may be a network image; the annotation operation may be performed by a professional data annotator, thus ensuring that the second candidate text sample has a higher data quality assessment result.
[0159] In a specific example, after obtaining the second candidate image sample and the second candidate text sample, the second candidate image sample can be used as the second image sample, and at the same time, the second candidate text sample can be used as the second text sample, and the second image sample and the second text sample can be used to form a second image-text sample group to construct a second image-text sample set used to perform training operations on the multimodal model.
[0160] In another specific example, after obtaining the second candidate image sample and the second candidate text sample, a data quality evaluation result of the second candidate text sample can be further obtained, and when the data quality evaluation result of the second candidate text sample is greater than or equal to a second preset high quality threshold, the second candidate image sample is used as the second image sample, and at the same time, the second candidate text sample is used as the second text sample, and the second image sample and the second text sample are used to form a second image-text sample group to construct a second image-text sample set used when performing training operations on the multimodal model; when the data quality evaluation result of the second candidate text sample is less than the second preset high quality threshold, the second candidate image sample and the second candidate text sample are discarded.
[0161] Among them, the data quality assessment result of the second candidate text sample can be determined by comprehensive assessment from multiple assessment dimensions. For details, please refer to the method adopted when obtaining the data quality assessment result of the first candidate text sample mentioned above, which will not be repeated here; the second preset high-quality threshold can be set according to actual application requirements, and the embodiment of the present disclosure does not limit this.
[0162] In addition, in the embodiments of the present disclosure, "using the second image and text sample set to perform overall training on the preliminarily trained multimodal model to obtain a trained multimodal model" can be understood as: making the weight parameters of the image encoder, bridging component and LLM in the preliminarily trained multimodal model in an adjustable state, and using the second image and text sample set to perform overall training on the preliminarily trained multimodal model (that is, training the image encoder, bridging component and LLM in the preliminarily trained multimodal model) to adjust the weight parameters of the image encoder, bridging component and LLM in the preliminarily trained multimodal model to obtain a trained multimodal model.
[0163] Further, in the embodiment of the present disclosure, step S302-2, that is, "using the second image and text sample set to perform overall training on the preliminarily trained multimodal model to obtain a trained multimodal model" may include:
[0164] Inputting the second image samples in the second target sample group into the image encoder in the preliminarily trained multimodal model, so as to encode the second image samples in the second target image-text sample group using the image encoder in the preliminarily trained multimodal model, and obtain a second image encoding result sample; wherein the second target sample group is any second image-text sample group among the plurality of second image-text sample groups;
[0165] Using a bridge component in the preliminarily trained multimodal model, perform feature conversion on the second image encoding result sample to obtain a second feature conversion result sample;
[0166] Using the LLM in the preliminarily trained multimodal model, according to the second indication information sample, the second feature conversion result sample is processed to obtain a second text output result sample corresponding to the second image sample in the second target image-text sample group; wherein the second indication information sample is used to instruct the LLM in the preliminarily trained multimodal model to perform at least one of the multiple image processing tasks;
[0167] Based on the second text samples and the second text output result samples in the second target sample group, the preliminarily trained multimodal model is trained as a whole to obtain a trained multimodal model.
[0168] In one example, an autoregressive loss function can be used to calculate a second loss value between a second text sample in a second target sample group and a second text output result sample, and based on the second loss value, the preliminarily trained multimodal model is trained as a whole to adjust the weight parameters of the image encoder, bridging component and LLM in the preliminarily trained multimodal model to obtain a trained multimodal model.
[0169] Step S303, determining whether the trained multimodal model meets a preset convergence condition.
[0170] In one example, whether the trained multimodal model meets the preset convergence condition can be determined in the following manner:
[0171] Inputting the test image into the image encoder in the trained multimodal model, so as to encode the test image using the image encoder in the trained multimodal model to obtain a test image encoding result;
[0172] Using the bridge component in the trained multimodal model, feature conversion is performed on the result test image encoding result to obtain the test image feature conversion result;
[0173] Using the LLM in the trained multimodal model, the test image feature conversion result is processed according to the test instruction information to obtain a test output text corresponding to the test image; wherein the test instruction information is used to instruct the LLM in the trained multimodal model to perform at least one of multiple image processing tasks.
[0174] After obtaining the test output text, the autoregressive loss function can be used to calculate the third loss value between the test output text and the preset text label, and when the third loss value meets the preset loss requirement, it is determined that the trained multimodal model meets the preset convergence condition; or, when the third loss value does not meet the preset loss requirement, it is determined that the trained multimodal model does not meet the preset convergence condition; wherein the preset text label can be a target label obtained by a professional data annotator who performs an annotation operation on the test image according to the test instruction information; the preset loss requirement can be set according to the actual application requirements, and the embodiments of the present disclosure are not limited to this.
[0175] When the trained multimodal model does not meet the preset convergence conditions, a "+1" operation can be performed on i and return to step S302, that is, after i+1, return to step S302 to perform the i+1th training operation on the multimodal model to obtain a trained multimodal model; when the trained multimodal model meets the preset convergence conditions, enter step S304.
[0176] Step S304, obtaining the target model.
[0177] Among them, the target model is a trained multimodal model.
[0178] In addition, it should be noted that, in the embodiment of the present disclosure, the data quality of the first image and text sample set used when performing the i+1th training operation for the multimodal model is better than the data quality of the first image and text sample set used when performing the i-th training operation for the multimodal model. Based on this, in the embodiment of the present disclosure, the multimodal model training method may also include:
[0179] Obtaining a first candidate image sample;
[0180] Using the data production model, a first candidate text sample corresponding to the first candidate image sample is obtained; wherein the data production model is a trained multimodal model obtained when performing the i-th training operation on the multimodal model;
[0181] Obtaining a data quality assessment result of the first candidate text sample;
[0182] Obtaining a data quality assessment threshold corresponding to the i+1th training operation for the multimodal model; wherein the data quality assessment threshold is positively correlated with the value of i+1;
[0183] When the data quality assessment result is greater than or equal to the data quality assessment threshold, a first image and text sample set used for performing the (i+1)th training operation on the multimodal model is constructed based on the first candidate image sample and the first candidate text sample.
[0184] The first candidate image sample may be a network image.
[0185] In one example, the first candidate text sample corresponding to the first candidate image sample may be obtained in the following manner:
[0186] Inputting the first candidate image sample into the image encoder in the data production model, so as to encode the first candidate image sample using the image encoder in the data production model to obtain a first candidate image encoding result sample;
[0187] Using the bridge component in the data production model, perform feature conversion on the first candidate image encoding result sample to obtain a first candidate feature conversion result sample;
[0188] Using the LLM in the data production model, the first candidate feature conversion result sample is processed according to the candidate indication information sample to obtain a first candidate text sample corresponding to the first candidate image sample; wherein the candidate indication information sample is used to instruct the LLM in the data production model to perform at least one of multiple image processing tasks.
[0189] In one example, the data quality evaluation result of the first candidate text sample can be obtained in the following manner:
[0190] (1) Obtaining data quality assessment scores of the first candidate text sample from multiple assessment dimensions respectively to obtain multiple data quality assessment scores.
[0191] As mentioned above, in the embodiment of the present disclosure, the multiple evaluation dimensions may include at least one of the following:
[0192] a. Fluency;
[0193] b. Accuracy;
[0194] c. Task completion;
[0195] d. Content relevance.
[0196] Among them, fluency is used to characterize the readability of the first candidate text sample; accuracy is used to characterize whether there are factual errors in the first candidate text sample; task completion is used to characterize the degree of conformity between the first candidate text sample and the task requirements of the image processing task when a certain image processing task is performed with the first candidate image sample as the processing object and the obtained task processing result (that is, content description text, text extraction result or target detection result) is the first candidate text sample; content relevance is used to characterize whether the first candidate text sample is the accurate text content in the first candidate image sample when the image processing task performed with the first candidate image sample as the processing object is a text extraction task and the obtained text extraction result is the first candidate text sample.
[0197] In a specific example, for the three evaluation dimensions of fluency, accuracy and task completion, ChatGPT can be used to obtain the data quality evaluation score of the first candidate text sample; for the evaluation dimension of content relevance, the OCR model can be used to obtain the data quality evaluation score of the first candidate text sample.
[0198] (2) Obtain multiple score weights corresponding to the multiple data quality assessment scores.
[0199] Among them, multiple score weights corresponding to multiple data quality assessment scores can be set according to actual application requirements, and the embodiments of the present disclosure do not limit this.
[0200] (3) Based on the multiple data quality assessment scores and the multiple score weights, a data quality assessment result of the first candidate text sample is obtained.
[0201] In a specific example, for each data quality assessment score among multiple data quality assessment scores, the product of the data quality assessment score and the score weight corresponding to the data quality assessment score can be calculated as the value to be superimposed to obtain multiple scores to be superimposed, and the sum of the multiple scores to be superimposed is calculated as the data quality assessment result of the first candidate text sample.
[0202] In addition, in the embodiment of the present disclosure, "the data quality assessment threshold is positively correlated with the value of i+1" can be understood as: the larger the value of i+1, the larger the data quality assessment threshold; the smaller the value of i+1, the smaller the data quality assessment threshold.
[0203] In an embodiment of the present disclosure, after obtaining the data quality assessment result of the first candidate text sample and the data quality assessment threshold corresponding to the i+1th training operation for the multimodal model, the first candidate image sample can be used as the first image sample if the data quality assessment result is greater than or equal to the data quality assessment threshold. At the same time, the first candidate text sample can be used as the first text sample, and the first image sample and the first text sample can be used to form a first image-text sample group to construct a first image-text sample set used when performing the i+1th training operation for the multimodal model. If the data quality assessment result is less than the data quality assessment threshold, the first candidate image sample and the first candidate text sample are discarded.
[0204] See also Figure 4 , which is a scenario diagram of a multimodal model training method provided in an embodiment of the present disclosure.
[0205] As mentioned above, the multimodal model training method provided by the embodiment of the present disclosure is applied to an electronic device. Among them, the electronic device can be a server or a terminal device. Here, the terminal device can be a workbench, a mainframe computer, a conventional computer (for example, a desktop computer, a laptop computer, a car computer, etc.), a personal digital processing or other similar computing device.
[0206] Electronic devices are used for:
[0207] A multimodal model is obtained; wherein the multimodal model includes an image encoder, a bridge component and an LLM connected in sequence.
[0208] Performing a training operation on the multimodal model to obtain a trained multimodal model; wherein the training operation includes:
[0209] When the weight parameters of the LLM in the multimodal model are fixed, the image encoder and the bridging component in the multimodal model are trained using the first image and text sample set to obtain a preliminarily trained multimodal model; and the preliminarily trained multimodal model is trained as a whole using the second image and text sample set to obtain a trained multimodal model.
[0210] It should be noted that in the embodiments of the present disclosure, Figure 4 The scene diagram shown is only illustrative and not restrictive. Those skilled in the art can Figure 4 Various obvious changes and / or substitutions are made to the examples, and the obtained technical solutions still fall within the scope of the disclosure of the embodiments of the present disclosure.
[0211] The present disclosure provides an image processing method, which can be applied to an electronic device. The electronic device can be a server or a terminal device. The terminal device can be a workbench, a mainframe computer, a conventional computer (e.g., a desktop computer, a laptop computer, a car computer, etc.), a personal digital assistant or other similar computing device. Figure 5 The flowchart diagram shown illustrates an image processing method provided by an embodiment of the present disclosure. It should be noted that although a logical sequence is shown in the flowchart diagram, in some cases, the steps shown or described may also be performed in other sequences.
[0212] Step S501, acquiring a target image.
[0213] The target image may be a network image.
[0214] Step S502, obtaining prompt information (Prompt).
[0215] The indication information is used to instruct the target model to perform at least one of a plurality of image processing tasks. Here, the plurality of image processing tasks may include at least one of an image description task, a text extraction task, and an object detection task, that is, the target model has a large applicable range.
[0216] Step S503: using the target model and according to the instruction information, obtaining a text output result corresponding to the target image.
[0217] Among them, the target model is a trained multimodal model obtained through a multimodal model training method.
[0218] In one example, the electronic device is a terminal device. Then, after acquiring the target image and the instruction information, the target image and the instruction information can be sent to the server so that the server uses the target model and obtains the text output result corresponding to the target image according to the instruction information.
[0219] In one example, the text output result corresponding to the target image can be obtained in the following manner:
[0220] Inputting the target image into the image encoder in the target model, so as to encode the target image using the image encoder in the target model to obtain an image encoding result;
[0221] Using the bridge component in the target model, the image encoding result is subjected to feature conversion to obtain a feature conversion result;
[0222] Using the LLM in the target model, the feature conversion results are processed according to the instruction information to obtain the text output result corresponding to the target image.
[0223] By using the image processing method provided by the embodiment of the present disclosure, after acquiring the target image and prompt information, the target model can be used to obtain the text output result corresponding to the target image according to the instruction information, and the target model is a trained multimodal model obtained by the multimodal model training method. Since the target model has excellent working performance, the accuracy of the text output result can be ensured.
[0224] See also Figure 6 , which is a scene diagram of an image processing method provided in an embodiment of the present disclosure.
[0225] As mentioned above, the image processing method provided by the embodiment of the present disclosure is applied to an electronic device. The electronic device may be a server or a terminal device. Here, the terminal device may be a workbench, a mainframe computer, a conventional computer (e.g., a desktop computer, a laptop computer, a vehicle-mounted computer, etc.), a personal digital processing device or other similar computing devices.
[0226] Electronic devices are used for:
[0227] Get the target image;
[0228] Get prompt information;
[0229] Using the target model, according to the instruction information, a text output result corresponding to the target image is obtained; wherein the target model is a trained multimodal model obtained by a multimodal model training method.
[0230] Among them, when the electronic device is a terminal device, after obtaining the target image and instruction information, the target image and instruction information can be sent to the server, so that the server uses the target model and obtains the text output result corresponding to the target image according to the instruction information.
[0231] It should be noted that in the embodiments of the present disclosure, Figure 6 The scene diagram shown is only illustrative and not restrictive. Those skilled in the art can Figure 6 Various obvious changes and / or substitutions are made to the examples, and the obtained technical solutions still fall within the scope of the disclosure of the embodiments of the present disclosure.
[0232] In order to better implement the multimodal model training method, the embodiment of the present disclosure also provides a multimodal model training device, which can be integrated in an electronic device. The electronic device can be a server or a terminal device. Here, the terminal device can be a workbench, a mainframe computer, a conventional computer (for example, a desktop computer, a laptop computer, a car computer, etc.), a personal digital processing or other similar computing device. Figure 7The schematic structural block diagram shown illustrates a multimodal model training device 700 provided in the disclosed embodiment.
[0233] The multimodal model training device 700 comprises:
[0234] A model acquisition unit 701 is used to acquire a multimodal model; wherein the multimodal model includes an image encoder, a bridge component and an LLM connected in sequence;
[0235] The training operation unit 702 is used to perform a training operation on the multimodal model to obtain a trained multimodal model; wherein the training operation includes:
[0236] When the weight parameters of the LLM in the multimodal model are fixed, the image encoder and the bridging component in the multimodal model are trained using the first image and text sample set to obtain a preliminarily trained multimodal model; and the preliminarily trained multimodal model is trained as a whole using the second image and text sample set to obtain a trained multimodal model.
[0237] In some optional implementations, the training operation unit 702 is used to:
[0238] By cyclically executing the training operation for the multimodal model n times, a trained multimodal model is obtained; wherein n ≥ 2 and is a positive integer;
[0239] Among them, the data quality of the first image and text sample set used when performing the i+1th training operation for the multimodal model is better than the data quality of the first image and text sample set used when performing the i-th training operation for the multimodal model; 1≤i≤n-1, and i is a positive integer.
[0240] In some optional implementations, the multimodal model training device 700 further includes a first sample preparation unit, which is used to:
[0241] Obtaining a first candidate image sample;
[0242] Using the data production model, a first candidate text sample corresponding to the first candidate image sample is obtained; wherein the data production model is a trained multimodal model obtained when performing the i-th training operation on the multimodal model;
[0243] Based on the first candidate image sample and the first candidate text sample, a first image-text sample set used for performing the (i+1)th training operation on the multimodal model is constructed.
[0244] In some optional implementations, the first sample preparation unit is used to:
[0245] Obtaining a data quality assessment result of the first candidate text sample;
[0246] Obtaining a data quality assessment threshold corresponding to the i+1th training operation for the multimodal model; wherein the data quality assessment threshold is positively correlated with the value of i+1;
[0247] When the data quality assessment result is greater than or equal to the data quality assessment threshold, a first image and text sample set used for performing the (i+1)th training operation on the multimodal model is constructed based on the first candidate image sample and the first candidate text sample.
[0248] In some optional implementations, the first sample preparation unit is used to:
[0249] Obtaining data quality assessment scores of the first candidate text sample from multiple assessment dimensions respectively, so as to obtain multiple data quality assessment scores;
[0250] Obtaining multiple score weights corresponding to multiple data quality assessment scores;
[0251] Based on the multiple data quality assessment scores and the multiple score weights, a data quality assessment result of the first candidate text sample is obtained.
[0252] In some optional implementations, the multimodal model training device 700 further includes a second sample preparation unit, which is used to:
[0253] Obtaining a second candidate image sample;
[0254] In response to the labeling operation on the second candidate image sample, a second candidate text sample corresponding to the second candidate image sample is obtained;
[0255] Based on the second candidate image samples and the second candidate text samples, a second image-text sample set used for performing a training operation on the multimodal model is constructed.
[0256] In some optional embodiments, the first image and text sample set includes multiple first image and text sample groups, and the multiple first image and text sample groups correspond one-to-one to multiple image processing tasks; and / or the second image and text sample set includes multiple second image and text sample groups, and the multiple second image and text sample groups correspond one-to-one to multiple image processing tasks.
[0257] In some optional implementations, the plurality of image processing tasks include at least one of an image description task, a text extraction task, and an object detection task.
[0258] For the description of the specific functions and examples of each unit of the multimodal model training device 700 of the embodiment of the present disclosure, please refer to the relevant description of the corresponding steps in the aforementioned multimodal model training method embodiment, which will not be repeated here.
[0259] In order to better implement the image processing method, the embodiment of the present disclosure also provides an image processing device, which can be integrated in an electronic device. The electronic device can be a terminal device. Here, the terminal device can be a workbench, a large computer, a conventional computer (for example, a desktop computer, a laptop computer, a car computer, etc.), a personal digital processing or other similar computing device. Figure 8 The schematic structural block diagram shown is used to illustrate an image processing device 800 provided in the disclosed embodiment.
[0260] The image processing device 800 comprises:
[0261] An image acquisition unit 801 is used to acquire a target image;
[0262] An indication acquisition unit 802, used to acquire indication information;
[0263] The image processing unit 803 is used to use the target model to obtain a text output result corresponding to the target image according to the instruction information; wherein the target model is a trained multimodal model obtained by a multimodal model training device.
[0264] In some optional implementations, the instruction information is used to instruct the target model to perform at least one of an image description task, a text extraction task, and a target detection task.
[0265] For the description of specific functions and examples of each unit of the image processing device 800 of the embodiment of the present disclosure, please refer to the relevant description of the corresponding steps in the aforementioned object processing multimodal model training method embodiment, which will not be repeated here.
[0266] In the technical solution disclosed herein, the acquisition, storage and application of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.
[0267] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.
[0268] Fig. 9 A schematic block diagram of an example electronic device 900 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.
[0269] like Fig. 9 As shown, the device 900 includes a computing unit 901, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 902 or a computer program loaded from a storage unit 908 to a random access memory (RAM) 903. In the RAM 903, various programs and data required for the operation of the device 900 can also be stored. The computing unit 901, the ROM 902, and the RAM 903 are connected to each other via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.
[0270] A number of components in the device 900 are connected to the I / O interface 905, including: an input unit 906, such as a keyboard, a mouse, etc.; an output unit 907, such as various types of displays, speakers, etc.; a storage unit 908, such as a disk, an optical disk, etc.; and a communication unit 909, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 909 allows the device 900 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.
[0271] The computing unit 901 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSP), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 901 performs the various methods and processes described above, such as a multimodal model training method and / or an image processing method. For example, in some embodiments, the multimodal model training method and / or the image processing method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 908. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 900 via the ROM 902 and / or the communication unit 909. When the computer program is loaded into RAM 903 and executed by the computing unit 901, one or more steps in the multimodal model training method and / or image processing method described above may be performed. Alternatively, in other embodiments, the computing unit 901 may be configured as a multimodal model training method and / or an image processing method in any other appropriate manner (e.g., by means of firmware).
[0272] Various embodiments of the systems and techniques described above herein may be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.
[0273] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine as a stand-alone software package and partially on a remote machine, or entirely on a remote machine or server.
[0274] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a RAM, a ROM, an erasable programmable read-only memory (EPROM) or a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
[0275] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a cathode ray tube (CRT) display or a liquid crystal display (LCD)) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices are also used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).
[0276] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: Local Area Network (LAN), Wide Area Network (WAN), and the Internet.
[0277] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship to each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.
[0278] An embodiment of the present disclosure also provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable the computer to execute a multimodal model training method and / or an image processing method.
[0279] The embodiments of the present disclosure also provide a computer program product, including a computer program, which implements a multimodal model training method and / or an image processing method when executed by a processor.
[0280] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in the present disclosure can be executed in parallel, sequentially or in a different order, as long as the desired results of the technical solutions disclosed in the present disclosure can be achieved, and this document is not limited here. In addition, in the present disclosure, relational terms such as "first", "second", "third", etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply the existence of any such actual relationship or order between these entities or operations. In addition, "multiple" in the present disclosure can be understood as at least two.
[0281] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the principles of the present disclosure shall be included in the protection scope of the present disclosure.
Claims
1. A multimodal model training method, comprising: Acquire a multimodal model; wherein the multimodal model includes an image encoder, a bridge component, and a large language model connected in sequence; Performing a training operation on the multimodal model to obtain a trained multimodal model; wherein the training operation includes: When the weight parameters of the large language model in the multimodal model are fixed, the image encoder and the bridging component in the multimodal model are trained using the first image and text sample set to obtain a preliminarily trained multimodal model; and the preliminarily trained multimodal model is trained as a whole using the second image and text sample set to obtain a trained multimodal model.
2. The method according to claim 1, wherein: The performing of a training operation on the multimodal model to obtain a trained multimodal model includes: By cyclically executing the training operation for the multimodal model n times, a trained multimodal model is obtained; wherein n ≥ 2 and is a positive integer; Among them, the data quality of the first image and text sample set used when performing the i+1th training operation on the multimodal model is better than the data quality of the first image and text sample set used when performing the i-th training operation on the multimodal model; 1≤i≤n-1, and i is a positive integer.
3. The method according to claim 2, further comprising: Obtaining a first candidate image sample; Using a data production model, obtaining a first candidate text sample corresponding to the first candidate image sample; wherein the data production model is a trained multimodal model obtained when performing the i-th training operation on the multimodal model; Based on the first candidate image sample and the first candidate text sample, a first image-text sample set used for performing the (i+1)th training operation on the multimodal model is constructed.
4. The method according to claim 3, wherein: The step of constructing a first image-text sample set used for performing the (i+1)th training operation on the multimodal model based on the first candidate image sample and the first candidate text sample includes: Obtaining a data quality assessment result of the first candidate text sample; Obtaining a data quality assessment threshold corresponding to the i+1th training operation for the multimodal model; wherein the data quality assessment threshold is positively correlated with the value of i+1; When the data quality assessment result is greater than or equal to the data quality assessment threshold, a first image and text sample set used to perform the (i+1)th training operation on the multimodal model is constructed based on the first candidate image sample and the first candidate text sample.
5. The method according to claim 4, wherein: The obtaining of the data quality evaluation result of the first candidate text sample includes: Obtaining data quality assessment scores of the first candidate text sample from multiple assessment dimensions respectively, so as to obtain multiple data quality assessment scores; Obtaining a plurality of score weights corresponding one-to-one to the plurality of data quality assessment scores; Based on the multiple data quality assessment scores and the multiple score weights, a data quality assessment result of the first candidate text sample is obtained.
6. The method according to claim 2, further comprising: Obtaining a second candidate image sample; In response to the labeling operation on the second candidate image sample, a second candidate text sample corresponding to the second candidate image sample is obtained; Based on the second candidate image sample and the second candidate text sample, a second image-text sample set used for performing a training operation on the multimodal model is constructed.
7. The method according to any one of claims 1 to 6, wherein: The first image-text sample set includes multiple first image-text sample groups, and the multiple first image-text sample groups correspond one-to-one to multiple image processing tasks; and / or the second image-text sample set includes multiple second image-text sample groups, and the multiple second image-text sample groups correspond one-to-one to multiple image processing tasks.
8. The method according to claim 7, wherein: The plurality of image processing tasks include at least one of an image description task, a text extraction task and an object detection task.
9. An image processing method, comprising: Get the target image; Get instructions; Using the target model, according to the indication information, a text output result corresponding to the target image is obtained; wherein the target model is a trained multimodal model obtained by the method described in any one of claims 1 to 8.
10. The method according to claim 9, wherein: The instruction information is used to instruct the target model to perform at least one of an image description task, a text extraction task, and a target detection task.
11. A multimodal model training device, comprising: A model acquisition unit, used to acquire a multimodal model; wherein the multimodal model includes an image encoder, a bridge component and a large language model connected in sequence; A training operation unit, configured to perform a training operation on the multimodal model to obtain a trained multimodal model; wherein the training operation includes: When the weight parameters of the large language model in the multimodal model are fixed, the image encoder and the bridging component in the multimodal model are trained using the first image and text sample set to obtain a preliminarily trained multimodal model; and the preliminarily trained multimodal model is trained as a whole using the second image and text sample set to obtain a trained multimodal model.
12. The device according to claim 11, wherein The training operation unit is used for: By cyclically executing the training operation for the multimodal model n times, a trained multimodal model is obtained; wherein n ≥ 2 and is a positive integer; Among them, the data quality of the first image and text sample set used when performing the i+1th training operation on the multimodal model is better than the data quality of the first image and text sample set used when performing the i-th training operation on the multimodal model; 1≤i≤n-1, and i is a positive integer.
13. The device according to claim 12, further comprising a first sample preparation unit, configured to: Obtaining a first candidate image sample; Using the data production model, a first candidate text sample corresponding to the first candidate image sample is obtained; wherein, The data production model is a trained multimodal model obtained when performing the i-th training operation on the multimodal model; Based on the first candidate image sample and the first candidate text sample, a first image-text sample set used for performing the (i+1)th training operation on the multimodal model is constructed.
14. The device according to claim 13, wherein: The first sample making unit is used for: Obtaining a data quality assessment result of the first candidate text sample; Obtaining a data quality assessment threshold corresponding to the i+1th training operation for the multimodal model; wherein the data quality assessment threshold is positively correlated with the value of i+1; When the data quality assessment result is greater than or equal to the data quality assessment threshold, a first image and text sample set used to perform the (i+1)th training operation on the multimodal model is constructed based on the first candidate image sample and the first candidate text sample.
15. The device according to claim 14, wherein: The first sample making unit is used for: Obtaining data quality assessment scores of the first candidate text sample from multiple assessment dimensions respectively, so as to obtain multiple data quality assessment scores; Obtaining a plurality of score weights corresponding one-to-one to the plurality of data quality assessment scores; Based on the multiple data quality assessment scores and the multiple score weights, a data quality assessment result of the first candidate text sample is obtained.
16. The device according to claim 12, further comprising a second sample preparation unit, configured to: Obtaining a second candidate image sample; In response to the labeling operation on the second candidate image sample, a second candidate text sample corresponding to the second candidate image sample is obtained; Based on the second candidate image sample and the second candidate text sample, a second image-text sample set used for performing a training operation on the multimodal model is constructed.
17. The device according to any one of claims 11 to 16, wherein: The first image-text sample set includes multiple first image-text sample groups, and the multiple first image-text sample groups correspond one-to-one to multiple image processing tasks; and / or the second image-text sample set includes multiple second image-text sample groups, and the multiple second image-text sample groups correspond one-to-one to multiple image processing tasks.
18. The device according to claim 17, wherein: The plurality of image processing tasks include at least one of an image description task, a text extraction task and an object detection task.
19. An image processing device, comprising: An image acquisition unit, used for acquiring a target image; An indication acquisition unit, used for acquiring indication information; An image processing unit, used to use the target model to obtain a text output result corresponding to the target image according to the indication information; wherein the target model is a trained multimodal model obtained by the device described in any one of claims 11 to 18.
20. The device according to claim 19, wherein The instruction information is used to instruct the target model to perform at least one of an image description task, a text extraction task, and a target detection task.
21. An electronic device, comprising: at least one processor; a memory communicatively coupled to the at least one processor; The memory stores instructions that can be executed by the at least one processor. The instructions are executed by the at least one processor so that the at least one processor can perform the method according to any one of claims 1 to 10.
22. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 10.
23. A computer program product, comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 10.
Citation Information
Cited By
Training method and apparatus for multi-modal large model
WO2026179596A1
