Training method, device, storage medium and electronic device for multimodal large model

Abstract

The present application discloses a training method, device, storage medium and electronic device for a multimodal large model. The training method includes: using a pre-trained label expert LMM for labeling to predict labels for original multimodal training data; filtering the original multimodal training data based on answer rating labels to obtain a filtered data set; based on the image labels and instruction labels of all data samples in the filtered data set, selecting data samples from the filtered data set as sampling data to maximize the label entropy determined based on the image labels and instruction labels of all the sampled data; and using the sampled data as training samples to train the multimodal large model. By using the present application, the filtered training data can be used for multimodal large model training, and the performance of the multimodal large model can be effectively guaranteed.

Description

Technical Field

[0001] The present application relates to neural network technology, and in particular to a training method, storage medium and electronic device for a multimodal large model. Background Art

[0002] The training of a large multimodal model (LMM) requires the use of multimodal instruction question-answering data in the form of "image-instruction-answer" triplets. In order for a large multimodal model to acquire general capabilities, multimodal training data generally has a large amount of data and needs to meet the requirements of diversity and high quality.

[0003] The sources of multimodal training data include: multimodal training data sets publicly available on the Internet, synthesized or modified single-modal training data sets, data generated by strong LMM, manual calibration, etc., and exist in different data sets. Although these multimodal training data sets come from various sources, if these data sets are directly mixed for LMM training without screening, there will be the following problems: 1. Poor image diversity: Because different data sets will frequently use images of certain specific scenes, such as daily pictures and close-ups; 2. Poor instruction diversity: Different data sets have instruction preferences, and the data volumes of these data sets are different. Directly mixing data sets will cause the instruction types that appear in data sets with large data volumes to occupy a larger proportion of the total data, resulting in an imbalance in diversity. Therefore, it is very necessary to further process and screen the multimodal data sets collected from various sources. The unscreened data is called the original multimodal training data to distinguish the data actually used for training after screening. The existing method of screening the original multimodal training data, after using the screened training data for LMM training, found that it would affect the model performance of LMM and make the model performance worse. Summary of the invention

[0004] The present application provides a training method, device, storage medium and electronic device for a multimodal large model, which can use screened training data to train the multimodal large model and effectively ensure the performance of the multimodal large model.

[0005] To achieve the above objectives, this application adopts the following technical solutions:

[0006] A training method for a multimodal large model, comprising:

[0007] Predicting labels for the original multimodal training data using a pre-trained label expert multimodal large model LMM for label annotation; wherein the labels include image labels for marking image categories, instruction labels for marking instruction categories, and answer rating labels for marking answer quality;

[0008] Filtering the original multimodal training data based on the answer rating labels to obtain a filtered data set;

[0009] Based on the image labels and instruction labels of all data samples in the filtered data set, selecting data samples from the filtered data set as sampled data, so as to maximize the label entropy determined based on the image labels and instruction labels of all sampled data;

[0010] The sampled data is used as training samples to train a large multimodal model.

[0011] Preferably, the method further comprises: pre-training and generating the label expert LMM based on the general LMM using the image label seed training data, instruction label seed training data and answer rating label seed training data determined based on the original multimodal data;

[0012] The image label seed training data includes a picture selected from the original multimodal training data, an instruction for instructing to predict an image label for the picture, and an answer for recording the image label;

[0013] The instruction label seed training data includes instructions selected from the original multimodal training data, pictures corresponding to the selected instructions, and answers for recording instruction labels;

[0014] The answer rating label seed training data includes pictures in the data samples selected based on the answers from the original multimodal training data, instructions for indicating answer rating, and answers for recording label ratings, wherein the label rating is the rating result corresponding to the answer label to which the answer in the selected data sample belongs.

[0015] Preferably, the label rating is a label classification of each label to which the answer in the selected data sample belongs, and the method further comprises: determining a comprehensive label classification based on the label classification of each label; or, the answer rating label comprises a comprehensive label classification determined based on the label classification of each label to which the answer in the selected data sample belongs;

[0016] The filtering of the original multimodal data based on the answer rating labels includes: filtering out data samples whose comprehensive label classification does not meet the set requirements.

[0017] Preferably, the selecting a data sample from the filtered data set as the sampling data comprises:

[0018] a. Using the filtered data set as the current filtered data set and setting the current sampled data set to an empty set;

[0019] b. In the current filtering data set, for each current data sample, the label entropy is calculated jointly with the current data sample and all data samples in the current sampling data set as the entropy corresponding to the current data sample;

[0020] c. Determine the maximum entropy value corresponding to each data sample, add the data sample corresponding to the maximum entropy value to the current sampling data set, delete the corresponding current data sample from the current filtered data set, and return to step b until the number of samples in the current sampling data set meets the requirements.

[0021] Preferably, the selecting a data sample from the filtered data set as the sampling data comprises:

[0022] a. Set the current sampling data set to an empty set;

[0023] b. Selecting N data samples from the untraversed data samples in the filtering data set as the current filtering data set; N is a preset positive integer that is less than the total number of data samples in the filtering data set and greater than 1;

[0024] c. In the current filtered data set, traverse each current data sample, and jointly calculate the label entropy of the current data sample and all the data samples in the current sampling data set as the entropy corresponding to the current data sample; determine the maximum value of the entropy corresponding to each data sample in the current filtered data set. If the maximum value is greater than the label entropy jointly calculated by all the data samples in the current sampling data set, add the current data sample corresponding to the maximum entropy value to the current sampling data set, and return to step b until the number of samples in the current sampling data set meets the requirement; otherwise, directly return to step b until the number of samples in the current sampling data set meets the requirement.

[0025] Preferably, the selecting a data sample from the filtered data set as the sampling data comprises:

[0026] a. Set the current sampling data set to an empty set;

[0027] b. In the filtered data set, select a current data sample in turn, and jointly calculate the label entropy of the current data sample and all the data samples in the current sampling data set as the entropy corresponding to the current data sample; if the entropy corresponding to the current data sample is greater than the label entropy jointly calculated by all the data samples in the current sampling data set, add the current data sample to the current sampling data set, and process the next selected data sample until the number of samples in the current sampling data set meets the requirement; if the entropy corresponding to the current data sample is not greater than the label entropy jointly calculated by all the data samples in the current sampling data set, directly process the next selected data sample until the number of samples in the current sampling data set meets the requirement.

[0028] Preferably, the method further comprises: using the trained multimodal large model to further train the label expert LMM; and / or,

[0029] The labels and corresponding data samples predicted by the label expert LMM for the data samples in the original multimodal data are used as training data to further train the label expert LMM.

[0030] A multimodal large model training device, the device comprising: a label prediction unit, a data filtering unit, a data sampling unit and a multimodal large model training unit;

[0031] The label prediction unit is used to predict a label for each data sample in the original multimodal data using a pre-trained label expert multimodal large model LMM for label annotation; wherein the label includes an image label for marking an image category, an instruction label for marking an instruction category, and an answer rating label for marking an answer quality;

[0032] The data filtering unit is used to filter the original multimodal data based on the answer rating label to obtain a filtered data set;

[0033] The data sampling unit is used to select data samples from the filtered data set as sampling data based on the image labels and instruction labels of all data samples in the filtered data set, so as to maximize the label entropy determined based on the image labels and instruction labels of all the sampling data;

[0034] The multimodal large model training unit is used to train the multimodal large model using the sampled data as training samples.

[0035] Preferably, the device further comprises a label expert training unit, which is used to train and generate the label expert LMM in advance based on the general LMM using image label seed training data, instruction label seed training data and answer rating label seed training data determined based on the original multimodal data;

[0036] The image label seed training data includes a picture selected from the original multimodal data, an instruction for instructing to predict an image label for the picture, and an answer for recording the image label;

[0037] The instruction label seed training data includes instructions selected from the original multimodal data, pictures corresponding to the selected instructions, and answers for recording instruction labels;

[0038] The answer rating label seed training data includes pictures and instructions in the data samples selected from the original multimodal data based on the answers, and answers for recording label ratings, wherein the label ratings are rating results corresponding to the answer labels to which the answers in the selected data samples belong.

[0039] Preferably, the label rating is a label classification of each label to which the answer in the selected data sample belongs, and the label expert training unit is further used to determine a comprehensive label classification based on the label classification of each label; or, the answer rating label is a comprehensive label classification determined based on the label classification of each label to which the answer in the selected data sample belongs;

[0040] The data filtering unit is used to filter out data samples whose comprehensive label classification does not meet the set requirements.

[0041] Preferably, in the data sampling unit, selecting a data sample from the filtered data set as the sampling data includes:

[0042] a. Using the filtered data set as the current filtered data set and setting the current sampled data set to an empty set;

[0043] b. In the current filtering data set, for each current data sample, the label entropy is calculated jointly with the current data sample and all data samples in the current sampling data set as the entropy corresponding to the current data sample;

[0044] c. Determine the maximum entropy value corresponding to each data sample, add the data sample corresponding to the maximum entropy value to the current sampling data set, delete the corresponding current data sample from the current filtered data set, and return to step b until the number of samples in the current sampling data set meets the requirements.

[0045] Preferably, in the data sampling unit, selecting a data sample from the filtered data set as the sampling data includes:

[0046] a. Set the current sampling data set to an empty set;

[0047] b. Selecting N data samples from the untraversed data samples in the filtering data set as the current filtering data set;

[0048] c. In the current filtered data set, traverse each current data sample, and jointly calculate the label entropy of the current data sample and all the data samples in the current sampling data set as the entropy corresponding to the current data sample; determine the maximum value of the entropy corresponding to each data sample in the current filtered data set. If the maximum value is greater than the label entropy jointly calculated by all the data samples in the current sampling data set, add the current data sample corresponding to the maximum entropy value to the current sampling data set, and return to step b until the number of samples in the current sampling data set meets the requirement; otherwise, directly return to step b until the number of samples in the current sampling data set meets the requirement.

[0049] Preferably, in the data sampling unit, selecting a data sample from the filtered data set as the sampling data includes:

[0050] a. Set the current sampling data set to an empty set;

[0051] b. In the filtered data set, select a current data sample in turn, and jointly calculate the label entropy of the current data sample and all the data samples in the current sampling data set as the entropy corresponding to the current data sample; if the entropy corresponding to the current data sample is greater than the label entropy jointly calculated by all the data samples in the current sampling data set, add the current data sample to the current sampling data set, and process the next selected data sample until the number of samples in the current sampling data set meets the requirement; if the entropy corresponding to the current data sample is not greater than the label entropy jointly calculated by all the data samples in the current sampling data set, directly process the next selected data sample until the number of samples in the current sampling data set meets the requirement.

[0052] Preferably, the label expert training unit is used to further train the label expert LMM using the trained multimodal large model; and / or,

[0053] The label expert training unit is used to use the labels and corresponding data samples predicted by the label expert LMM for the data samples in the original multimodal data as training data, so as to further train the label expert LMM.

[0054] A computer-readable storage medium having computer instructions stored thereon, wherein the instructions, when executed by a processor, can implement any of the multimodal large model training methods described above.

[0055] An electronic device comprising at least a computer-readable storage medium and a processor;

[0056] The processor is used to read executable instructions from the computer-readable storage medium and execute the instructions to implement any of the multimodal large model training methods described above.

[0057] It can be seen from the above technical solution that in this application, firstly, a pre-trained label expert LMM for label annotation is used to predict labels for each data sample in the original multimodal training data to obtain image labels, instruction labels and answer rating labels, thereby adding three new dimensions of information to each data sample, which are used to characterize the image category, instruction category and answer quality of the data sample respectively; next, the original multimodal data is filtered based on the answer rating labels of each data sample to obtain a filtered data set. In this process, since the answer rating labels can characterize the answer quality of the data samples, the corresponding filtering process can filter out the data samples whose quality does not meet the requirements in the original multimodal training data, thereby improving the quality of the training data of the multimodal large model, and then improving the performance of the trained multimodal large model; next, based on the image labels and instruction labels of all data samples in the filtered data set, data samples are selected from the filtered data set as sampling data. The principle of data selection is to maximize the label entropy determined based on the image labels and instruction labels of all the sampled data. Since the image labels and instruction labels respectively characterize the image category and instruction category of the data sample, the diversity of the sampled data can be achieved based on the principle of maximizing label entropy; finally, the sampled data is used to train the multimodal large model. On the one hand, low-quality training data is filtered out through answer rating labels, and on the other hand, more diverse data is selected for model training through image labels and instruction labels. Therefore, the data used to train multimodal large models can meet the requirements of diversity and high quality at the same time. Training based on such data can effectively guarantee the performance of multimodal large models. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 A schematic diagram of the basic process of the LMM model training method in this application;

[0059] Figure 2 This is a schematic diagram of a specific process of the LMM model training method in a specific embodiment of the present application;

[0060] Figure 3 This is an example diagram of the image label space in the specific embodiment of this application;

[0061] Figure 4 This is an example diagram of the instruction label space in a specific embodiment of the present application;

[0062] Figure 5 This is an example diagram of the answer label space in a specific embodiment of this application;

[0063] Figure 6 This is an example diagram of the implementation process of selecting sampling data through method 1 in a specific embodiment of the present application;

[0064] Figure 7This is an example diagram of the implementation process of selecting sampling data by way 2 in a specific embodiment of the present application;

[0065] Figure 8 This is an example diagram of the implementation process of selecting sampling data by way three in a specific embodiment of the present application;

[0066] Fig. 9 This is a schematic diagram of the basic structure of the LMM model training device in this application;

[0067] Fig.10 This is a schematic diagram of the basic structure of the electronic device provided in this application. DETAILED DESCRIPTION

[0068] In order to make the objectives, technical means and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings.

[0069] The training data of multimodal large models are usually in the form of "image-instruction-answer" triples, as shown in Table 1. Since the training data includes datasets of various types and sizes, if these datasets are directly mixed for LMM training without screening, there will be the following problems: 1. Poor image diversity: Because different datasets will frequently use images of certain specific scenes, such as daily pictures and close-ups; 2. Poor instruction diversity: Different datasets have instruction preferences, and the data volume of these datasets is different. Directly mixing datasets will cause the instruction types that appear in the datasets with large data volumes to occupy a larger proportion of the total data, resulting in an imbalance in diversity. For example, the instructions of the Dm dataset are all multiple-choice questions, with a data volume of 1 million, and the instructions of the Dn dataset are various open-ended questions, with a data volume of only 1,000. If the Dm and Dn datasets are directly mixed, the trained LMM will answer multiple-choice questions well, but answer open-ended questions poorly, resulting in the lack of general capabilities of the LMM. Therefore, multimodal datasets collected from various sources need to be further processed and screened.

[0070] Table 1

[0071]

[0072] The present application provides a training method for a multimodal large model. The original multimodal training data is screened for data quality based on the accuracy of the answer, and then the data sample diversity is screened based on the image category and the instruction category, so as to select training data with high quality and more balanced diversity for training the multimodal large model, which can effectively ensure the performance of the trained large model.

[0073] Figure 1 Schematic diagram of the basic process of the multimodal large model training method in this application. Figure 1 As shown, the method includes:

[0074] Step 101 , using a pre-trained label expert LMM for label annotation, predicting labels for data samples in the original multimodal training data.

[0075] Among them, the labels include image labels for marking image categories, instruction labels for marking instruction categories, and answer rating labels for marking answer quality.

[0076] The input data of the label expert LMM comes from the original multimodal training data. As mentioned earlier, the multimodal training data includes "image-instruction-answer". The label expert LMM gives image labels for the images in the input data to mark the image categories, gives instruction labels for the instructions in the input data to mark the instruction categories, and gives answer rating labels for the answers in the input data to mark the answer quality.

[0077] The label expert LMM is obtained through pre-training, and can be obtained through fine-tuning based on the general LMM model.

[0078] Step 102: Filter the original multimodal training data based on the answer rating labels to obtain a filtered data set.

[0079] As mentioned above, the answer rating label is used to characterize the quality of the answer, and can also characterize the quality of the corresponding training data. In LMM model training, if the quality of the training data is low, it will affect the accuracy of the model training. Therefore, in this application, the training data with lower quality in the original multimodal training data is filtered out, leaving the training data with the required quality. Among them, the answer rating label is used to measure the quality of the training data. Generally, the original multimodal training data whose answer rating represented by the answer rating label is lower than the standard rating can be filtered out. The specific standard rating can be set according to actual experience. The original multimodal training data that has not been filtered out will form a filtered data set.

[0080] Step 103 , based on the image labels and instruction labels of all data samples in the filtered data set, select data samples from the filtered data set as sampled data, so as to maximize the label entropy determined based on the image labels and instruction labels of all sampled data.

[0081] As mentioned above, the image label and the instruction label are used to mark the image category and instruction category of the training data, respectively. In order to make the screened training data meet the diversity requirements, the training data is screened based on the image category and instruction category of the training data in this application, that is, the screening is performed based on the image label and instruction label of the training data. During the screening, the label entropy is calculated based on the image labels and instruction labels of all the sampled data, and the sampled data is screened out using the label entropy maximization principle. Among them, the sampled data is the data sample selected from the filtered data set for LMM model training.

[0082] Step 104: Use the sampled data as training samples to train the multimodal large model LMM.

[0083] The sampled data obtained after screening in the aforementioned steps 102 and 103 are used as training samples to train the LMM. The specific process of using the sampled data to train the LMM model can be performed in an existing sampling manner. Since the data samples that meet the quality requirements are screened out through the aforementioned step 102, and the data sample set that meets the diversity requirements is further screened out through step 103, the sampled data obtained thereby meets both the quality requirements and the diversity requirements, and can effectively guarantee the performance of the trained LMM model.

[0084] At this point, the multimodal large model training method process in this application ends. Through the above processing of this application, on the one hand, the label expert LMM is used to label the original multimodal training data with image labels, instruction labels, and answer rating labels. On the other hand, the training data is screened based on the three types of labeled labels to find sampling data that meets the requirements of quality and diversity for training the LMM model, thereby effectively ensuring the performance of the LMM model.

[0085] The specific implementation of the multimodal large model training method of the present application is explained below through specific examples. Figure 2 Schematic diagram of the specific process of the multimodal large model training method in the embodiment of this application. Figure 2 As shown, the method includes:

[0086] Step 201 , based on the general LMM, using the image label seed training data, instruction label seed training data and answer rating label seed training data determined based on the original multimodal data, train and generate a label expert LMM.

[0087] In this application, it is necessary to use a label expert LMM to predict labels for the original multimodal training data. The label expert LMM used is obtained through pre-training. Optionally, the label expert LMM can be obtained through further training based on the general LMM.

[0088] When training the label expert LMM, image label seed training data, instruction label seed training data and answer rating label seed training data are determined based on the original multimodal training data to be used as training data for the label expert LMM.

[0089] Among them, the image label seed training data is mainly used to train the label expert LMM's prediction ability for image labels, the instruction label seed training data is mainly used to train the label expert LMM's prediction ability for instruction labels, and the answer rating label seed training data is mainly used to train the label expert LMM's prediction ability for answer rating labels.

[0090] When determining the image label seed training data based on the original multimodal training data, the specific processing may include:

[0091] First, define the label space of the image label; the image label space can be defined in a tree structure as follows Figure 3 As shown, this label space is used as label candidates when annotating images in the original multimodal training data;

[0092] Secondly, representative images are selected from the original multimodal training data. These images can belong to one or more labels in the label space, and the selected image set has a certain diversity in the label level.

[0093] Then, receive the image labels corresponding to the image content, the number of labels can be one or more;

[0094] Finally, the image and the corresponding image label are organized into a triplet of "image-instruction-answer" to obtain an image label seed training data; wherein the image refers to the selected image, the instruction can be the content corresponding to the instruction column in Table 2, which is used to indicate the marking of the image label, and the answer refers to the image label determined for the image. For different image label seed training data, the instructions can be the same.

[0095] Table 2

[0096]

[0097] For each selected picture, the corresponding image label seed training data is obtained in the above manner to train the image label prediction ability of the label expert LMM.

[0098] When determining instruction label seed training data based on original multimodal training data, specific processing may include:

[0099] First, define the label space of the instruction label; the image label space can be defined in a tree structure as follows Figure 5As shown in Figure 1, this label space is used as a label candidate for annotating instructions in the original multimodal training data; wherein, the setting of the label space can be extended based on the application field. For example, in the field of monitoring, it can be set in the image label space. Figure 4 The label category of the overlay information shown;

[0100] Secondly, representative instructions are selected from the original multimodal training data. These instructions can belong to one or more labels in the label space, and the selected instruction set has a certain diversity in the label level.

[0101] Then, receive instruction labels corresponding to the instructions, which are in accordance with the instruction content. The number of labels can be one or more.

[0102] Finally, the image corresponding to the instruction, the instruction, and the instruction label corresponding to the instruction are organized into a triplet of "image-instruction-answer" to obtain an instruction label seed training data; wherein, the image refers to the image corresponding to the selected instruction, the instruction refers to the selected instruction, and may also include extended information for indicating the marking of the instruction label (the exemplary specific content can be seen in the instruction column content in Table 3 below, which may include the major categories of instruction labels and all detailed labels under each major category as well as the specific selected instruction content itself), and the answer refers to the instruction label determined for the selected instruction. It is worth noting that since the instruction label is related to the image, it is impossible to give the correct instruction label without providing the image, so the corresponding image must be added when organizing the instruction label.

[0103] Table 3

[0104]

[0105] For each selected instruction, the corresponding instruction label seed training data is obtained in the above manner to train the instruction label prediction capability of the label expert LMM.

[0106] When determining answer rating label seed training data based on original multimodal training data, specific processing may include:

[0107] First, define the label space of the answer label; the answer label space can be defined in a tree structure as follows Figure 5 As shown, this label space is used as label candidates and corresponding rating results when annotating answers in the original multimodal training data; specifically, in order to characterize the quality of the answer, in addition to various types of label candidates, the label space also gives the rating results corresponding to each label candidate to indicate the quality of the answer corresponding to the corresponding label candidate; in this embodiment, Figure 5The rating results given in the answer are expressed by scores, and of course they can be expressed in other ways, such as letter grades, etc.); In addition, different categories of answer labels can be arranged in parallel or in a progressive manner. When an answer is scored 0 in a certain evaluation item, all subsequent scores are 0. For example, the four label categories of "relevance", "format style", "language expression" and "credibility" in this embodiment are not arranged in parallel, but in a progressive manner;

[0108] Secondly, representative multimodal data triplets are selected from the original multimodal training data. The answers in these data triplets can belong to one or more answer labels in the label space, and the answers in the selected data triplets have a certain diversity in the answer label level.

[0109] Then, answer labels and corresponding rating results corresponding to the answers are received, and the number of labels can be one or more; optionally, on the basis of giving the rating results of each different category label, a comprehensive rating result obtained based on the rating results of each different category label can be further given. Compared with the method of giving a comprehensive rating result alone without giving the rating results of each category label, the method of giving both rating results uses a thinking chain method, which increases the reasoning and thinking process of the model and can effectively optimize the training results of the model;

[0110] Finally, the image in the selected multimodal data triplet, the instruction for indicating the answer label rating, and the answer for recording the label rating are organized into an "image-instruction-answer" triplet to obtain an answer rating label seed training data; wherein, the image refers to the picture in the selected data triplet, the instruction refers to the mark for indicating the answer rating label (the specific content can be seen in the instruction column in Table 4 below), and the answer refers to the rating label determined for the given answer label, which can be the rating result corresponding to the answer label under each answer label category, or it can be based on the rating results of the above-mentioned various answer labels, and further include a comprehensive rating result.

[0111] Table 4

[0112]

[0113] For each selected multimodal data triple, the corresponding answer rating label seed training data is obtained in the above manner to train the answer rating label prediction ability of the label expert LMM.

[0114] The selection of the above-mentioned image label training data and label marking can be manually selected or marked. Because the general LMM has strong comprehensive capabilities, only a small part (several hundred images) of the training data of the three labels of image, instruction, and answer are needed to train the label expert LMM, namely the above-mentioned image label seed training data, instruction label seed training data, and answer rating label seed training data. The cost of manual selection and labeling is not high.

[0115] Through the processing of this step, the label expert LMM can be obtained by training the general LMM using the above training data.

[0116] Step 202: input the original multimodal training data into the label expert LMM for label prediction.

[0117] As mentioned above, each piece of data in the original multimodal training data is a triplet of "image-command-answer". In this step, the triplet will be expanded into a sextuple of "image-command-answer-image label-command label-answer label" through the processing of the label expert LMM. The newly added image label, command label and answer rating label will be used for subsequent quality filtering and diversity sampling of the original multimodal training data. For example, the original multimodal training data and its label prediction results can be represented as DT i ={(M j , Q j , A j , MT j , QT j , AT j )} j =1:n i , where MT, QT, and AT represent image labels, instruction labels, and answer rating labels, respectively. Optionally, the answer rating label can be a comprehensive evaluation result of the rating results of answer labels of each category, or can also be the rating result of answer labels of each category.

[0118] Regarding the label expert LMM obtained through the aforementioned step 201, since it is fine-tuned based on the general LMM and is trained by mixing three types of label data, a label expert LMM can have the ability to give corresponding label prediction results given relevant label instructions, and one model can complete the prediction of three labels at the same time. Use the label expert LMM to traverse each data in the original multimodal training data, and use the image label instructions in Table 2, the instruction label instructions in Table 3, and the answer rating label instructions in Table 4 to generate three labels for each data, and complete the above-mentioned six-tuple for standby.

[0119] Step 203 , based on the answer rating labels predicted in step 202 , filter out data samples whose instructions do not meet the requirements from the original multimodal training data.

[0120] There are two situations in the answer rating labels predicted in step 202: 1. The given answer rating labels only have the rating results of each label category, and no comprehensive rating results; 2. The given answer rating labels include the rating results of each label category and the comprehensive rating results.

[0121] Accordingly, in this step, for the first case, it is necessary to give a comprehensive rating result according to the rating results of each label category in accordance with the set rules. The specific processing of obtaining the comprehensive rating result based on the single rating results can be set according to actual needs; for the second case, the comprehensive rating result can be found directly in the answer rating label.

[0122] According to the comprehensive rating results of the original multimodal training data, filtering is performed to select data samples that meet the quality requirements. Specifically, a comprehensive rating result threshold can be set, and data samples above the corresponding threshold are retained, and the rest are filtered out, and the retained data samples form a filtered data set. The comprehensive rating result threshold in the swamp can be flexibly set according to the requirements for data quality. For example, in this embodiment, the comprehensive rating result is the sum of the individual rating results, and the comprehensive rating result threshold is set to 4.

[0123] Step 204 : In the filtered data set, data samples are selected as sampled data based on the image labels and the instruction labels, so as to maximize the label entropy determined based on the image labels and the instruction labels of all the sampled data.

[0124] In this embodiment, the data sample is selected based on diversity sampling that maximizes label entropy. Specifically, a data sample is selected from the filtered data set DT', which, after being added to the actual multimodal training data (i.e., the sampled data obtained after sampling), can maximize the label entropy based on all the sampled data after sampling.

[0125] In this embodiment, the label entropy may be calculated, for example, as follows:

[0126] Label entropy = Image label entropy + Instruction label entropy =

[0127] in, n MT is the total number of image label types in all sampled data obtained after sampling, p i It is i The frequency of image labels appearing in all image labels, p i = Image Tag i The number of / total number of all image labels, here, image label iWhen calculating the number, the count of image tag i is increased by 1 each time it appears, and it is counted as multiple times. The total number of all image tags refers to the total number of times all image tags appear. The count of image tag i is increased by 1 each time it appears, and it is counted as multiple times; similarly, n QT is the number of instruction labels in the sampled data, q j It is j The frequency of this instruction tag appearing in all instruction tags, q i = Instruction label j The total number of all instruction tags.

[0128] Regarding the specific processing of sampling data selection based on the label entropy maximization principle, this embodiment provides the following exemplary implementation methods.

[0129] Method 1:

[0130] In the first method of selecting sampling data, the specific process may include:

[0131] a, set the filtered data set as the current filtered data set and set the current sampled data set to the empty set;

[0132] b. In the current filtering data set, for each current data sample, the label entropy is calculated jointly with the current data sample and all the data samples in the current sampling data set as the entropy corresponding to the current data sample;

[0133] c. Determine the maximum entropy value corresponding to each data sample, add the current data sample corresponding to the maximum entropy value to the current sampling data set, delete the corresponding current data sample from the current filtered data set, and return to step b until the number of samples in the current sampling data set meets the requirements.

[0134] In practical applications, the implementation process of the sampling data selection of the above method 1 can be as follows: Figure 6 As shown, specifically including:

[0135] s601, initialize the current sampling data set to an empty set, and initialize the current filtering data set to a filtering data set;

[0136] Accordingly, the label entropy of the current sampled data set is 0;

[0137] s602, determine whether the sampled data reaches the set sample number, if so, end this process, otherwise, execute s603;

[0138] s603, set the maximum label entropy to 0, and set the data index corresponding to the maximum label entropy to -1;

[0139] s604, determine whether all data samples in the current filtered data set have been traversed, if so, execute s608, otherwise, execute s605;

[0140] s605, jointly calculating the label entropy of the traversed current data sample and all data samples in the current sampling data set, and using the calculation result as the entropy corresponding to the current data sample;

[0141] Among them, the processing of this step is to assume that the current data sample is added to the current sampling data set to obtain the updated current sampling data set, and calculate the label entropy of the updated current sampling data set, that is, based on the current data sample and the image labels and instruction labels of all data samples in the current sampling data set before the update, calculate the label entropy. Of course, adding the current sampling data set here is just an assumption. In fact, the current sampling data set remains unchanged before and after this step is executed.

[0142] s606, determine whether the entropy corresponding to the current data sample is greater than the maximum label entropy. If so, execute s607, otherwise return to s604 to determine whether the traversal is completed, and continue to traverse the next data sample if it is determined that the traversal is not completed;

[0143] s607, updating the maximum label entropy to the entropy corresponding to the current data sample, and updating the data index corresponding to the maximum label entropy to the index of the current data sample, and then returning to s604 to determine whether the traversal is completed, and continuing to traverse the next data sample when it is determined that the traversal is not completed;

[0144] s608, adding the data sample pointed to by the data index corresponding to the maximum label entropy to the current sampling data set, and deleting it from the current filtering data set, and continuing to execute s602.

[0145] The above implementation method is a sampling data selection method based on the maximum global label entropy. In each round, the currently unselected data samples are used to form the current filtered data set, and the label entropy of all data samples in the current filtered data set is calculated one by one after being added to the sampling data set, and the data sample corresponding to the maximum label entropy is selected to be added to the sampling data set; that is, in each round of selection, among all the currently unselected data samples, the corresponding data sample with the largest entropy is selected as the sampling data to be added to the sampling data set, thereby maximizing the global label entropy. This selection method is theoretically the best approach for result diversity, and can ensure that the label entropy of the sampled data is the largest. However, from the perspective of algorithm complexity, its time complexity is relatively high. Assuming that the amount of data of the original multimodal training data DT' after filtering is N, and the amount of data of the sampling data set is M, the time complexity of the above selection method is O(MN).

[0146] The following two selection methods strike a balance between diversity selection and time complexity, effectively reducing time complexity while ensuring better diversity selection.

[0147] Method 2:

[0148] In the second method of selecting sampling data, the specific process may include:

[0149] a. Set the current sampling data set to an empty set;

[0150] b. Select N data samples from the untraversed data samples in the filtered data set as the current filtered data set;

[0151] c. In the current filtered data set, traverse each current data sample, and jointly calculate the label entropy of the current data sample and all the data samples in the current sampling data set as the entropy corresponding to the current data sample; determine the maximum value of the entropy corresponding to each data sample in the current filtered data set. If the maximum value is greater than the label entropy jointly calculated by all the data samples in the current sampling data set, add the current data sample corresponding to the maximum entropy value to the current sampling data set, and return to step b until the number of samples in the current sampling data set meets the requirement; otherwise, directly return to step b until the number of samples in the current sampling data set meets the requirement.

[0152] In practical applications, the implementation process of the sampling data selection of the above method 2 can be as follows: Figure 7 As shown, specifically including:

[0153] s701, initialize the current sampling data set to an empty set;

[0154] Accordingly, the label entropy of the current sampled data set is 0;

[0155] s702, determine whether the sampled data reaches the set sample number, if so, end this process, otherwise, execute s703;

[0156] The set number of samples refers to the total number of sampled data;

[0157] s703, sequentially selecting N data samples from the untraversed data samples in the filtering data set as the current filtering data set;

[0158] N is a preset positive integer less than the total number of data samples in the filtered data set and greater than 1. It can be determined according to the requirements for data diversity and the tolerance for time complexity. The larger N is, the better the diversity of the selected sampled data is and the higher the time complexity is. The smaller N is, the lower the time complexity of selecting the sampled data is and the diversity of the sampled data will be sacrificed.

[0159] s704, set the maximum label entropy to 0, and set the data index corresponding to the maximum label entropy to -1;

[0160] s705, determine whether all data samples in the current filtered data set have been traversed, if so, execute s709, otherwise, execute s706;

[0161] s706, jointly calculating the label entropy of the traversed current data sample and all data samples in the current sampling data set, and using the calculation result as the entropy corresponding to the current data sample;

[0162] Among them, the processing of this step is to assume that the current data sample is added to the current sampling data set to obtain the updated current sampling data set, and calculate the label entropy of the updated current sampling data set, that is, based on the current data sample and the image labels and instruction labels of all data samples in the current sampling data set before the update, calculate the label entropy. Of course, adding the current sampling data set here is just an assumption. In fact, the current sampling data set remains unchanged before and after this step is executed.

[0163] s707, determine whether the entropy corresponding to the current data sample is greater than the maximum label entropy, if so, execute s708, otherwise return to s705 to determine whether the traversal is completed, and continue to traverse the next data sample of the current filtered data set if it is determined that the traversal is not completed;

[0164] s708, update the maximum label entropy to the entropy of the current data sample, and update the data index corresponding to the maximum label entropy to the index of the current data sample, return to s705 to determine whether the traversal is completed, and continue to traverse the next data sample of the current filtered data set if it is determined that the traversal is not completed;

[0165] s709, determine whether the maximum label entropy is greater than the label entropy of the current sampling data set, if so, execute s710, otherwise, return to s703;

[0166] Here, the label entropy of the current sampling data set refers to the label entropy calculated based on the image labels and instruction labels of all data samples in the current sampling data set;

[0167] s710, adding the data sample pointed to by the data index corresponding to the maximum label entropy to the current sampling data set, and continuing to execute s702.

[0168] The above implementation method is a sampling data selection method based on the maximum local label entropy. In which, the filtered data set is divided into multiple batch data groups, each batch data group constitutes the current filtered data set; in each batch data group, the data sample that maximizes the label entropy is selected, and on the premise that the data sample can increase the label entropy relative to the current sample data set, the corresponding data sample is added to the sample data set, thereby maximizing the local label entropy; this selection method can also well realize the diversity of the sampled data, and at the same time, can effectively reduce the time complexity.

[0169] Method 3:

[0170] In the third method of selecting sampling data, the specific process may include:

[0171] a. Set the current sampling data set to an empty set;

[0172] b. In the filtered data set, select a current data sample in turn, and jointly calculate the label entropy of the current data sample and all the data samples in the current sampling data set as the entropy corresponding to the current data sample; if the entropy corresponding to the current data sample is greater than the label entropy jointly calculated by all the data samples in the current sampling data set, then add the current data sample to the current sampling data set, and process the next selected data sample until the number of samples in the current sampling data set meets the requirement; if the entropy corresponding to the current data sample is not greater than the label entropy jointly calculated by all the data samples in the current sampling data set, then directly process the next selected data sample until the number of samples in the current sampling data set meets the requirement.

[0173] In practical applications, the implementation process of sampling data selection in the above method 3 can be as follows: Figure 8 As shown, specifically including:

[0174] s801, initialize the current sampling data set to an empty set;

[0175] Accordingly, the label entropy of the current sampled data set is 0;

[0176] s802, determine whether the sampled data reaches the set sample number, if so, end this process, otherwise, execute s803;

[0177] s803, selecting a data sample from the untraversed data samples in the filtered data set as the current data sample;

[0178] s804, jointly calculating the label entropy of the current data sample and all data samples in the current sampling data set, and using the calculation result as the entropy corresponding to the current data sample;

[0179] Among them, the processing of this step is to assume that the current data sample is added to the current sampling data set to obtain the updated current sampling data set, and calculate the label entropy of the updated current sampling data set, that is, based on the current data sample and the image labels and instruction labels of all data samples in the current sampling data set before the update, calculate the label entropy. Of course, adding the current sampling data set here is just an assumption. In fact, the current sampling data set remains unchanged before and after this step is executed.

[0180] s805, determine whether the entropy corresponding to the current data sample is greater than the label entropy of the current sampling data set. If so, execute s806, otherwise return to s803 to select the next data sample;

[0181] s806, adding the current data sample as sampling data to the current sampling data set, updating the label entropy of the current sampling data set, and returning to s802.

[0182] Method 3 is a processing method that sets N to 1 based on the above method 2. In this case, the sampling data selection method of method 3 degenerates into a greedy algorithm, and the specific logic becomes: traverse all filtered original multimodal training data, and for each data sample, if the addition of the data sample can increase the label entropy of the sampling data set, then add the data sample as sampling data to the sampling data set. This processing method minimizes the time complexity while ensuring that the sampling data has a certain diversity.

[0183] In practical applications, you can choose from the above three implementation methods according to different requirements for diversity and diversity sampling efficiency. Generally speaking, in order to balance the effect and efficiency, you can use the diversity sampling algorithm of local maximum label entropy in method 2.

[0184] As described above, the process of selecting sampling data based on the principle of maximizing label entropy is given.

[0185] Step 205: Use the selected sampled data as training data to train the multimodal large model.

[0186] When the sampled data is used as training data, the sampled data can be restored to the triple form of "image-instruction-answer", and the image label, instruction label and answer rating label can be omitted. The specific processing of using the selected training data for LMM training can be implemented in various existing ways, which will not be repeated here.

[0187] In this embodiment, a small-scale multimodal training data is screened out from the original multimodal training data, and the selected training data has better diversity and higher quality. The selected training data is used to train the LMM model, which can effectively ensure the performance of the LMM model while reducing training data and saving training time.

[0188] In order to verify the effectiveness of the above-mentioned diversity sampling data for LMM model training, 2 million data were selected as the original multimodal training data, and then the "10,000 training data with optimal diversity sampling" were sampled using the above-mentioned method of the present application. At the same time, as comparison data, "10,000 randomly sampled training data" were also randomly sampled. Then, the data sampling principle in the above-mentioned method of the present application was modified, and the sampling data was selected based on the method of minimizing label entropy, and the "10,000 training data with the worst diversity sampling" were selected. The three types of sampled data and the weighted mixed 2 million training data were used to train an LMM model with the same starting point, and the ability of the trained LMM was comprehensively measured by evaluating 15 evaluation sets. The results are shown in Table 5.

[0189] Table 5

[0190]

[0191] It should be noted that the original 2 million training data used an empirical weight ratio and was selected through multiple experiments. The 10,000 training data with the best diversity sampling was close to the effect of 2 million training data when only 0.5% of the data was used. The performance of the experimental control group with the worst diversity and random sampling as a control was significantly reduced, proving the beneficial effect of the LMM model training method in this application. Since the basis for sampling data selection in this application is image labels, instruction labels, and answer rating labels, the above experiments also indirectly indicate that the text labels are of sufficient quality.

[0192] Step 206: further train the label expert LMM based on the trained LMM.

[0193] The label expert LMM training in the aforementioned step 201 is based on the general LMM. In this step, a better LMM is obtained through training. The label expert LMM is further trained based on this LMM, which can effectively improve the effect of the label expert LMM.

[0194] In addition, through the processing of the aforementioned step 202, the labels of the original multimodal training data can be automatically obtained, so more training samples of the label expert LMM can be selected from these labeled data. The labels of these training samples do not need to be manually marked and can be automatically obtained, which greatly expands the training sample scale of the label expert LMM and balances the label distribution in the new label expert LMM training set as much as possible, so that the label expert LMM benefits from the training data with balanced label distribution, thereby effectively improving the performance of the label expert LMM. After obtaining a label expert LMM with better performance, you can return to step 202.

[0195] The above process of further training the label expert LMM with the trained LMM and expanding the label expert LMM training data can improve the label quality of the original multimodal training data and improve the quality of the final diversity sampled data. This step reflects the closed-loop iterative improvement of the label expert LMM, the general LMM and the high-quality multimodal training data, which benefit each other.

[0196] At this point, the method flow in this embodiment ends.

[0197] It can be seen from the description of the LMM model training method in the above application and the specific embodiments that the LMM model training method in the present application uses text labels (image labels, instruction labels, and answer rating labels) as carriers and the principle of maximizing label entropy as a means to obtain high-quality and diverse training data from original multimodal training data with complex sources, thereby improving the comprehensive ability of the general LMM trained using these data; at the same time, the labels obtained by the label expert LMM can expand the training data of the label expert LMM and improve the effect of the label expert LMM; in addition, better multimodal training data improves the effect of the general LMM, and as a pre-training model of the label expert LMM, it can also improve the effect of the label expert LMM. Both angles can iteratively improve the label expert LMM, thereby allowing the entire LMM model training system to continuously evolve and improve.

[0198] The above is a specific implementation of the LMM model training method in this application. The LMM model training method can be used in multiple fields, such as monitoring and other applications. The application also provides an LMM model training device that can be used to implement the above training method. Fig. 9 Schematic diagram of the basic structure of the LMM model training device in this application. Fig. 9 As shown, the training device includes: a label prediction unit, a data filtering unit, a data sampling unit and a multimodal large model training unit.

[0199] The label prediction unit is used to predict the label for each data sample in the original multimodal data using a pre-trained label expert multimodal large model LMM for label annotation; the labels here include image labels for marking image categories, instruction labels for marking instruction categories, and answer rating labels for marking answer quality;

[0200] A data filtering unit, used for filtering the original multimodal data based on the answer rating labels to obtain a filtered data set;

[0201] A data sampling unit, for selecting data samples from the filtered data set as sampling data based on image labels and instruction labels of all data samples in the filtered data set, so as to maximize label entropy determined based on image labels and instruction labels of all sampling data;

[0202] The multimodal large model training unit is used to train the multimodal large model using the sampled data as training samples.

[0203] Optionally, the apparatus may further include a label expert training unit for training and generating the label expert LMM in advance based on the general LMM using image label seed training data, instruction label seed training data, and answer rating label seed training data determined based on the original multimodal data;

[0204] The image label seed training data includes a picture selected from the original multimodal data, an instruction for instructing to predict an image label for the picture, and an answer for recording the image label;

[0205] The instruction label seed training data includes instructions selected from the original multimodal data, pictures corresponding to the selected instructions, and answers for recording instruction labels;

[0206] The answer rating label seed training data includes pictures and instructions in the data samples selected based on the answers from the original multimodal data, and the answers used to record the label ratings. The label ratings are the rating results corresponding to the answer labels to which the answers in the selected data samples belong.

[0207] Optionally, the label rating may be a label classification of each label to which the answer in the selected data sample belongs, and the label expert training unit is further used to determine a comprehensive label classification based on the label classification of each label; or, the answer rating label may be a comprehensive label classification determined based on the label classification of each label to which the answer in the selected data sample belongs;

[0208] The data filtering unit is used to filter out data samples whose comprehensive label classification does not meet the set requirements.

[0209] Optionally, in the data sampling unit, the process of selecting a data sample from the filtered data set as the sampling data may specifically include:

[0210] a. Set the filtered data set as the current filtered data set and set the current sampled data set to an empty set;

[0211] b. In the current filtering data set, for each current data sample, the label entropy is calculated jointly with the current data sample and all data samples in the current sampling data set as the entropy corresponding to the current data sample;

[0212] c. Determine the maximum entropy value corresponding to each data sample, add the data sample corresponding to the maximum entropy value to the current sampling data set, delete the corresponding current data sample from the current filtered data set, and return to step b until the number of samples in the current sampling data set meets the requirements.

[0213] Optionally, in the data sampling unit, the process of selecting a data sample from the filtered data set as the sampling data specifically includes:

[0214] a. Set the current sampling data set to an empty set;

[0215] b. Select N data samples from the untraversed data samples in the filtered data set as the current filtered data set;

[0216] c. In the current filtered data set, traverse each current data sample, and jointly calculate the label entropy of the current data sample and all the data samples in the current sampling data set as the entropy corresponding to the current data sample; determine the maximum value of the entropy corresponding to each data sample in the current filtered data set. If the maximum value is greater than the label entropy jointly calculated by all the data samples in the current sampling data set, add the current data sample corresponding to the maximum entropy value to the current sampling data set, and return to step b until the number of samples in the current sampling data set meets the requirement; otherwise, directly return to step b until the number of samples in the current sampling data set meets the requirement.

[0217] Optionally, in the data sampling unit, the process of selecting a data sample from the filtered data set as the sampling data may specifically include:

[0218] a. Set the current sampling data set to an empty set;

[0219] b. In the filtered data set, select a current data sample in turn, and jointly calculate the label entropy of the current data sample and all the data samples in the current sampling data set as the entropy corresponding to the current data sample; if the entropy corresponding to the current data sample is greater than the label entropy jointly calculated by all the data samples in the current sampling data set, add the current data sample to the current sampling data set, and process the next selected data sample until the number of samples in the current sampling data set meets the requirement; if the entropy corresponding to the current data sample is not greater than the label entropy jointly calculated by all the data samples in the current sampling data set, directly process the next selected data sample until the number of samples in the current sampling data set meets the requirement.

[0220] Optionally, the label expert training unit may also be used to further train the label expert LMM using the trained multimodal large model; and / or,

[0221] The label expert training unit can also be used to use the labels and corresponding data samples predicted by the label expert LMM for the data samples in the original multimodal data as training data to further train the label expert LMM.

[0222] The present application also provides a computer-readable storage medium that stores instructions, which, when executed by a processor, can execute the steps in the training method for implementing the LMM model as described above. In practical applications, the computer-readable medium can be included in each device / apparatus / system in the above-mentioned embodiments, or it can exist independently without being assembled into the device / apparatus / system. Instructions are stored in a computer-readable storage medium, and the instructions stored therein can execute the steps in the training method for the LMM model as described above when executed by a processor.

[0223] According to the embodiments disclosed in the present application, the computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, it may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof, but it is not used to limit the scope of protection of the present application. In the embodiments disclosed in the present application, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, apparatus, or device.

[0224] Fig.10 This application also provides an electronic device. Fig.10 As shown, it shows a schematic diagram of the structure of the electronic device involved in the embodiment of the present application, specifically:

[0225] The electronic device may include a processor 1001 with one or more processing cores, a memory 1002 with one or more computer-readable storage media, and a computer program stored in the memory and executable on the processor. When executing the program in the memory 1002, the LMM model training method may be implemented.

[0226] Specifically, in practical applications, the electronic device may further include components such as a power supply 1003, an input / output unit 1004, etc. Those skilled in the art may understand that Fig.10 The structure of the electronic device shown in the figure does not constitute a limitation on the electronic device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0227] Processor 1001 is the control center of the electronic device. It uses various interfaces and lines to connect various parts of the entire electronic device. It executes various functions of the server and processes data by running or executing software programs and / or modules stored in memory 1002 and calling data stored in memory 1002, thereby monitoring the electronic device as a whole.

[0228] The memory 1002 can be used to store software programs and modules, that is, the above-mentioned computer-readable storage medium. The processor 1001 executes various functional applications and data processing by running the software programs and modules stored in the memory 1002. The memory 1002 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function, etc.; the data storage area may store data created according to the use of the server, etc. In addition, the memory 1002 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. Accordingly, the memory 1002 may also include a memory controller to provide the processor 1001 with access to the memory 1002.

[0229] The electronic device also includes a power supply 1003 for supplying power to various components, which can be logically connected to the processor 1001 through a power management system, so as to manage charging, discharging, power consumption and other functions through the power management system. The power supply 1003 can also include any components such as one or more DC or AC power supplies, recharging systems, power failure detection circuits, power converters or inverters, and power status indicators.

[0230] The electronic device may also include an input and output unit 1004, which may be used to receive input digital or character information and generate keyboard, mouse, joystick, and optical signal inputs related to user settings and function control. The input unit output 1004 may also be used to display information input by the user or information provided to the user and various graphical user interfaces, which may be composed of graphics, text, icons, videos, and any combination thereof.

[0231] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for training a multimodal large model, characterized in that: include: Predicting labels for the original multimodal training data using a pre-trained label expert multimodal large model LMM for label annotation; wherein the labels include image labels for marking image categories, instruction labels for marking instruction categories, and answer rating labels for marking answer quality; Filtering the original multimodal training data based on the answer rating labels to obtain a filtered data set; Based on the image labels and instruction labels of all data samples in the filtered data set, selecting data samples from the filtered data set as sampled data, so as to maximize the label entropy determined based on the image labels and instruction labels of all sampled data; Using the sampled data as training samples to train a multimodal large model; The method further includes: pre-training and generating the label expert LMM based on the general LMM using image label seed training data, instruction label seed training data and answer rating label seed training data determined based on the original multimodal data; The image label seed training data includes a picture selected from the original multimodal training data, an instruction for instructing to predict an image label for the picture, and an answer for recording the image label; The instruction label seed training data includes instructions selected from the original multimodal training data, pictures corresponding to the selected instructions, and answers for recording instruction labels; The answer rating label seed training data includes pictures in the data sample selected based on the answer from the original multimodal training data, instructions for instructing to rate the answer, and answers for recording label ratings, wherein the label rating is a rating result corresponding to the answer label to which the answer in the selected data sample belongs; The label rating is the label grade of each label to which the answer in the selected data sample belongs, and the method further includes: determining a comprehensive label grade based on the label grades of each label; or, the answer rating label includes a comprehensive label grade determined based on the label grades of each label to which the answer in the selected data sample belongs.

2. The method according to claim 1, wherein: The filtering of the original multimodal data based on the answer rating labels includes: filtering out data samples whose comprehensive label classification does not meet the set requirements.

3. The method according to claim 1 or 2, characterized in that: The selecting a data sample from the filtered data set as sampling data comprises: a. Using the filtered data set as the current filtered data set and setting the current sampled data set to an empty set; b. For each current data sample in the current filtering data set, the label entropy is calculated jointly with the current data sample and all data samples in the current sampling data set as the entropy corresponding to the current data sample; c. Determine the maximum value of the entropy corresponding to each data sample, add the data sample corresponding to the maximum entropy value to the current sampling data set, delete the corresponding current data sample from the current filtered data set, and return to step b until the number of samples in the current sampling data set meets the requirements.

4. The method according to claim 1 or 2, characterized in that: The selecting a data sample from the filtered data set as sampling data comprises: a. Set the current sampling data set to an empty set; b. Selecting N data samples from the untraversed data samples in the filtering data set as the current filtering data set; N is a preset positive integer that is less than the total number of data samples in the filtering data set and greater than 1; c. In the current filtered data set, traverse each current data sample, and jointly calculate the label entropy of the current data sample and all the data samples in the current sampling data set as the entropy corresponding to the current data sample; determine the maximum value of the entropy corresponding to each data sample in the current filtered data set. If the maximum value is greater than the label entropy jointly calculated by all the data samples in the current sampling data set, add the current data sample corresponding to the maximum entropy value to the current sampling data set, and return to step b until the number of samples in the current sampling data set meets the requirement; otherwise, directly return to step b until the number of samples in the current sampling data set meets the requirement.

5. The method according to claim 1 or 2, characterized in that: The selecting a data sample from the filtered data set as sampling data comprises: a. Set the current sampling data set to an empty set; b. In the filtered data set, select a current data sample in turn, and jointly calculate the label entropy of the current data sample and all the data samples in the current sampling data set as the entropy corresponding to the current data sample; if the entropy corresponding to the current data sample is greater than the label entropy jointly calculated by all the data samples in the current sampling data set, add the current data sample to the current sampling data set, and process the next selected data sample until the number of samples in the current sampling data set meets the requirement; if the entropy corresponding to the current data sample is not greater than the label entropy jointly calculated by all the data samples in the current sampling data set, directly process the next selected data sample until the number of samples in the current sampling data set meets the requirement.

6. The method according to claim 1, characterized in that The method further comprises: using the trained multimodal large model to further train the label expert LMM; and / or, The labels and corresponding data samples predicted by the label expert LMM for the data samples in the original multimodal data are used as training data to further train the label expert LMM.

7. A multi-modal large model training device, characterized in that: The device comprises: a label expert training unit, a label prediction unit, a data filtering unit, a data sampling unit and a multi-modal large model training unit; The label prediction unit is used to predict a label for each data sample in the original multimodal data using a pre-trained label expert multimodal large model LMM for label annotation; wherein the label includes an image label for marking an image category, an instruction label for marking an instruction category, and an answer rating label for marking an answer quality; The data filtering unit is used to filter the original multimodal data based on the answer rating label to obtain a filtered data set; The data sampling unit is used to select data samples from the filtered data set as sampling data based on the image labels and instruction labels of all data samples in the filtered data set, so as to maximize the label entropy determined based on the image labels and instruction labels of all the sampling data; The multimodal large model training unit is used to train the multimodal large model using the sampled data as training samples; The label expert training unit is used to train and generate the label expert LMM in advance based on the general LMM using the image label seed training data, the instruction label seed training data and the answer rating label seed training data determined based on the original multimodal data; The image label seed training data includes a picture selected from the original multimodal data, an instruction for instructing to predict an image label for the picture, and an answer for recording the image label; The instruction label seed training data includes instructions selected from the original multimodal data, pictures corresponding to the selected instructions, and answers for recording instruction labels; The answer rating label seed training data includes pictures and instructions in the data sample selected based on the answer from the original multimodal data, and the answer for recording the label rating, wherein the label rating is the rating result corresponding to the answer label to which the answer in the selected data sample belongs; The label rating is the label grade of each label to which the answer in the selected data sample belongs, and the label expert training unit is further used to determine the comprehensive label grade based on the label grade of each label; or, the answer rating label is the comprehensive label grade determined based on the label grade of each label to which the answer in the selected data sample belongs.

8. The device according to claim 7, characterized in that The data filtering unit is used to filter out data samples whose comprehensive label classification does not meet the set requirements.

9. The device according to claim 7 or 8, characterized in that In the data sampling unit, selecting a data sample from the filtered data set as sampling data includes: a. Using the filtered data set as the current filtered data set and setting the current sampled data set to an empty set; b. In the current filtering data set, for each current data sample, the label entropy is calculated jointly with the current data sample and all data samples in the current sampling data set as the entropy corresponding to the current data sample; c. Determine the maximum entropy value corresponding to each data sample, add the data sample corresponding to the maximum entropy value to the current sampling data set, delete the corresponding current data sample from the current filtered data set, and return to step b until the number of samples in the current sampling data set meets the requirements.

10. The device according to claim 7 or 8, characterized in that In the data sampling unit, selecting a data sample from the filtered data set as sampling data includes: a. Set the current sampling data set to an empty set; b. Selecting N data samples from the untraversed data samples in the filtering data set as the current filtering data set; c. In the current filtered data set, traverse each current data sample, and jointly calculate the label entropy of the current data sample and all the data samples in the current sampling data set as the entropy corresponding to the current data sample; determine the maximum value of the entropy corresponding to each data sample in the current filtered data set. If the maximum value is greater than the label entropy jointly calculated by all the data samples in the current sampling data set, add the current data sample corresponding to the maximum entropy value to the current sampling data set, and return to step b until the number of samples in the current sampling data set meets the requirement; otherwise, directly return to step b until the number of samples in the current sampling data set meets the requirement.

11. The device according to claim 7 or 8, characterized in that In the data sampling unit, selecting a data sample from the filtered data set as sampling data includes: a. Set the current sampling data set to an empty set; b. In the filtered data set, select a current data sample in turn, and jointly calculate the label entropy of the current data sample and all the data samples in the current sampling data set as the entropy corresponding to the current data sample; if the entropy corresponding to the current data sample is greater than the label entropy jointly calculated by all the data samples in the current sampling data set, add the current data sample to the current sampling data set, and process the next selected data sample until the number of samples in the current sampling data set meets the requirement; if the entropy corresponding to the current data sample is not greater than the label entropy jointly calculated by all the data samples in the current sampling data set, directly process the next selected data sample until the number of samples in the current sampling data set meets the requirement.

12. The device according to claim 7, characterized in that The label expert training unit is used to further train the label expert LMM using the trained multimodal large model; and / or, The label expert training unit is used to use the labels and corresponding data samples predicted by the label expert LMM for the data samples in the original multimodal data as training data, so as to further train the label expert LMM.

13. A computer-readable storage medium having computer instructions stored thereon, characterized in that: When the instructions are executed by the processor, the multimodal large model training method described in any one of claims 1 to 6 can be implemented.

14. An electronic device, characterized in that: The electronic device includes at least a computer-readable storage medium and also includes a processor; The processor is used to read executable instructions from the computer-readable storage medium and execute the instructions to implement the multimodal large model training method described in any one of claims 1 to 6 above.

Images