Multi-modal large model training method and device, storage medium and electronic equipment

Abstract

The invention discloses a multi-modal large model training method and device, a storage medium and electronic equipment, and the training method comprises the steps: predicting a label for original multi-modal training data through employing a pre-trained label expert LMM for label labeling; filtering the original multi-modal training data based on the answer rating tag to obtain a filtered data set; based on the image tags and the instruction tags of all the data samples in the filtered data set, selecting the data samples from the filtered data set as sampling data, and maximizing tag entropy determined based on the image tags and the instruction tags of all the sampling data; and carrying out multi-modal large model training by taking the sampling data as a training sample. By applying the method, the screened training data can be utilized to train the multi-modal large model, and the performance of the multi-modal large model is effectively ensured.

Description

Technical Field

[0001] The present application relates to neural network technology, and in particular to a training method, storage medium, and electronic device for a multimodal large model. Background Art

[0002] Training large multimodal models (LMMs) requires multimodal question-and-answer data, typically in the form of "image-command-answer" triplets. To ensure the generalizability of large multimodal models, multimodal training data typically requires a large amount of data, meeting both diversity and high quality requirements.

[0003] Sources of multimodal training data include publicly available multimodal training datasets on the internet, synthesized or modified single-modality training datasets, data generated by powerful LMMs, and manually calibrated datasets, and they exist in various dataset forms. Although these multimodal training datasets come from diverse sources, directly mixing them for LMM training without filtering presents the following issues: 1. Poor image diversity: Different datasets frequently use images from specific scenes, such as everyday scenes and close-ups; 2. Poor instruction diversity: Different datasets have instruction preferences, and these datasets vary in data volume. Directly mixing datasets results in the instruction type in the larger dataset accounting for a larger proportion of the total data, leading to an imbalance in diversity. Therefore, further processing and filtering of multimodal datasets collected from various sources is essential. Unfiltered data is referred to as raw multimodal training data to distinguish it from filtered data actually used for training. Existing methods of filtering raw multimodal training data have been found to affect LMM model performance when the filtered training data is used for LMM training, leading to poor model performance. Summary of the Invention

[0004] The present application provides a training method, device, storage medium and electronic device for a multimodal large model, which can use screened training data to train the multimodal large model and effectively ensure the performance of the multimodal large model.

[0005] To achieve the above objectives, this application adopts the following technical solutions: A method for training a large multimodal model, comprising: Predicting labels for the original multimodal training data using a pre-trained label expert multimodal large model (LMM) for labeling; wherein the labels include image labels for marking image categories, instruction labels for marking instruction categories, and answer rating labels for marking answer quality; Filtering the original multimodal training data based on the answer rating labels to obtain a filtered dataset; Based on the image labels and instruction labels of all data samples in the filtered data set, selecting data samples from the filtered data set as sampled data, so as to maximize the label entropy determined based on the image labels and instruction labels of all the sampled data; The sampled data is used as training samples to train a large multimodal model.

[0006] Preferably, the method further comprises: pre-training and generating the label expert LMM based on a general LMM using image label seed training data, instruction label seed training data, and answer rating label seed training data determined based on the original multimodal data; The image label seed training data includes pictures selected from the original multimodal training data, instructions for instructing image label prediction for the pictures, and answers for recording image labels; The instruction label seed training data includes instructions selected from the original multimodal training data, pictures corresponding to the selected instructions, and answers for recording instruction labels; The answer rating label seed training data includes pictures from the data samples selected based on the answers from the original multimodal training data, instructions for instructing answer rating, and answers for recording label ratings, where the label rating is the rating result corresponding to the answer label to which the answer in the selected data sample belongs.

[0007] Preferably, the label rating is a label ranking of each label to which the answer in the selected data sample belongs, and the method further comprises: determining a comprehensive label ranking based on the label ranking of each label; or, the answer rating label comprises a comprehensive label ranking determined based on the label ranking of each label to which the answer in the selected data sample belongs; The filtering of the original multimodal data based on the answer rating labels includes filtering out data samples whose comprehensive label classification does not meet the set requirements.

[0008] Preferably, the selecting a data sample from the filtered data set as the sampling data includes: a. Using the filtered dataset as the current filtered dataset and setting the current sampled dataset to an empty set; b. For each current data sample in the current filtered data set, the label entropy of the current data sample and all data samples in the current sampled data set are jointly calculated as the entropy corresponding to the current data sample; c. Determine the maximum entropy value corresponding to each data sample, add the data sample corresponding to the maximum entropy value to the current sampling data set, and delete the corresponding current data sample from the current filtered data set, and return to step b until the number of samples in the current sampling data set meets the requirement.

[0009] Preferably, the selecting a data sample from the filtered data set as the sampling data includes: a. Set the current sampling data set to an empty set; b. Selecting N data samples from the untraversed data samples in the filtering data set as the current filtering data set; N is a preset positive integer that is less than the total number of data samples in the filtering data set and greater than 1; c. In the current filtered data set, traverse each current data sample, and jointly calculate the label entropy of the current data sample and all data samples in the current sampling data set as the entropy corresponding to the current data sample; determine the maximum value of the entropy corresponding to each data sample in the current filtered data set; if the maximum value is greater than the label entropy jointly calculated by all data samples in the current sampling data set, then add the current data sample corresponding to the maximum entropy to the current sampling data set, and return to step b until the number of samples in the current sampling data set meets the requirement; otherwise, directly return to step b until the number of samples in the current sampling data set meets the requirement.

[0010] Preferably, the selecting a data sample from the filtered data set as the sampling data includes: a. Set the current sampling data set to an empty set; b. In the filtered data set, select a current data sample in turn, and jointly calculate the label entropy of the current data sample and all the data samples in the current sampling data set as the entropy corresponding to the current data sample; if the entropy corresponding to the current data sample is greater than the label entropy jointly calculated by all the data samples in the current sampling data set, add the current data sample to the current sampling data set, and process the next selected data sample until the number of samples in the current sampling data set meets the requirement; if the entropy corresponding to the current data sample is not greater than the label entropy jointly calculated by all the data samples in the current sampling data set, directly process the next selected data sample until the number of samples in the current sampling data set meets the requirement.

[0011] Preferably, the method further comprises: using the trained multimodal model to further train the label expert LMM; and / or, The labels and corresponding data samples predicted by the label expert LMM for the data samples in the original multimodal data are used as training data to further train the label expert LMM.

[0012] A multimodal model training device, comprising: a label prediction unit, a data filtering unit, a data sampling unit, and a multimodal model training unit; The label prediction unit is configured to predict a label for each data sample in the original multimodal data using a pre-trained label expert multimodal large model LMM for label annotation; wherein the label includes an image label for marking the image category, an instruction label for marking the instruction category, and an answer rating label for marking the answer quality; The data filtering unit is configured to filter the original multimodal data based on the answer rating labels to obtain a filtered data set; The data sampling unit is configured to select data samples from the filtered data set as sampling data based on the image labels and instruction labels of all data samples in the filtered data set, so as to maximize the label entropy determined based on the image labels and instruction labels of all the sampling data; The multimodal model training unit is used to train the multimodal model using the sampled data as training samples.

[0013] Preferably, the apparatus further comprises a label expert training unit for pre-training and generating the label expert LMM based on a general LMM and using image label seed training data, instruction label seed training data, and answer rating label seed training data determined based on the original multimodal data; The image label seed training data includes pictures selected from the original multimodal data, instructions for instructing image label prediction for the pictures, and answers for recording image labels; The instruction label seed training data includes instructions selected from the original multimodal data, pictures corresponding to the selected instructions, and answers for recording instruction labels; The answer rating label seed training data includes pictures and instructions in the data samples selected based on the answers from the original multimodal data, and answers for recording label ratings, where the label ratings are rating results corresponding to the answer labels to which the answers in the selected data samples belong.

[0014] Preferably, the label rating is a label classification of each label to which the answer in the selected data sample belongs, and the label expert training unit is further used to determine a comprehensive label classification based on the label classification of each label; or, the answer rating label is a comprehensive label classification determined based on the label classification of each label to which the answer in the selected data sample belongs; The data filtering unit is used to filter out data samples whose comprehensive label classification does not meet the set requirements.

[0015] Preferably, in the data sampling unit, selecting a data sample from the filtered data set as the sampling data includes: a. Using the filtered dataset as the current filtered dataset and setting the current sampled dataset to an empty set; b. For each current data sample in the current filtered data set, the label entropy of the current data sample and all data samples in the current sampled data set are jointly calculated as the entropy corresponding to the current data sample; c. Determine the maximum entropy value corresponding to each data sample, add the data sample corresponding to the maximum entropy value to the current sampling data set, and delete the corresponding current data sample from the current filtered data set, and return to step b until the number of samples in the current sampling data set meets the requirement.

[0016] Preferably, in the data sampling unit, selecting a data sample from the filtered data set as the sampling data includes: a. Set the current sampling data set to an empty set; b. Selecting N data samples from the untraversed data samples in the filtering data set as the current filtering data set; c. In the current filtered data set, traverse each current data sample, and jointly calculate the label entropy of the current data sample and all data samples in the current sampling data set as the entropy corresponding to the current data sample; determine the maximum value of the entropy corresponding to each data sample in the current filtered data set; if the maximum value is greater than the label entropy jointly calculated by all data samples in the current sampling data set, then add the current data sample corresponding to the maximum entropy to the current sampling data set, and return to step b until the number of samples in the current sampling data set meets the requirement; otherwise, directly return to step b until the number of samples in the current sampling data set meets the requirement.

[0017] Preferably, in the data sampling unit, selecting a data sample from the filtered data set as the sampling data includes: a. Set the current sampling data set to an empty set; b. In the filtered data set, select a current data sample in turn, and jointly calculate the label entropy of the current data sample and all the data samples in the current sampling data set as the entropy corresponding to the current data sample; if the entropy corresponding to the current data sample is greater than the label entropy jointly calculated by all the data samples in the current sampling data set, add the current data sample to the current sampling data set, and process the next selected data sample until the number of samples in the current sampling data set meets the requirement; if the entropy corresponding to the current data sample is not greater than the label entropy jointly calculated by all the data samples in the current sampling data set, directly process the next selected data sample until the number of samples in the current sampling data set meets the requirement.

[0018] Preferably, the label expert training unit is used to further train the label expert LMM using the trained multimodal model; and / or, The label expert training unit is used to use the labels and corresponding data samples predicted by the label expert LMM for the data samples in the original multimodal data as training data, so as to further train the label expert LMM.

[0019] A computer-readable storage medium having computer instructions stored thereon, which, when executed by a processor, can implement any of the multimodal model training methods described above.

[0020] An electronic device comprising at least a computer-readable storage medium and a processor; The processor is used to read executable instructions from the computer-readable storage medium and execute the instructions to implement any of the multimodal model training methods described above.

[0021] As can be seen from the above technical solution, in this application, a pre-trained label expert LMM for label annotation is first used to predict labels for each data sample in the original multimodal training data to obtain image labels, instruction labels, and answer rating labels, thereby adding three new dimensions of information to each data sample, which are used to characterize the image category, instruction category, and answer quality of the data sample respectively; next, the original multimodal data is filtered based on the answer rating labels of each data sample to obtain a filtered data set. In this process, since the answer rating labels can characterize the answer quality of the data samples, the corresponding filtering process can filter out data samples in the original multimodal training data that do not meet the requirements, thereby improving the quality of the training data of the multimodal model and further improving the performance of the trained multimodal model; next, based on the image labels and instruction labels of all data samples in the filtered data set, data samples are selected from the filtered data set as sampling data. The principle of data selection is to maximize the label entropy determined based on the image labels and instruction labels of all sampled data. Since the image labels and instruction labels respectively characterize the image category and instruction category of the data samples, the diversity of the sampled data can be achieved based on the principle of maximizing label entropy; finally, the sampled data is used to train the multimodal large model. On the one hand, low-quality training data is filtered out through answer rating labels, and on the other hand, more diverse data is selected for model training through image labels and instruction labels. Therefore, the data used to train multimodal large models can meet the requirements of diversity and high quality at the same time. Training based on such data can effectively guarantee the performance of multimodal large models. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 Schematic diagram of the basic process of the LMM model training method in this application; Figure 2 This is a schematic diagram of the specific process of the LMM model training method in the specific embodiment of this application; Figure 3 This is an example diagram of the image label space in the specific embodiment of this application; Figure 4 This is an example diagram of the instruction label space in a specific embodiment of this application; Figure 5 This is an example diagram of the answer label space in a specific embodiment of this application; Figure 6 This is an example diagram of the implementation process of selecting sampled data through method 1 in a specific embodiment of the present application; Figure 7 This is an example diagram of the implementation process of selecting sampled data by using the second method in a specific embodiment of the present application; Figure 8 This is an example diagram of the implementation process of selecting sampling data through method three in a specific embodiment of this application; Figure 9 Schematic diagram of the basic structure of the LMM model training device in this application; Figure 10 This is a schematic diagram of the basic structure of the electronic device provided in this application. DETAILED DESCRIPTION

[0023] In order to make the purpose, technical means and advantages of this application more clear, this application is further described in detail below with reference to the accompanying drawings.

[0024] The training data for large multimodal models typically takes the form of "image-command-answer" triplets, as shown in Table 1. Since training data includes datasets of varying types and sizes, directly mixing these datasets for training LMMs without filtering will result in the following issues: 1. Poor image diversity: Different datasets frequently use images of specific scenes, such as everyday scenes and close-ups; 2. Poor command diversity: Different datasets have different command preferences, and the amount of data varies. Directly mixing datasets will cause the command types in the larger dataset to account for a larger proportion of the total data, leading to an imbalance in diversity. For example, the Dm dataset consists of multiple-choice questions with a dataset size of 1 million, while the Dn dataset consists of various open-ended questions with a dataset size of only 1,000. Directly mixing the Dm and Dn datasets will result in the trained LMM performing well on multiple-choice questions but poorly on open-ended questions, resulting in a lack of generalizability. Therefore, further processing and filtering of multimodal datasets collected from various sources is necessary.

[0025] Table 1

[0026] The present application provides a training method for a large multimodal model. The original multimodal training data is screened for data quality based on the accuracy of the answer, and then the data sample diversity is screened based on the image category and instruction category, so as to select high-quality and more balanced training data for training the large multimodal model, which can effectively ensure the performance of the trained large model.

[0027] Figure 1 This is a basic flow chart of the multimodal large model training method in this application. Figure 1 As shown, the method includes: Step 101 : Use a pre-trained label expert LMM for label annotation to predict labels for data samples in the original multimodal training data.

[0028] Among them, the labels include image labels for marking image categories, instruction labels for marking instruction categories, and answer rating labels for marking answer quality.

[0029] The input data of the label expert LMM comes from the original multimodal training data. As mentioned above, the multimodal training data includes "image-instruction-answer". The label expert LMM gives image labels for the images in the input data, which are used to mark the image categories, gives instruction labels for the instructions in the input data, which are used to mark the instruction categories, and gives answer rating labels for the answers in the input data, which are used to mark the answer quality.

[0030] The label expert LMM is obtained through pre-training, and can be obtained through fine-tuning based on the general LMM model.

[0031] Step 102: Filter the original multimodal training data based on the answer rating labels to obtain a filtered data set.

[0032] As mentioned above, the answer rating label is used to characterize the quality of the answer, which can also characterize the quality of the corresponding training data. In LMM model training, if the quality of the training data is low, it will affect the accuracy of the model training. Therefore, in this application, the training data with lower quality in the original multimodal training data is filtered out, leaving the training data with the required quality. Among them, the answer rating label is used to measure the quality of the training data. Generally, the original multimodal training data whose answer rating represented by the answer rating label is lower than the standard rating can be filtered out. The specific standard rating can be set according to actual experience. The original multimodal training data that has not been filtered out is used to form a filtered data set.

[0033] Step 103 : Based on the image labels and instruction labels of all data samples in the filtered data set, data samples are selected from the filtered data set as sampling data, so as to maximize the label entropy determined based on the image labels and instruction labels of all the sampling data.

[0034] As mentioned above, image labels and instruction labels are used to mark the image category and instruction category of the training data, respectively. In order to make the screened training data meet the diversity requirements, the training data is screened based on the image category and instruction category of the training data in this application, that is, the training data is screened based on the image label and instruction label of the training data. During the screening, the label entropy is calculated based on the image label and instruction label of all the sampled data, and the sampled data is screened out using the label entropy maximization principle. Among them, the sampled data is the data sample selected from the filtered data set for LMM model training.

[0035] Step 104: Use the sampled data as training samples to train the multimodal large model LMM.

[0036] The LMM is trained using the sampled data obtained after screening in steps 102 and 103 as training samples. The specific process of training the LMM model using the sampled data can be performed using existing methods. Since step 102 filters out data samples that meet quality requirements, and step 103 further filters out a set of data samples that meet diversity requirements, the resulting sampled data meets both quality and diversity requirements, effectively ensuring the performance of the trained LMM model.

[0037] At this point, the multimodal large model training method process in this application is completed. Through the above processing of this application, on the one hand, the label expert LMM is used to label the original multimodal training data with image labels, instruction labels, and answer rating labels. On the other hand, the training data is screened based on the three types of labeled labels to find sampling data that meets the quality and diversity requirements for training the LMM model, thereby effectively ensuring the performance of the LMM model.

[0038] The specific implementation of the multimodal large model training method of this application is explained below through specific examples. Figure 2 This is a schematic diagram of the specific process of the multimodal large model training method in the embodiment of this application. Figure 2 As shown, the method includes: Step 201 : Based on the general LMM, a label expert LMM is trained and generated using image label seed training data, instruction label seed training data, and answer rating label seed training data determined based on the original multimodal data.

[0039] In this application, a label expert LMM is required to predict labels for the original multimodal training data. The label expert LMM used is obtained through pre-training. Optionally, the label expert LMM can be obtained through further training based on the general LMM.

[0040] When training the label expert LMM, image label seed training data, instruction label seed training data, and answer rating label seed training data are determined based on the original multimodal training data and used as training data for the label expert LMM.

[0041] Among them, the image label seed training data is mainly used to train the label expert LMM's ability to predict image labels, the instruction label seed training data is mainly used to train the label expert LMM's ability to predict instruction labels, and the answer rating label seed training data is mainly used to train the label expert LMM's ability to predict answer rating labels.

[0042] When determining image label seed training data based on original multimodal training data, the specific processing may include: First, define the label space of the image label; the image label space can be defined in a tree structure as follows Figure 3 As shown, this label space is used as label candidates when annotating images in the original multimodal training data; Second, representative images are selected from the original multimodal training data. These images can belong to one or more labels in the label space, and the selected image set has a certain degree of diversity in the label level. Then, receive the image labels corresponding to the image content, which can be one or more. Finally, organize the image and its corresponding image label into a triplet of "image-command-answer" to obtain image label seed training data. Here, "image" refers to the selected image, "command" can be the content corresponding to the "command" column in Table 2, which instructs the image label to be marked, and "answer" refers to the image label determined for the image. The command can be the same for different image label seed training data.

[0043] Table 2

[0044] For each selected picture, the corresponding image label seed training data is obtained according to the above method, which is used to train the image label prediction ability of the label expert LMM.

[0045] When determining instruction label seed training data based on original multimodal training data, specific processing may include: First, define the label space of the instruction label; the image label space can be defined in a tree structure as follows Figure 5 As shown in Figure 2, this label space is used as a label candidate for annotating instructions in the original multimodal training data; the setting of the label space can be extended based on the application field. For example, in the field of monitoring, it can be set in the image label space. Figure 4 The label category of the overlay information shown; Second, representative instructions are selected from the original multimodal training data. These instructions can belong to one or more labels in the label space, and the selected instruction set has a certain degree of diversity in the label level. Then, receive the instruction tags that match the instruction content and are marked on these instructions. The number of tags can be one or more. Finally, the image corresponding to the instruction, the instruction, and the instruction label corresponding to the instruction are organized into an "image-instruction-answer" triplet to obtain instruction label seed training data. Here, the image refers to the image corresponding to the selected instruction, the instruction refers to the selected instruction, and may also include extended information to indicate the marking of the instruction label (for example, see the instruction column in Table 3 below, which may include the major categories of instruction labels, all detailed labels under each major category, and the specific selected instruction content itself). The answer refers to the instruction label determined for the selected instruction. It is worth noting that since the instruction label is related to the image, it is impossible to give the correct instruction label without providing the image. Therefore, the corresponding image must be included when organizing the instruction label.

[0046] Table 3

[0047] For each selected instruction, the corresponding instruction label seed training data is obtained in the above manner to train the instruction label prediction capability of the label expert LMM.

[0048] When determining answer rating label seed training data based on original multimodal training data, the specific processing may include: First, define the label space of the answer label; the answer label space can be defined in a tree structure as follows Figure 5 As shown, this label space is used as the label candidates and corresponding rating results when annotating the answers in the original multimodal training data; specifically, in order to characterize the quality of the answer, in addition to giving various label candidates, the label space also gives the rating results corresponding to each label candidate to indicate the quality of the answer corresponding to the corresponding label candidate; in this embodiment, Figure 5The rating results given in the answer are expressed as scores, but can also be expressed in other ways, such as letter grades;); In addition, different categories of answer labels can be arranged in parallel or in a progressive manner. When an answer is scored 0 in a certain evaluation item, all subsequent scores are 0. For example, in this embodiment, the four label categories of "relevance", "formatting style", "language expression", and "credibility" are not arranged in parallel, but in a progressive manner; Secondly, representative multimodal data triplets are selected from the original multimodal training data. The answers in these data triplets can belong to one or more answer labels in the label space, and the answers in the selected data triplets have a certain degree of diversity in the answer label level. Then, answer labels corresponding to the answer content and corresponding rating results are received. The number of labels can be one or more. Optionally, on the basis of giving the rating results of each different category label, a comprehensive rating result obtained based on the rating results of each different category label can be further given. Compared with the method of giving a comprehensive rating result alone without giving the rating results of each category label, the method of giving both rating results uses a thinking chain method, which increases the reasoning and thinking process of the model and can effectively optimize the training results of the model. Finally, the image in the selected multimodal data triple, the instruction for indicating the answer label rating, and the answer for recording the label rating are organized into an "image-instruction-answer" triplet to obtain an answer rating label seed training data; wherein, the image refers to the picture in the selected data triplet, the instruction refers to the mark for indicating the answer rating label (the specific content can be seen in the instruction column in Table 4 below), and the answer refers to the rating label determined for the given answer label, which can be the rating result corresponding to the answer label under each answer label category, or it can further include a comprehensive rating result based on the rating results of the above-mentioned various answer labels.

[0049] Table 4

[0050] For each selected multimodal data triple, the corresponding answer rating label seed training data is obtained in the above manner to train the answer rating label prediction ability of the label expert LMM.

[0051] The selection and labeling of the above-mentioned image label training data can be manually selected or labeled. Because the general LMM has strong comprehensive capabilities, in order to train the label expert LMM, only a small part (a few hundred images) of the training data of the three labels of image, instruction, and answer is needed, namely the above-mentioned image label seed training data, instruction label seed training data, and answer rating label seed training data. The cost of manual selection and labeling is not high.

[0052] Through the processing of this step, the label expert LMM can be obtained by training the general LMM using the above training data.

[0053] Step 202: Input the original multimodal training data into the label expert LMM for label prediction.

[0054] As mentioned above, each piece of data in the original multimodal training data is a triplet of "image-command-answer". In this step, the triplet will be expanded into a sextuple of "image-command-answer-image label-command label-answer label" through the processing of the label expert LMM. The newly added image label, command label and answer rating label will be used for subsequent quality filtering and diversity sampling of the original multimodal training data. For example, the original multimodal training data and its label prediction results can be represented as DT i ={(M j , Q j , A j , MT j , QT j , AT j )} j =1:n i , where MT, QT, and AT represent the image label, instruction label, and answer rating label, respectively. Optionally, the answer rating label can be a comprehensive evaluation result of the rating results of the answer labels of each category, or it can also be the rating result of the answer labels of each category.

[0055] The label expert LMM obtained in step 201 is fine-tuned based on a general LMM and trained on a mixture of three types of labeled data. Therefore, a single label expert LMM is capable of predicting the corresponding label given a relevant label instruction, allowing a single model to simultaneously predict all three labels. Using the label expert LMM, each piece of data in the original multimodal training data is traversed, and the three labels are generated for each piece of data using the image label instructions in Table 2, the instruction label instructions in Table 3, and the answer rating label instructions in Table 4, respectively, to complete the aforementioned six-tuple for future use.

[0056] In step 203 , based on the answer rating labels predicted in step 202 , data samples whose instructions do not meet the requirements are filtered out from the original multimodal training data.

[0057] There are two situations in the answer rating labels predicted in step 202: 1. The given answer rating labels only have the rating results of each label category, and no comprehensive rating results; 2. The given answer rating labels include the rating results of each label category and the comprehensive rating results.

[0058] Accordingly, in this step, for the first case, it is necessary to give a comprehensive rating result according to the rating results of each label category in accordance with the set rules. The specific processing of obtaining the comprehensive rating result based on the individual rating results can be set according to actual needs; for the second case, the comprehensive rating result can be found directly in the answer rating label.

[0059] Filtering is performed based on the comprehensive rating results of the original multimodal training data to select data samples that meet the quality requirements. Specifically, a comprehensive rating result threshold can be set. Data samples above the corresponding threshold are retained, and the rest are filtered out. The retained data samples form a filtered data set. The comprehensive rating result threshold in the swamp can be flexibly set based on the level of data quality requirements. For example, in this embodiment, the comprehensive rating result is the sum of the individual rating results, and the comprehensive rating result threshold is set to 4.

[0060] Step 204 : In the filtered data set, data samples are selected as sampled data based on the image labels and the instruction labels, so as to maximize the label entropy determined based on the image labels and the instruction labels of all the sampled data.

[0061] In this embodiment, data samples are selected based on diversity sampling that maximizes label entropy. Specifically, a data sample is selected from the filtered dataset DT' such that, when added to the actual multimodal training data (i.e., the sampled data obtained after sampling), the label entropy based on all the sampled data after sampling is maximized.

[0062] In this embodiment, the label entropy may be calculated, for example, as follows: Label entropy = Image label entropy + Instruction label entropy =

[0063] in, n MT is the total number of image label types in all sampled data obtained after sampling, p i It is i The frequency of image labels appearing in all image labels, p i = Image Tag i The number of / total number of all image labels, here, image label iWhen calculating the number of, the image tag i is counted plus 1 each time it appears, and is counted as multiple times. The total number of all image tags refers to the total number of times all image tags appear. The image tag appears once and is counted plus 1, and is counted as multiple times. Similarly, n QT is the number of instruction labels in the sampled data, q j It is j The frequency of this instruction tag appearing in all instruction tags, q i = instruction label j The total number of all instruction tags.

[0064] Regarding the specific processing of sampling data selection based on the principle of maximizing label entropy, this embodiment provides the following exemplary implementation methods.

[0065] Method 1: In the first method of selecting sampling data, the specific process may include: a. Set the filtered dataset as the current filtered dataset and set the current sampled dataset to an empty set; b. In the current filtered data set, for each current data sample, the label entropy of the current data sample and all data samples in the current sampling data set are jointly calculated as the entropy corresponding to the current data sample; c. Determine the maximum entropy value corresponding to each data sample, add the current data sample corresponding to the maximum entropy value to the current sampling data set, and delete the corresponding current data sample from the current filtered data set, and return to step b until the number of samples in the current sampling data set meets the requirements.

[0066] In practical applications, the implementation process of the sampling data selection of the above method 1 can be as follows: Figure 6 As shown, specifically including: s601, initialize the current sampling data set to an empty set, and initialize the current filtering data set to a filtered data set; Accordingly, the label entropy of the current sampled dataset is 0; s602, determine whether the sampled data reaches the set number of samples, if so, end this process, otherwise, execute s603; s603, set the maximum label entropy to 0, and set the data index corresponding to the maximum label entropy to -1; s604, determine whether all data samples in the current filtered data set have been traversed, if so, execute s608, otherwise execute s605; s605, jointly calculating the label entropy of the traversed current data sample and all data samples in the current sampling data set, and using the calculation result as the entropy corresponding to the current data sample; This step assumes that the current data sample is added to the current sampling dataset to obtain an updated current sampling dataset, and then calculates the label entropy of the updated current sampling dataset. This is done based on the image and instruction labels of the current data sample and all the data samples in the current sampling dataset before the update. Of course, the addition of the current sampling dataset is merely an assumption; in reality, the current sampling dataset remains unchanged before and after this step.

[0067] s606, determine whether the entropy corresponding to the current data sample is greater than the maximum label entropy. If so, execute s607, otherwise return to s604 to determine whether the traversal is completed, and continue to traverse the next data sample if it is determined that the traversal is not completed; s607, updating the maximum label entropy to the entropy corresponding to the current data sample, and updating the data index corresponding to the maximum label entropy to the index of the current data sample, and then returning to s604 to determine whether the traversal is completed. If it is determined that the traversal is not completed, continue to traverse the next data sample; s608, adding the data sample pointed to by the data index corresponding to the maximum label entropy to the current sampling data set, and deleting it from the current filtering data set, and continuing to execute s602.

[0068] The above implementation method is based on the sampling data selection method with the maximum global label entropy. In each round, the currently unselected data samples are combined into the current filtered data set. At the same time, the label entropy of all data samples in the current filtered data set after being added to the sampling data set is calculated one by one, and the data sample corresponding to the maximum label entropy is selected to be added to the sampling data set; that is, in each round of selection, the data sample with the largest entropy is selected from all the currently unselected data samples as the sampling data and added to the sampling data set, thereby maximizing the global label entropy. This selection method is theoretically the best approach for result diversity and can ensure that the label entropy of the sampled data is the largest. However, from the perspective of algorithm complexity, its time complexity is relatively high. Assuming that the amount of data of the original multimodal training data DT' after filtering is N and the amount of data of the sampling data set is M, the time complexity of the above selection method is O(MN).

[0069] The following two selection methods strike a balance between diversity selection and time complexity, effectively reducing time complexity while ensuring better diversity selection.

[0070] Method 2: In the second method of selecting sampling data, the specific process may include: a. Set the current sampling data set to an empty set; b. Select N data samples from the untraversed data samples in the filtered data set as the current filtered data set; c. In the current filtered data set, traverse each current data sample, and jointly calculate the label entropy of the current data sample and all data samples in the current sampling data set as the entropy corresponding to the current data sample; determine the maximum entropy corresponding to each data sample in the current filtered data set. If the maximum entropy is greater than the label entropy jointly calculated by all data samples in the current sampling data set, then add the current data sample corresponding to the maximum entropy to the current sampling data set, and return to step b until the number of samples in the current sampling data set meets the requirement; otherwise, directly return to step b until the number of samples in the current sampling data set meets the requirement.

[0071] In practical applications, the implementation process of the sampling data selection of the above method 2 can be as follows: Figure 7 As shown, specifically including: s701, initialize the current sampling data set to an empty set; Accordingly, the label entropy of the current sampled dataset is 0; s702, determine whether the sampled data reaches the set number of samples, if so, end this process, otherwise, execute s703; The set number of samples refers to the total number of sampled data; s703, sequentially selecting N data samples from the untraversed data samples in the filtered data set as the current filtered data set; N is a preset positive integer that is smaller than the total number of data samples in the filtered data set and greater than 1. It can be determined based on the requirements for data diversity and the tolerance for time complexity. The larger N is, the better the diversity of the selected sample data and the higher the time complexity. The smaller N is, the lower the time complexity of selecting the sample data, but the diversity of the sample data will be sacrificed. s704, set the maximum label entropy to 0, and set the data index corresponding to the maximum label entropy to -1; s705, determine whether all data samples in the current filtered data set have been traversed, if so, execute s709, otherwise execute s706; s706, jointly calculating the label entropy of the traversed current data sample and all data samples in the current sampling data set, and using the calculation result as the entropy corresponding to the current data sample; This step assumes that the current data sample is added to the current sampling dataset to obtain an updated current sampling dataset, and then calculates the label entropy of the updated current sampling dataset. This is done based on the image and instruction labels of the current data sample and all the data samples in the current sampling dataset before the update. Of course, the addition of the current sampling dataset is merely an assumption; in reality, the current sampling dataset remains unchanged before and after this step.

[0072] s707, determine whether the entropy corresponding to the current data sample is greater than the maximum label entropy. If so, execute s708, otherwise return to s705 to determine whether the traversal is completed, and continue to traverse the next data sample of the current filtered data set if it is determined that the traversal is not completed; s708, update the maximum label entropy to the entropy of the current data sample, and update the data index corresponding to the maximum label entropy to the index of the current data sample, return to s705 to determine whether the traversal is completed, and continue to traverse the next data sample of the current filtered data set if it is determined that the traversal is not completed; s709, determine whether the maximum label entropy is greater than the label entropy of the current sample data set. If so, execute s710, otherwise, return to s703; Here, the label entropy of the current sampling data set refers to the label entropy calculated based on the image labels and instruction labels of all data samples in the current sampling data set; s710, adding the data sample pointed to by the data index corresponding to the maximum label entropy to the current sampling data set, and continuing to execute s702.

[0073] The above implementation method selects sample data based on the maximum local label entropy. The filtered dataset is divided into multiple batches, each of which constitutes the current filtered dataset. Within each batch, a data sample that maximizes label entropy is selected. If this data sample increases the label entropy relative to the current sampled dataset, the corresponding data sample is added to the sampled dataset, thereby maximizing the local label entropy. This selection method also effectively achieves diversity in the sampled data while effectively reducing time complexity.

[0074] Method 3: In the third method of selecting sampling data, the specific process may include: a. Set the current sampling data set to an empty set; b. In the filtered data set, select a current data sample in turn, and jointly calculate the label entropy of the current data sample and all the data samples in the current sampling data set as the entropy corresponding to the current data sample; if the entropy corresponding to the current data sample is greater than the label entropy jointly calculated by all the data samples in the current sampling data set, then add the current data sample to the current sampling data set and process the next selected data sample until the number of samples in the current sampling data set meets the requirement; if the entropy corresponding to the current data sample is not greater than the label entropy jointly calculated by all the data samples in the current sampling data set, then directly process the next selected data sample until the number of samples in the current sampling data set meets the requirement.

[0075] In practical applications, the implementation process of the sampling data selection of the above method 3 can be as follows: Figure 8 As shown, specifically including: s801, initialize the current sampling data set to an empty set; Accordingly, the label entropy of the current sampled dataset is 0; s802, determine whether the sampled data reaches the set number of samples. If so, end this process; otherwise, execute s803; s803, selecting a data sample from the untraversed data samples in the filtered data set as the current data sample; s804, jointly calculating the label entropy of the current data sample and all data samples in the current sampling data set, and using the calculation result as the entropy corresponding to the current data sample; This step assumes that the current data sample is added to the current sampling dataset to obtain an updated current sampling dataset, and then calculates the label entropy of the updated current sampling dataset. This is done based on the image and instruction labels of the current data sample and all the data samples in the current sampling dataset before the update. Of course, the addition of the current sampling dataset is merely an assumption; in reality, the current sampling dataset remains unchanged before and after this step.

[0076] s805, determine whether the entropy corresponding to the current data sample is greater than the label entropy of the current sampling data set. If so, execute s806, otherwise return to s803 to select the next data sample; s806, adding the current data sample as sampling data to the current sampling data set, updating the label entropy of the current sampling data set, and returning to s802.

[0077] Method 3 builds on Method 2 above, but sets N to 1. In this case, Method 3's sampling data selection method degenerates into a greedy algorithm. The specific logic becomes: traverse all filtered original multimodal training data and, for each data sample, add it to the sampling dataset if its inclusion increases the label entropy of the sampling dataset. This method minimizes time complexity while ensuring a certain level of diversity in the sampling data.

[0078] In practical applications, you can choose from the three implementation methods mentioned above based on your requirements for diversity and diversity sampling efficiency. Generally speaking, to balance effectiveness and efficiency, you can use the diversity sampling algorithm with local maximum label entropy in method 2.

[0079] As described above, the process of selecting sampling data based on the principle of maximizing label entropy is given.

[0080] Step 205: Use the selected sample data as training data to train the multimodal large model.

[0081] When using the sampled data as training data, the sampled data can be restored to the form of "image-command-answer" triples, and the image label, command label, and answer rating label can be omitted. The specific processing of using the selected training data for LMM training can be implemented using various existing methods, which will not be detailed here.

[0082] In this embodiment, a small-scale multimodal training data is screened out from the original multimodal training data, and the selected training data has better diversity and higher quality. These selected training data are used to train the LMM model, which can effectively ensure the performance of the LMM model while reducing training data and saving training time.

[0083] In order to verify the effectiveness of the above-mentioned diversity sampling data for LMM model training, 2 million data were selected as the original multimodal training data, and then the "10,000 training data with optimal diversity sampling" were sampled using the above-mentioned method of the present application. At the same time, as comparison data, "10,000 randomly sampled training data" were also randomly sampled. Then, the data sampling principle in the above-mentioned method of the present application was modified, and the sampling data was selected based on the method of minimizing label entropy to select "10,000 training data with the worst diversity sampling". The three types of sampled data and the weighted mixed 2 million training data were used to train an LMM model with the same starting point. The ability of the trained LMM was comprehensively measured by evaluating 15 evaluation sets. The results are shown in Table 5.

[0084] Table 5

[0085] It should be noted that the original 2 million training data used an empirical weight ratio and were selected through multiple experiments. The 10,000 training data with the best diversity sampling was close to the effect of 2 million training data when only 0.5% of the data was used. The performance of the experimental control group with the worst diversity and random sampling as a control was significantly reduced, which proves the beneficial effect of the LMM model training method in this application. Since the benchmark for sampling data selection in this application is image labels, instruction labels and answer rating labels, the above experiments also indirectly show that the text labels are of sufficient quality.

[0086] Step 206: further train the label expert LMM based on the trained LMM.

[0087] The label expert LMM training in the aforementioned step 201 is based on the general LMM. In this step, a better LMM is obtained through training. The label expert LMM is trained based on this LMM, which can effectively improve the effect of the label expert LMM.

[0088] Furthermore, through the processing of step 202, the labels of the original multimodal training data can be automatically obtained. Therefore, more training samples for the label expert LMM can be selected from these labeled data. The labels of these training samples do not need to be manually labeled but can be automatically obtained, which greatly expands the training sample size of the label expert LMM and makes the label distribution in the new label expert LMM training set as balanced as possible, so that the label expert LMM benefits from the training data with balanced label distribution, thereby effectively improving the performance of the label expert LMM. After obtaining the label expert LMM with better performance, the process can return to step 202.

[0089] The above process of using the trained LMM to further train the label expert LMM and expanding the label expert LMM training data can both improve the quality of the labels assigned to the original multimodal training data and enhance the quality of the final diverse sampled data. This step demonstrates the closed-loop iterative improvement of the label expert LMM, the general LMM, and high-quality multimodal training data, with each benefiting from the other.

[0090] At this point, the method flow in this embodiment ends.

[0091] As can be seen from the description of the LMM model training method in the above application and specific embodiments, the LMM model training method in this application uses text labels (image labels, instruction labels, and answer rating labels) as a carrier and the principle of maximizing label entropy as a means to obtain high-quality and diverse training data from original multimodal training data with complex sources, thereby improving the comprehensive capabilities of the general LMM trained using this data. At the same time, the labels obtained by the label expert LMM can expand the training data of the label expert LMM, improving the effectiveness of the label expert LMM. In addition, better multimodal training data improves the effectiveness of the general LMM and can also improve the effectiveness of the label expert LMM as a pre-training model for the label expert LMM. Both perspectives can iteratively improve the label expert LMM, thereby allowing the entire LMM model training system to continuously evolve and improve.

[0092] The above is a specific implementation of the LMM model training method in this application. The LMM model training method can be used in multiple fields, such as monitoring applications. This application also provides an LMM model training device that can be used to implement the above training method. Figure 9 This is a schematic diagram of the basic structure of the LMM model training device in this application. Figure 9 As shown, the training device includes: a label prediction unit, a data filtering unit, a data sampling unit and a multimodal model training unit.

[0093] The label prediction unit is used to predict labels for each data sample in the original multimodal data using a pre-trained label expert multimodal large model LMM for label annotation. The labels here include image labels for marking image categories, instruction labels for marking instruction categories, and answer rating labels for marking answer quality. A data filtering unit, configured to filter the original multimodal data based on the answer rating labels to obtain a filtered data set; a data sampling unit, configured to select data samples from the filtered data set as sampling data based on the image labels and instruction labels of all data samples in the filtered data set, so as to maximize the label entropy determined based on the image labels and instruction labels of all the sampling data; The multimodal model training unit is used to train the multimodal model using the sampled data as training samples.

[0094] Optionally, the apparatus may further include a label expert training unit for pre-training and generating the label expert LMM based on a general LMM using image label seed training data, instruction label seed training data, and answer rating label seed training data determined based on the original multimodal data; The image label seed training data includes pictures selected from the original multimodal data, instructions for instructing image label prediction for the pictures, and answers for recording image labels; The instruction label seed training data includes instructions selected from the original multimodal data, pictures corresponding to the selected instructions, and answers for recording instruction labels; The answer rating label seed training data includes pictures and instructions in the data samples selected based on the answers from the original multimodal data, as well as the answers used to record the label ratings. The label ratings are the rating results corresponding to the answer labels to which the answers in the selected data samples belong.

[0095] Optionally, the label rating may be a label ranking of each label to which the answer in the selected data sample belongs, and the label expert training unit is further configured to determine a comprehensive label ranking based on the label ranking of each label; or, the answer rating label may be a comprehensive label ranking determined based on the label ranking of each label to which the answer in the selected data sample belongs; The data filtering unit is used to filter out data samples whose comprehensive label classification does not meet the set requirements.

[0096] Optionally, in the data sampling unit, the process of selecting a data sample from the filtered data set as the sampling data may specifically include: a. Set the filtered dataset as the current filtered dataset and set the current sampled dataset to an empty set; b. For each current data sample in the current filtered data set, the label entropy of the current data sample and all data samples in the current sampled data set are jointly calculated as the entropy corresponding to the current data sample; c. Determine the maximum entropy value corresponding to each data sample, add the data sample corresponding to the maximum entropy value to the current sampling data set, and delete the corresponding current data sample from the current filtered data set, and return to step b until the number of samples in the current sampling data set meets the requirements.

[0097] Optionally, in the data sampling unit, the process of selecting a data sample from the filtered data set as the sampling data specifically includes: a. Set the current sampling data set to an empty set; b. Select N data samples from the untraversed data samples in the filtered data set as the current filtered data set; c. In the current filtered data set, traverse each current data sample, and jointly calculate the label entropy of the current data sample and all data samples in the current sampling data set as the entropy corresponding to the current data sample; determine the maximum value of the entropy corresponding to each data sample in the current filtered data set. If the maximum value is greater than the label entropy jointly calculated by all data samples in the current sampling data set, then add the current data sample corresponding to the maximum entropy to the current sampling data set, and return to step b until the number of samples in the current sampling data set meets the requirement; otherwise, directly return to step b until the number of samples in the current sampling data set meets the requirement.

[0098] Optionally, in the data sampling unit, the process of selecting a data sample from the filtered data set as the sampling data may specifically include: a. Set the current sampling data set to an empty set; b. In the filtered data set, select a current data sample in turn, and jointly calculate the label entropy of the current data sample and all the data samples in the current sampling data set as the entropy corresponding to the current data sample; if the entropy corresponding to the current data sample is greater than the label entropy jointly calculated by all the data samples in the current sampling data set, add the current data sample to the current sampling data set, and process the next selected data sample until the number of samples in the current sampling data set meets the requirement; if the entropy corresponding to the current data sample is not greater than the label entropy jointly calculated by all the data samples in the current sampling data set, directly process the next selected data sample until the number of samples in the current sampling data set meets the requirement.

[0099] Optionally, the label expert training unit may also be used to further train the label expert LMM using the trained multimodal model; and / or, The label expert training unit can also be used to use the labels and corresponding data samples predicted by the label expert LMM for the data samples in the original multimodal data as training data for further training the label expert LMM.

[0100] The present application also provides a computer-readable storage medium that stores instructions that, when executed by a processor, can execute the steps in the method for training an LMM model as described above. In practical applications, the computer-readable medium can be included in each of the devices / apparatuses / systems described in the above embodiments, or it can exist independently without being incorporated into the device / apparatus / system. The computer-readable storage medium stores instructions that, when executed by a processor, can execute the steps in the method for training an LMM model as described above.

[0101] According to the embodiments disclosed herein, a computer-readable storage medium may be a non-volatile computer-readable storage medium, including, but not limited to, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof, but this is not intended to limit the scope of protection of this application. In the embodiments disclosed herein, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0102] Figure 10 This application also provides an electronic device. Figure 10 , which shows a schematic diagram of the structure of the electronic device involved in the embodiment of the present application, specifically: The electronic device may include a processor 1001 with one or more processing cores, a memory 1002 with one or more computer-readable storage media, and a computer program stored in the memory and executable on the processor. When executing the program in the memory 1002, the LMM model training method may be implemented.

[0103] Specifically, in actual applications, the electronic device may further include components such as a power supply 1003 and an input / output unit 1004. Those skilled in the art will appreciate that Figure 10 The structure of the electronic device shown in the figure does not constitute a limitation on the electronic device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently. The processor 1001 is the control center of the electronic device. It uses various interfaces and lines to connect various parts of the entire electronic device. By running or executing software programs and / or modules stored in the memory 1002 and calling data stored in the memory 1002, it executes various functions of the server and processes data, thereby monitoring the electronic device as a whole.

[0104] The memory 1002 can be used to store software programs and modules, that is, the above-mentioned computer-readable storage medium. The processor 1001 executes various functional applications and data processing by running the software programs and modules stored in the memory 1002. The memory 1002 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function, etc.; the data storage area may store data created according to the use of the server, etc. In addition, the memory 1002 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage device. Accordingly, the memory 1002 may also include a memory controller to provide the processor 1001 with access to the memory 1002.

[0105] The electronic device also includes a power supply 1003 for supplying power to various components. This power supply 1003 can be logically connected to the processor 1001 via a power management system, thereby enabling the power management system to manage charging, discharging, and power consumption. The power supply 1003 can also include one or more DC or AC power supplies, a recharging system, a power failure detection circuit, a power converter or inverter, a power status indicator, and other arbitrary components.

[0106] The electronic device may further include an input / output unit 1004. The input / output unit 1004 may be used to receive input digital or character information and generate keyboard, mouse, joystick, or optical signal input related to user settings and function control. The input / output unit 1004 may also be used to display information input by the user or information provided to the user, as well as various graphical user interfaces. These graphical user interfaces may be composed of graphics, text, icons, videos, or any combination thereof.

[0107] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for training a multimodal large model, characterized in that: include: Predicting labels for the original multimodal training data using a pre-trained label expert multimodal large model LMM for label annotation; wherein the labels include image labels for marking image categories, instruction labels for marking instruction categories, and answer rating labels for marking answer quality; Filtering the original multimodal training data based on the answer rating labels to obtain a filtered data set; Based on the image labels and instruction labels of all data samples in the filtered data set, selecting data samples from the filtered data set as sampled data, so as to maximize the label entropy determined based on the image labels and instruction labels of all sampled data; The sampled data is used as training samples to train a large multimodal model.

2. The method according to claim 1, characterized in that The method further includes: pre-training and generating the label expert LMM based on the general LMM using image label seed training data, instruction label seed training data and answer rating label seed training data determined based on the original multimodal data; The image label seed training data includes a picture selected from the original multimodal training data, an instruction for instructing to predict an image label for the picture, and an answer for recording the image label; The instruction label seed training data includes instructions selected from the original multimodal training data, pictures corresponding to the selected instructions, and answers for recording instruction labels; The answer rating label seed training data includes pictures in the data samples selected based on the answers from the original multimodal training data, instructions for indicating answer rating, and answers for recording label ratings, wherein the label rating is the rating result corresponding to the answer label to which the answer in the selected data sample belongs.

3. The method according to claim 2, characterized in that The label rating is a label ranking of each label to which the answer in the selected data sample belongs, and the method further includes: determining a comprehensive label ranking based on the label ranking of each label; or, the answer rating label includes a comprehensive label ranking determined based on the label ranking of each label to which the answer in the selected data sample belongs; The filtering of the original multimodal data based on the answer rating labels includes: filtering out data samples whose comprehensive label classification does not meet the set requirements.

4. The method according to claim 1, 2 or 3, characterized in that: The selecting a data sample from the filtered data set as sampling data comprises: a. Using the filtered data set as the current filtered data set and setting the current sampled data set to an empty set; b. In the current filtering data set, for each current data sample, the label entropy is calculated jointly with the current data sample and all data samples in the current sampling data set as the entropy corresponding to the current data sample; c. Determine the maximum entropy value corresponding to each data sample, add the data sample corresponding to the maximum entropy value to the current sampling data set, delete the corresponding current data sample from the current filtered data set, and return to step b until the number of samples in the current sampling data set meets the requirements.

5. The method according to claim 1, 2 or 3, characterized in that: The selecting a data sample from the filtered data set as sampling data comprises: a. Set the current sampling data set to an empty set; b. Selecting N data samples from the untraversed data samples in the filtering data set as the current filtering data set; N is a preset positive integer that is less than the total number of data samples in the filtering data set and greater than 1; c. In the current filtered data set, traverse each current data sample, and jointly calculate the label entropy of the current data sample and all the data samples in the current sampling data set as the entropy corresponding to the current data sample; determine the maximum value of the entropy corresponding to each data sample in the current filtered data set. If the maximum value is greater than the label entropy jointly calculated by all the data samples in the current sampling data set, add the current data sample corresponding to the maximum entropy value to the current sampling data set, and return to step b until the number of samples in the current sampling data set meets the requirement; otherwise, directly return to step b until the number of samples in the current sampling data set meets the requirement.

6. The method according to claim 1, 2 or 3, characterized in that: The selecting a data sample from the filtered data set as sampling data comprises: a. Set the current sampling data set to an empty set; b. In the filtered data set, select a current data sample in turn, and jointly calculate the label entropy of the current data sample and all the data samples in the current sampling data set as the entropy corresponding to the current data sample; if the entropy corresponding to the current data sample is greater than the label entropy jointly calculated by all the data samples in the current sampling data set, add the current data sample to the current sampling data set, and process the next selected data sample until the number of samples in the current sampling data set meets the requirement; if the entropy corresponding to the current data sample is not greater than the label entropy jointly calculated by all the data samples in the current sampling data set, directly process the next selected data sample until the number of samples in the current sampling data set meets the requirement.

7. The method according to claim 2, characterized in that The method further comprises: using the trained multimodal model to further train the label expert LMM; and / or, The labels and corresponding data samples predicted by the label expert LMM for the data samples in the original multimodal data are used as training data to further train the label expert LMM.

8. A multimodal model training device, characterized in that: The device comprises: a label prediction unit, a data filtering unit, a data sampling unit and a multimodal model training unit; The label prediction unit is used to predict a label for each data sample in the original multimodal data using a pre-trained label expert multimodal large model LMM for label annotation; wherein the label includes an image label for marking an image category, an instruction label for marking an instruction category, and an answer rating label for marking an answer quality; The data filtering unit is used to filter the original multimodal data based on the answer rating label to obtain a filtered data set; The data sampling unit is used to select data samples from the filtered data set as sampling data based on the image labels and instruction labels of all data samples in the filtered data set, so as to maximize the label entropy determined based on the image labels and instruction labels of all the sampling data; The multimodal model training unit is used to train the multimodal model using the sampled data as training samples.

9. The device according to claim 8, characterized in that The device further includes a label expert training unit for training and generating the label expert LMM in advance based on the general LMM using image label seed training data, instruction label seed training data and answer rating label seed training data determined based on the original multimodal data; The image label seed training data includes a picture selected from the original multimodal data, an instruction for instructing to predict an image label for the picture, and an answer for recording the image label; The instruction label seed training data includes instructions selected from the original multimodal data, pictures corresponding to the selected instructions, and answers for recording instruction labels; The answer rating label seed training data includes pictures and instructions in the data samples selected from the original multimodal data based on the answers, and answers for recording label ratings, wherein the label ratings are rating results corresponding to the answer labels to which the answers in the selected data samples belong.

10. The device according to claim 9, characterized in that The label rating is the label classification of each label to which the answer in the selected data sample belongs, and the label expert training unit is further used to determine the comprehensive label classification based on the label classification of each label; or, the answer rating label is the comprehensive label classification determined based on the label classification of each label to which the answer in the selected data sample belongs; The data filtering unit is used to filter out data samples whose comprehensive label classification does not meet the set requirements.

11. The device according to claim 8, 9 or 10, characterized in that In the data sampling unit, selecting a data sample from the filtered data set as sampling data includes: a. Using the filtered data set as the current filtered data set and setting the current sampled data set to an empty set; b. In the current filtering data set, for each current data sample, the label entropy is calculated jointly with the current data sample and all data samples in the current sampling data set as the entropy corresponding to the current data sample; c. Determine the maximum entropy value corresponding to each data sample, add the data sample corresponding to the maximum entropy value to the current sampling data set, delete the corresponding current data sample from the current filtered data set, and return to step b until the number of samples in the current sampling data set meets the requirements.

12. The device according to claim 8, 9 or 10, characterized in that In the data sampling unit, selecting a data sample from the filtered data set as sampling data includes: a. Set the current sampling data set to an empty set; b. Selecting N data samples from the untraversed data samples in the filtering data set as the current filtering data set; c. In the current filtered data set, traverse each current data sample, and jointly calculate the label entropy of the current data sample and all the data samples in the current sampling data set as the entropy corresponding to the current data sample; determine the maximum value of the entropy corresponding to each data sample in the current filtered data set. If the maximum value is greater than the label entropy jointly calculated by all the data samples in the current sampling data set, add the current data sample corresponding to the maximum entropy value to the current sampling data set, and return to step b until the number of samples in the current sampling data set meets the requirement; otherwise, directly return to step b until the number of samples in the current sampling data set meets the requirement.

13. The device according to claim 8, 9 or 10, characterized in that In the data sampling unit, selecting a data sample from the filtered data set as sampling data includes: a. Set the current sampling data set to an empty set; b. In the filtered data set, select a current data sample in turn, and jointly calculate the label entropy of the current data sample and all the data samples in the current sampling data set as the entropy corresponding to the current data sample; if the entropy corresponding to the current data sample is greater than the label entropy jointly calculated by all the data samples in the current sampling data set, add the current data sample to the current sampling data set, and process the next selected data sample until the number of samples in the current sampling data set meets the requirement; if the entropy corresponding to the current data sample is not greater than the label entropy jointly calculated by all the data samples in the current sampling data set, directly process the next selected data sample until the number of samples in the current sampling data set meets the requirement.

14. The device according to claim 9, characterized in that The label expert training unit is used to further train the label expert LMM using the trained multimodal model; and / or, The label expert training unit is used to use the labels and corresponding data samples predicted by the label expert LMM for the data samples in the original multimodal data as training data, so as to further train the label expert LMM.

15. A computer-readable storage medium having computer instructions stored thereon, characterized in that: When the instructions are executed by the processor, the multimodal model training method described in any one of claims 1 to 7 can be implemented.

16. An electronic device, characterized in that: The electronic device includes at least a computer-readable storage medium and also includes a processor; The processor is used to read executable instructions from the computer-readable storage medium and execute the instructions to implement the multimodal model training method described in any one of claims 1 to 7 above.

Images