Multi-modal information processing model training method, sample screening method and device

Abstract

The invention provides a multi-modal information processing model training method and device and a sample screening method and device, belongs to the technical field of computers, and particularly relates to the technical fields of artificial intelligence, large models, data screening, information processing and the like. According to the specific implementation scheme, the method comprises the steps of labeling multi-modal data based on a multi-modal large model to obtain a capability label of the multi-modal data; the capability label is used for describing the model capability which can be trained by the multi-modal data; and performing supervised training on the to-be-trained model based on the multi-modal data and the capability label to train the to-be-trained model into a multi-modal information processing model capable of screening out useful samples for the target model.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to the technical fields of artificial intelligence, large models, data screening, information processing, etc. Background Art

[0002] With the development of the Internet, the amount of data generated by social media, IoT devices, etc. has increased dramatically. This makes it possible to train more complex large models.

[0003] Massive data samples help the model to fully learn the rules and knowledge therein, so as to facilitate accurate regression analysis and classification. Summary of the invention

[0004] The present disclosure provides a training method, a sample screening method and a device for a multimodal information processing model.

[0005] According to one aspect of the present disclosure, a method for training a multimodal information processing model is provided, comprising:

[0006] Based on the multimodal large model, the multimodal data is labeled to obtain the capability label of the multimodal data; the capability label is used to describe the model capability that can be trained by the multimodal data;

[0007] Based on the multimodal data and capability labels, the model to be trained is supervised to train the model to be trained into a multimodal information processing model that can filter out useful samples for the target model.

[0008] According to one aspect of the present disclosure, there is provided a sample screening method, comprising:

[0009] Get the target dataset;

[0010] Inputting the target data set into the multimodal information processing model to obtain capability labels of data samples in the target data set output by the multimodal information processing model;

[0011] Based on the capability labels of the data samples in the target dataset, training samples for training the target model are screened out from the target dataset.

[0012] According to another aspect of the present disclosure, there is provided a training device for a multimodal information processing model, comprising:

[0013] The labeling module is used to label the multimodal data based on the multimodal large model to obtain the capability label of the multimodal data; the capability label is used to describe the model capability that can be trained by the multimodal data;

[0014] The training module is used to perform supervised training on the model to be trained based on multimodal data and capability labels, so as to train the model to be trained into a multimodal information processing model that can screen out useful samples for the target model.

[0015] According to another aspect of the present disclosure, there is provided a sample screening device, comprising:

[0016] An acquisition module is used to acquire a target data set;

[0017] An output module, used to input the target data set into the multimodal information processing model, and obtain capability labels of data samples in the target data set output by the multimodal information processing model;

[0018] The screening module is used to screen out training samples for training the target model from the target data set based on the capability labels of the data samples in the target data set.

[0019] According to another aspect of the present disclosure, there is provided an electronic device, comprising:

[0020] at least one processor; and

[0021] a memory communicatively connected to the at least one processor; wherein,

[0022] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute any method in the embodiments of the present disclosure.

[0023] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute any method according to the embodiments of the present disclosure.

[0024] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, which implements any method according to the embodiments of the present disclosure when executed by a processor.

[0025] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0027] Figure 1 is a flowchart of a training method for a multimodal information processing model provided according to an embodiment of the present disclosure;

[0028] Figure 2 is a schematic diagram of a process for obtaining a capability label of multimodal data according to an embodiment of the present disclosure;

[0029] Figure 3 is a flowchart of supervised training provided according to an embodiment of the present disclosure;

[0030] Figure 4 is a schematic diagram of a process of training a model to be trained according to an embodiment of the present disclosure;

[0031] Figure 5 is a flow chart of a sample screening method provided according to an embodiment of the present disclosure;

[0032] Figure 6 It is a schematic diagram of an overall process provided according to an embodiment of the present disclosure;

[0033] Figure 7 is a structural schematic diagram of a training device for a multimodal information processing model provided according to an embodiment of the present disclosure;

[0034] Figure 8 is a schematic structural diagram of a sample screening device provided according to an embodiment of the present disclosure;

[0035] Fig. 9 It is a block diagram of an electronic device used to implement the training method and / or sample screening method of the multimodal information processing model of the embodiment of the present disclosure. DETAILED DESCRIPTION

[0036] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope of the present disclosure. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0037] The terms "first", "second", etc. in this disclosure are used to distinguish similar objects, and are not necessarily used to describe a specific order or precedence. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, including a series of steps or units. The method, system, product or device is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0038] In the era of big models, the importance of data has been further emphasized and enhanced. The training of big models requires a large amount of data as input to generate accurate and reliable predictions. Among them, data is the key factor in training models, which determines the performance and accuracy of the model. Without sufficient data, the model cannot learn and predict effectively.

[0039] In addition to the quantity of data, the quality of data also plays a vital role in model training. High-quality data can provide more accurate training signals, thereby improving the performance of the model. Therefore, in the process of selecting high-quality data, a lot of data analysis and processing operations are required.

[0040] However, with the development of technology and the accumulation of time, the time span for the formation of data samples is large, and the quality standards of data samples are constantly improving and changing. At present, the automatic screening of data samples is limited to the screening of plain text data. As for multimodal data, on the one hand, a large number of data samples have been generated over the years, and on the other hand, the quality of multimodal data is constantly being optimized, making the quality of current multimodal data samples inconsistent. For example, although the current open source data is huge in volume, not all data needs to be trained due to inconsistent data quality. Or, when some of the data is needed to train the model, there is a lack of strategies to automatically screen high-quality samples by mining the connection between data samples and the target model to be trained.

[0041] In view of this, the embodiments of the present disclosure provide a solution. In this solution, first of all, it relies on the reasoning and decision-making ability of the multimodal large model to help train a multimodal information processing model that can be labeled. The multimodal information processing model can mine the relationship between multimodal data samples and the target model that needs to be trained. In the embodiments of the present disclosure, this relationship is represented by the capability label of the multimodal data sample, so as to measure which model capabilities can be trained by different data samples. Thus, high-quality training samples can be mined based on the capability label, and low-quality redundant data samples can be eliminated. In the face of massive data, it can provide high-quality data for the target model while filtering out redundant low-quality data to improve the training efficiency of the target model.

[0042] In a first aspect of the embodiments of the present disclosure, a method for training a multimodal information processing model is provided. In the training method, a multimodal information processing model capable of labeling (also referred to as a label expert model) can be trained with the assistance of a multimodal large model. Specifically, Figure 1 As shown, the training method of the multimodal information processing model provided by the embodiment of the present disclosure includes the following contents:

[0043] S101, labeling the multimodal data based on the multimodal large model to obtain capability labels of the multimodal data; the capability labels are used to describe the model capabilities that can be trained with the multimodal data.

[0044] Multimodal data usually includes information in at least two of the following modes: text mode, image mode, audio mode, etc. Among them, video can be understood as data in image mode.

[0045] A multimodal big model refers to a model that combines multimodal information such as text, images, audio, and video for processing. This model can process multiple types of data and achieve more efficient information response and intelligent recognition by fusing data from different modalities.

[0046] It is understandable that the multimodal data in S101 is described by taking one data sample as an example. When there are multiple multimodal data, each multimodal data can be executed with reference to this, and the embodiments of the present disclosure are not limited to this.

[0047] Labeling these multimodal data based on the multimodal big model is to conduct in-depth analysis of the input multimodal data through the powerful understanding and reasoning capabilities of the multimodal big model, and generate corresponding capability labels for each multimodal data based on the analysis results.

[0048] As the name implies, capability labels mark which model capabilities of the target model can be trained by multimodal data. For example, speech recognition capability, text analysis capability, emotion recognition capability, image description capability, image classification capability, target object location capability, character recognition capability in images, semantic understanding capability, complex capability of finding relevant answers from images based on questions, etc. Specific capabilities can be defined according to business needs, and even model capabilities can be divided into different levels according to needs. During implementation, model capabilities can also be divided from different angles and granularity, and model capabilities can include thousands of categories.

[0049] On the basis of having capability labels, when a multimodal data sample A can only train a very small number of model capabilities, and these model capabilities can be covered by multiple other data samples, the data sample A is an unimportant, dispensable and redundant data sample. In the case of screening out a small number of partial samples from a massive amount of data samples, the data sample A will be eliminated. Therefore, through the capability label, the relationship between the data sample and the target model can be explicitly expressed, so as to facilitate the mining of reliable and useful samples to train the target model. In summary, the accuracy and comprehensiveness of the capability label are crucial to the subsequent training of the target model. Therefore, the embodiment of the present disclosure trains an accurate and reliable multimodal information processing model that can predict capability labels through subsequent steps. The model will be trained with the help of effective samples mined from the multimodal large model, and please refer to S102 for details.

[0050] S102, based on the multimodal data and the capability labels, supervised training is performed on the model to be trained, so as to train the model to be trained into a multimodal information processing model that can filter out useful samples for the target model.

[0051] The model to be trained is a model in its initial state. The model can be a lightweight model that can process multimodal information, or a large model with a complex structure and large model parameters. Based on the labeled multimodal data and the corresponding capability labels, supervised training can be performed on the model to be trained in this initial state. The trained model should have the ability to accurately mine the capability labels of multimodal data, so as to filter out useful samples that are conducive to training the target model from multiple data samples based on the capability labels.

[0052] In the disclosed embodiment, a training sample of a model to be trained is constructed through a multimodal large model so that a multimodal information processing model can be trained in a supervised training manner in the future. The multimodal information processing model can accurately predict the capability labels of multimodal data, thereby intuitively and conveniently reflecting the relationship between multimodal data and the target model. Thus, the capability labels predicted by the multimodal information processing model can be further used to screen high-quality samples for the target model.

[0053] In the embodiment of the present disclosure, the implementation method of labeling multimodal data based on the multimodal large model to obtain the capability label of the multimodal data is as follows: Figure 2 As shown, including:

[0054] S201, constructing input information of a multimodal large model based on multimodal data and a preset prompt word template.

[0055] That is, through the preset prompt word template, the multimodal large model is helped to start from the perspective of mining model capabilities, thereby deeply understanding and sorting out the characteristics and task requirements of multimodal data, so as to accurately mine the capability labels of multimodal data.

[0056] During implementation, appropriate prompt word templates can be designed based on the characteristics of multimodal data and task requirements. These templates should be able to guide the multimodal large model to focus on the key information of the data and guide the multimodal large model to generate the desired output. During implementation, the preset prompt word template may include at least one of the following requirements:

[0057] (1) It requires large multimodal models to focus on the detailed content of image modality information in multimodal data;

[0058] That is, when ensuring that the prompt word template can guide the multimodal large model to process multimodal data containing images, the multimodal large model is required to pay special attention to the detailed content in the image. This helps the multimodal large model to more accurately understand the model capabilities that can be trained by image modality information, thereby generating accurate and rich capability labels.

[0059] For example, in medical image analysis, the prompt word template can explicitly require the model to pay attention to detailed information such as the size, shape, and location of the lesion, with the goal of improving the accuracy of diagnosis, so that the multimodal large model can accurately output capability labels.

[0060] (2) When multimodal data includes question-answer pairs, the multimodal large model is required to combine the question-answer pairs to understand the query intent of the question in the question-answer pair for the image modality information in the multimodal data;

[0061] That is, when processing multimodal data containing question-answer pairs and images, the multimodal large model can combine the questions in the question-answer pairs to accurately understand the query intent of the question for the image modal information, so as to mine relevant content from the image modal information and predict the capability label of the multimodal data based on the mining results.

[0062] (3) Large multimodal models are required to analyze and reason about the input information to provide reasonable capability labels for multimodal data.

[0063] That is, it requires that the multimodal large model can perform comprehensive analysis and reasoning based on the input multimodal data and give reasonable capability labels. This helps to accurately label the target model based on the process and results of analysis and reasoning, so that when the target model is processing complex tasks, it can accurately screen out the required training samples based on the task requirements of the target model.

[0064] For example, the prompt word template can require the large model to understand the problem first in the form of a chain of thoughts, and then search for answers in the image modal information step by step for the key points in the problem, so as to provide an accurate answer. Therefore, the multimodal large model can accurately and meticulously analyze the model capabilities that can be trained by multimodal data based on the analysis and reasoning process. It can also be understood that the ability labels of multimodal data can be sorted out according to the analysis and reasoning methods and analysis and reasoning steps of the multimodal large model. Through reasoning analysis, the model capabilities can be grasped and classified more accurately. For example, if the input image and a question are "Please calculate the number of red apples in the picture", the multimodal large model should be guided to give reasonable ability labels such as "object counting ability", "object recognition ability", and "color recognition ability".

[0065] In the disclosed embodiment, attention is paid to the detailed content in the image modal information in the multimodal data, and the information content conveyed by the image modal information can be understood in detail and accurately, so as to better construct a complete situational picture and enhance the grasp of the overall situation of the image modal information and the cognition of the ability to detail. In the case where the multimodal data includes question-answer pairs, the question in the question-answer pair is combined with the question-answer pair to understand the query intent of the image modal information in the multimodal data, and different labels of intent recognition capabilities can be mined. The multimodal large model is required to provide reasonable and accurate capability labels based on the analysis and reasoning of the input information. The capability labels obtained by analysis and reasoning can accurately and meticulously express the model capabilities that can be trained by the multimodal data, so as to accurately screen data samples for the target model. In summary, the preset prompt word template plays a key role in multimodal data processing, which can guide the model to pay attention to specific information, understand the query intent, and provide accurate capability labels, thereby improving the ability of the model to be trained in mining the relationship between data samples and the target model.

[0066] S202, inputting the input information into the multimodal big model to obtain the capability label for the multimodal data annotated by the multimodal big model.

[0067] The multimodal big model will parse and understand the input information and obtain the capability labels that the multimodal big model annotates for the multimodal data. These capability labels represent the model capabilities that can be trained with multimodal data.

[0068] During implementation, an accurate and reliable multimodal large model can be selected to process the input information, such as the GPT4-o (a latest artificial intelligence model) large model, so as to ensure the accuracy of the capability labels obtained for the multimodal data annotation.

[0069] In the disclosed embodiment, by constructing input information based on multimodal data and a preset prompt word template, the preset prompt word template can guide the multimodal large model to focus more on how to mine capability labels. Even when the types of model capabilities are subdivided into thousands or tens of thousands, the preset prompt word template can be used to guide the multimodal large model to fully and deeply mine the capability labels of the multimodal data, which helps to accurately capture the important elements in the multimodal data, thereby establishing the relationship between the multimodal data and the target model.

[0070] In the disclosed embodiment, in order to make the multimodal information processing model obtained based on the training of the model to be trained have better label prediction ability and be able to better screen data samples for the target model, the model structure of the model to be trained and the model structure of the target model may be required to be the same. That is, the model structure of the model to be trained and the model structure of the target model are consistent in terms of hierarchical composition, connection mode between layers, and parameter settings of each layer. In this way, the model to be trained can screen out effective data samples for the target model based on almost the same cognition as the target model on a specific data set.

[0071] In other embodiments, in order to quickly train a multimodal information processing model that can be labeled, the model to be trained can also be a student model of the target model.

[0072] As the name implies, the teacher model is usually a complex model with excellent performance, while the student model is a model with a simpler structure and fewer parameters. The goal of the student model is to approximate the performance of the teacher model by learning the output of the teacher model. During the training process, the student model not only learns how to generate output directly from the input data, but also learns the intermediate representation of the input data of the teacher model, so that it can capture more knowledge in the teacher model.

[0073] It can be understood that before training the target model, the target model can be pre-trained, and then the student model of the target model can be obtained. The student model has a simple structure, lower requirements for memory and hardware, and higher reasoning speed, so the use of the student model can improve the training efficiency of the multimodal information processing model and maintain the annotation accuracy of the capability label.

[0074] In the disclosed embodiment, the model structure of the model to be trained is the same as the model structure of the target model. Without redesigning the model architecture, the structure of the target model can be quickly applied to new tasks or data sets, saving a lot of time and energy. Due to the same structure, its performance has a certain predictability. It is known that the target model can have a good performance under the target task, so the model to be trained with the same structure is also likely to achieve good results under similar conditions, reducing uncertainty, thereby providing a strong foundation for the target model to mine useful data samples. The student model usually has fewer parameters and a simpler structure, which can greatly reduce the amount of calculation and memory usage, thereby achieving more efficient reasoning and deployment. In addition, due to the small size of the student model, the amount of data and time required for training are relatively small, thereby reducing the cost of data collection and annotation, as well as the computational cost during training, and can also accurately predict capability labels.

[0075] In the disclosed embodiment, considering the hardware cost and time cost, the complexity of the model structure of the multimodal large model in the disclosed embodiment is greater than that of the model to be trained. Therefore, the labeling capability of the multimodal large model for capability labels can be transferred to the model to be trained. The model to be trained can occupy fewer hardware resources and have a lower reasoning process time. It is possible to complete the labeling of capability labels for massive data faster.

[0076] Of course, in other embodiments, the ability of the multimodal large model to mine capability labels can be transferred to other large models required by the user, so that the user can reasonably use the trained multimodal information processing model according to needs.

[0077] In the disclosed embodiment, the training model can be trained by supervised training. The training process is as follows: Figure 3 As shown, including the following:

[0078] S301, inputting multimodal data into a model to be trained, and obtaining a prediction label output by the model to be trained for the multimodal data.

[0079] The multimodal data is input into the model to be trained. The model to be trained will process the input data according to the current parameter settings and output the corresponding prediction label.

[0080] It can be understood that there are sub-information of multiple modes in multimodal data. The model to be trained has a processing module for the sub-information of each mode. For example, an image processing module, an audio processing module, and a text processing module. Multiple processing modules can exchange intermediate results to obtain feature expressions that can be used to predict capability labels. Alternatively, the intermediate results obtained by each of the multiple processing modules can be input into a feature fusion module to obtain feature expressions that can predict capability labels after feature fusion processing.

[0081] S302, determining a prediction loss based on the prediction label and the capability label.

[0082] The predicted labels of the model's output are compared with the labels of the multimodal large model's ability to annotate multimodal data, and the prediction loss is calculated through a loss function such as cross entropy loss, mean square error loss, etc. The choice of loss function can be determined according to the specific requirements of the task.

[0083] S303: Adjust model parameters of the model to be trained based on the prediction loss.

[0084] Based on the calculated prediction loss, the loss value can be transferred from the output layer to the input layer layer by layer through the back propagation algorithm, and the gradient of the parameters of each layer is calculated at the same time. The parameters of the model are updated using optimization algorithms such as gradient descent to reduce the prediction loss. This process will be iterated multiple times until the performance of the model to be trained on the validation set is no longer significantly improved, or until the specified number of iterative training is completed.

[0085] In the disclosed embodiment, by inputting multimodal data into the model to be trained and obtaining the prediction ability of its output, and determining the prediction loss based on the prediction ability and ability label, it is helpful to find the gap between the model to be trained in the training data and the real scene. This enables the model to be trained to better adapt to various different situations, rather than being limited to specific patterns in the training data, and enhances the generalization ability of the model, so that it can perform well on new and unseen data.

[0086] In the embodiment of the present disclosure, supervised training is performed on the training model based on multimodal data and capability labels, which can also be implemented as follows: Figure 4 The method shown includes the following:

[0087] S401, based on the capability labels respectively annotated by the multimodal large model for the multiple multimodal data, an initial training sample set is selected from the multiple multimodal data.

[0088] That is, representative high-quality samples are selected from the labeled multimodal data to construct an initial training sample set. The initial training sample set iteratively optimizes the target model based on the task requirements of the target model. These samples in the initial training sample set should cover the key capability labels that the target model should have. For example, all model capabilities that can be included in the capability label are divided into different levels of model capabilities according to the importance of the task, and key capability labels are constructed based on the core level model capabilities. In addition, the initial training sample set can also be required to maintain data diversity and balance as much as possible to avoid model overfitting or bias.

[0089] S402, using the initial training sample set to perform at least one round of iterative optimization on the target model, so as to use the iteratively optimized target model as the model to be trained.

[0090] The target model is optimized for at least one round of iteration using the initial training sample set. During the iterative optimization process, the performance of the target model on its target task is improved by adjusting the parameters, structure or training strategy of the model. It is understandable that the target task is the original task that the target model is expected to have, and it may not be a task with annotated capability labels.

[0091] S403: Perform supervised training on the model to be trained based on the multiple multimodal data and their respective capability labels.

[0092] Based on multiple multimodal data and their respective capability labels, supervised training is performed on the iteratively optimized model to be trained. During the training process, the model will learn how to map the input multimodal data to the corresponding capability labels and continuously optimize its internal parameters to improve the accuracy of capability label prediction.

[0093] In the disclosed embodiment, by screening the initial training sample set based on the capability labels that are labeled with multiple multimodal data based on the multimodal large model, high-quality samples required for the target task can be mined. The target model is optimized for at least one round of iteration using the high-quality samples to obtain the model to be trained, so that the obtained model to be trained can have a relatively high-quality model parameter from the initial state, and is based on the target task of the target model. On this basis, the model to be trained is continued to be trained so that it has the ability to label capability labels, which can make the training process of the model to be trained converge faster, improve the training efficiency, and also enable the model to be trained to have a relatively high-quality initial parameter, thereby improving the prediction ability of the capability label.

[0094] In the first aspect, the foregoing article describes how to train a multimodal information processing model. In the second aspect, in the embodiment of the present disclosure, based on the same technical concept, a sample screening method is also provided, which is applicable to the multimodal information processing model trained above. Figure 5 As shown, the sample screening method may include the following:

[0095] S501, obtaining a target data set.

[0096] The target dataset refers to a collection of data samples collected for training or testing a specific model. These data samples are usually multimodal data, which are massive. They can be obtained through various channels, such as public datasets, self-built datasets, or datasets provided by third parties.

[0097] S502, inputting the target data set into the multimodal information processing model to obtain capability labels of data samples in the target data set output by the multimodal information processing model.

[0098] That is, the target data set is input into the trained multimodal information processing model. The multimodal information processing model will output the capability label of each data sample based on the input data sample, thereby describing the training capability of different data samples for the target model.

[0099] S503: based on the capability labels of the data samples in the target data set, filter out training samples for training the target model from the target data set.

[0100] That is, according to the specific content and requirements of the capability label, set the screening criteria to screen out the qualified training samples. These training samples will be used for the subsequent training of the target model. In the embodiment of the present disclosure, the specific screening criteria can meet the following requirements:

[0101] Data samples whose number of capabilities included in the capability labels meets the first preset requirement are selected from the target data set as training samples; and / or data samples whose proportion of the total amount of training samples in the target data set meets the second preset requirement are selected from the target data set based on the capability labels as training samples.

[0102] Based on the content described above, the capability label represents which model capabilities of the target model can be trained by multimodal data. Therefore, in the acquired target data set, different target data can also train different model capabilities. Taking the question-answer pairs in the target data set as an example, simple samples may only have the image recognition capability of the training model, while complex samples can train multiple capabilities of the model, such as image recognition capability, positioning capability, classification capability, etc. Compared with the second type of data samples that can provide a large number of model capabilities, the first type of data samples that provide a small amount of model capabilities provide low-density capabilities. Compared with the second type of data samples, the first type of data samples are data samples with low information density. Conversely, the second type of data samples are data samples with high information density compared to the first type of data samples. When selecting limited data samples from the massive target data set, the second type of data samples will be given priority, thereby eliminating data samples with low information density.

[0103] Therefore, in the embodiment of the present disclosure, the first preset requirement refers to a specific threshold value of the number of capabilities that should be included in the capability label. For example, the first preset requirement may be that each target data sample contains at least m different capability labels.

[0104] The second preset requirement refers to the proportion of the total amount of training samples screened from the target data set in the target data set. That is, after screening out data samples that meet the first preset requirement, the proportion of the total amount of these samples in the target data set is calculated.

[0105] During implementation, when two preset requirements need to be met, the second preset requirement can be met by adjusting the number of labels included in the first preset requirement. For example, after preliminary screening, each target data sample contains at least k different capability labels, and the data samples that meet the first preset requirement account for 60% of the total data set. If the second preset requirement is 80%, it is necessary to adjust the screening conditions, such as lowering the threshold of the number of capabilities to n (n is less than k), and then re-screening until the 80% requirement is met.

[0106] Therefore, when the data set is very large, by setting the number of capability labels or the proportion of the total number of samples in the overall data set, the quality and number of training sample sets can be ensured, thereby improving the efficiency of model training and helping to avoid wasting computing resources on too much invalid or low-quality data.

[0107] In summary, in the disclosed embodiments, the target data set is input into the multimodal information processing model, and the corresponding capability labels generated by the multimodal information processing model for the data samples are screened out based on the capability labels, which are of higher quality and more in line with the training requirements of the target model, and help improve the training effect and generalization ability of the target model. Screening out the training samples used to train the target model reduces unnecessary data involved in training, reduces the consumption of computing resources, and improves training efficiency.

[0108] In summary, when faced with a large amount of multimodal data with training labels, such as Figure 6 As shown, the training data of the model to be trained can be generated by combining the existing multimodal large model, such as the GPT4-o large model, with the preset prompt (prompt template). A multimodal information processing model, namely the capability label expert model, is trained based on the target model through the training data. Afterwards, based on the trained multimodal information processing model, the massive multimodal data is labeled and filtered. Among them, by annotating each multimodal data, the model capability corresponding to each multimodal data can be understood, which is not only conducive to screening out high-quality training samples, but also can find out which model capabilities are lacking when the samples of a certain model capability are evaluated to be insufficient, and it is helpful to screen samples of different model capabilities, and even supplement the training samples corresponding to the model capability. Filtering each multimodal data can filter out the available target data for training the target model. By comparison, the solution provided by the present disclosure can filter out 50% of the low-density data to ensure the same accuracy as the original model.

[0109] Based on the same technical concept, the embodiment of the present disclosure also provides a training device 700 for a multimodal information processing model, such as Figure 7 As shown, including:

[0110] The labeling module 701 is used to label the multimodal data based on the multimodal large model to obtain the capability label of the multimodal data; the capability label is used to describe the model capability that can be trained by the multimodal data;

[0111] The training module 702 is used to perform supervised training on the model to be trained based on the multimodal data and capability labels, so as to train the model to be trained into a multimodal information processing model that can filter out useful samples for the target model.

[0112] In some embodiments, the annotation module includes:

[0113] A construction unit, used to construct input information of a multimodal large model based on multimodal data and a preset prompt word template;

[0114] The labeling unit is used to input the input information into the multimodal large model to obtain the ability label of the multimodal large model for labeling the multimodal data.

[0115] In some embodiments, the preset prompt word template includes at least one of the following requirements:

[0116] The multimodal large model is required to pay attention to the detailed content of the image modality information in the multimodal data;

[0117] When multimodal data includes question-answer pairs, the multimodal large model is required to combine the question-answer pairs to understand the query intent of the question in the question-answer pair for the image modality information in the multimodal data;

[0118] The multimodal large model is required to analyze and reason about the input information to provide reasonable capability labels for the multimodal data.

[0119] In some embodiments, the model structure of the model to be trained is the same as the model structure of the target model; or,

[0120] The model to be trained is the student model of the target model.

[0121] In some embodiments, the model structure of the multimodal large model is more complex than the model to be trained.

[0122] In some embodiments, the training module includes:

[0123] A prediction unit, used to input the multimodal data into the model to be trained, and obtain a prediction label output by the model to be trained for the multimodal data;

[0124] A determination unit, configured to determine a prediction loss based on the prediction label and the capability label;

[0125] The first optimization unit is used to adjust the model parameters of the to-be-trained model based on the prediction loss.

[0126] In some embodiments, the training module includes:

[0127] A screening unit, configured to screen out an initial training sample set from the multiple multimodal data based on the capability labels respectively labeled by the multimodal large model for the multiple multimodal data;

[0128] A second optimization unit is used to perform at least one round of iterative optimization on the target model using the initial training sample set, so as to use the iteratively optimized target model as a model to be trained;

[0129] The training unit is used to perform supervised training on the training model based on multiple multimodal data and their respective capability labels.

[0130] Based on the same technical concept, the embodiment of the present disclosure also provides a sample screening device 800, such as Figure 8 As shown, including:

[0131] An acquisition module 801 is used to acquire a target data set;

[0132] An output module 802 is used to input the target data set into the multimodal information processing model to obtain capability labels of data samples in the target data set output by the multimodal information processing model;

[0133] The screening module 803 is used to screen out training samples for training the target model from the target data set based on the capability labels of the data samples in the target data set.

[0134] In some embodiments, the screening module comprises:

[0135] A first screening unit is used to screen out data samples whose capability quantity included in the capability label meets a first preset requirement from the target data set as training samples; and / or,

[0136] The second screening unit is used to screen out data samples whose proportion of the total amount of training samples in the target data set meets a second preset requirement from the target data set based on the capability label as training samples.

[0137] For the description of specific functions and examples of each module and submodule of the device in the embodiment of the present disclosure, reference can be made to the relevant description of the corresponding steps in the above method embodiment, which will not be repeated here.

[0138] In the technical solution disclosed herein, the acquisition, storage and application of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0139] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0140] Fig. 9A schematic block diagram of an example electronic device 900 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0141] like Fig. 9 As shown, the device 900 includes a computing unit 901, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 902 or a computer program loaded from a storage unit 908 into a random access memory (RAM) 903. In the RAM 903, various programs and data required for the operation of the device 900 can also be stored. The computing unit 901, the ROM 902, and the RAM 903 are connected to each other via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0142] A number of components in the device 900 are connected to the I / O interface 905, including: an input unit 906, such as a keyboard, a mouse, etc.; an output unit 907, such as various types of displays, speakers, etc.; a storage unit 908, such as a disk, an optical disk, etc.; and a communication unit 909, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 909 allows the device 900 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0143] The computing unit 901 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 901 performs the various methods and processes described above, such as the training method and / or sample screening method of the multimodal information processing model. For example, in some embodiments, the training method and / or sample screening method of the multimodal information processing model may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 908. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 900 via the ROM 902 and / or the communication unit 909. When the computer program is loaded into the RAM 903 and executed by the computing unit 901, one or more steps of the training method and / or sample screening method of the multimodal information processing model described above may be performed. Alternatively, in other embodiments, the computing unit 901 may be configured to execute the training method and / or the sample screening method of the multimodal information processing model in any other appropriate manner (for example, by means of firmware).

[0144] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0145] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0146] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0147] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0148] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0149] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0150] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0151] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the principles of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A training method for a multimodal information processing model, comprising: Annotating the multimodal data based on the multimodal large model to obtain capability labels of the multimodal data; The capability label is used to describe the model capability that can be trained by the multimodal data; Based on the multimodal data and the capability labels, supervised training is performed on the model to be trained to train the model to be trained into a multimodal information processing model that can filter out useful samples for the target model.

2. According to the method of claim 1, the step of labeling the multimodal data based on the multimodal big model to obtain the capability label of the multimodal data comprises: Based on the multimodal data and a preset prompt word template, construct input information of the multimodal large model; The input information is input into the multimodal big model to obtain a capability label annotated by the multimodal big model for the multimodal data.

3. The method according to claim 2, wherein the preset prompt word template includes at least one of the following requirements: Require the multimodal large model to pay attention to the detailed content in the image modality information in the multimodal data; In the case where the multimodal data includes a question-answer pair, the multimodal large model is required to understand the query intent of the question in the question-answer pair for the image modality information in the multimodal data in combination with the question-answer pair; The multimodal large model is required to analyze and reason about the input information to provide reasonable capability labels for the multimodal data.

4. The method according to claim 1, wherein: The model structure of the model to be trained is the same as the model structure of the target model; or, The model to be trained is a student model of the target model.

5. According to the method of claim 1, the complexity of the model structure of the multimodal large model is greater than that of the model to be trained.

6. The method according to claim 1, wherein: The step of performing supervised training on the model to be trained based on the multimodal data and the capability label includes: Inputting the multimodal data into the model to be trained to obtain a prediction label output by the model to be trained for the multimodal data; determining a prediction loss based on the prediction label and the capability label; The model parameters of the model to be trained are adjusted based on the prediction loss.

7. The method according to claim 1, wherein: The step of performing supervised training on the model to be trained based on the multimodal data and the capability label includes: Based on the capability labels respectively labeled by the multimodal large model for the plurality of multimodal data, an initial training sample set is selected from the plurality of multimodal data; Using the initial training sample set to perform at least one round of iterative optimization on the target model, so as to use the iteratively optimized target model as the model to be trained; Based on the multiple multimodal data and their respective capability labels, supervised training is performed on the model to be trained.

8. A sample screening method, applied to a multimodal information processing model trained by any one of the methods of claims 1 to 7, comprising: Get the target dataset; Inputting the target data set into the multimodal information processing model to obtain capability labels of data samples in the target data set output by the multimodal information processing model; Based on the capability labels of the data samples in the target data set, training samples for training the target model are screened out from the target data set.

9. The method according to claim 8, wherein: The step of screening out training samples for training a target model from the target data set based on the capability labels of the data samples in the target data set includes: Filter out data samples whose capability quantity included in the capability label meets the first preset requirement from the target data set as the training samples; and / or, Based on the capability label, data samples whose proportion of the total amount of training samples in the target data set meets a second preset requirement are screened out from the target data set as the training samples.

10. A training device for a multimodal information processing model, comprising: A labeling module, used to label the multimodal data based on the multimodal large model to obtain a capability label of the multimodal data; The capability label is used to describe the model capability that can be trained by the multimodal data; A training module is used to perform supervised training on the model to be trained based on the multimodal data and the capability label, so as to train the model to be trained into a multimodal information processing model that can filter out useful samples for the target model.

11. The device according to claim 10, wherein the marking module comprises: A construction unit, configured to construct input information of the multimodal large model based on the multimodal data and a preset prompt word template; A labeling unit is used to input the input information into the multimodal big model to obtain a capability label that the multimodal big model labels for the multimodal data.

12. The device according to claim 11, wherein The preset prompt word template includes at least one of the following requirements: Require the multimodal large model to pay attention to the detailed content in the image modality information in the multimodal data; In the case where the multimodal data includes a question-answer pair, the multimodal large model is required to understand the query intent of the question in the question-answer pair for the image modality information in the multimodal data in combination with the question-answer pair; The multimodal large model is required to analyze and reason about the input information to provide reasonable capability labels for the multimodal data.

13. The device according to claim 10, wherein: The model structure of the model to be trained is the same as the model structure of the target model; or, The model to be trained is a student model of the target model.

14. According to the device of claim 10, the complexity of the model structure of the multimodal large model is greater than that of the model to be trained.

15. The device according to claim 10, wherein: The training module comprises: A prediction unit, configured to input the multimodal data into the model to be trained, and obtain a prediction label output by the model to be trained for the multimodal data; a determining unit, configured to determine a prediction loss based on the prediction label and the capability label; The first optimization unit is used to adjust the model parameters of the model to be trained based on the prediction loss.

16. The device according to claim 10, wherein: The training module comprises: A screening unit, configured to screen out an initial training sample set from the plurality of multimodal data based on the capability labels respectively labeled by the multimodal large model for the plurality of multimodal data; A second optimization unit, configured to perform at least one round of iterative optimization on the target model using the initial training sample set, so as to use the iteratively optimized target model as the model to be trained; A training unit is used to perform supervised training on the model to be trained based on the multiple multimodal data and their respective capability labels.

17. A training sample screening device, applied to a multimodal information processing model trained by the device of any one of claims 10 to 16, comprising: An acquisition module is used to acquire a target data set; an output module, configured to input the target data set into the multimodal information processing model, and obtain capability labels of data samples in the target data set output by the multimodal information processing model; The screening module is used to screen out training samples for training a target model from the target data set based on the capability labels of the data samples in the target data set.

18. The device according to claim 17, wherein: The screening module comprises: A first screening unit is used to screen out data samples whose capability quantity included in the capability label meets a first preset requirement from the target data set as the training samples; and / or, The second screening unit is used to screen out data samples whose proportion of the total amount of training samples in the target data set meets a second preset requirement from the target data set based on the capability label, as the training samples.

19. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 9.

20. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-9.

21. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 9.

Images