Multi-modal learning method and device based on element weight regulator, equipment and medium

Abstract

The invention provides a multi-modal learning method and device based on a meta-weight regulator, equipment and a medium, and relates to the technical field of model training. According to the method, the first gradient descent direction of the source data sample and the second gradient descent direction of the target data sample are compared through the meta weight regulator to calculate the direction similarity, and the part, similar to the target data in distribution, in the source data can be recognized and enhanced. The weights of the source data samples are adjusted based on the direction similarity, so that the model pays more attention to the source samples similar to the target data in distribution during training, the distribution difference between the source data and the target data is effectively reduced, and the adaptability of the model to the target data is improved. By continuously adjusting the source weight and calculating the loss until the model loss difference value of the source data set and the target data set is lower than the preset threshold value, the retrieval accuracy of the multi-modal learning model in a business scene with less data volume is effectively improved.

Description

Technical Field

[0001] The present application relates to the field of model training technology, and in particular to a multimodal learning method, device, equipment and medium based on a meta-weight regulator. Background Art

[0002] Multimodal Machine Learning studies machine learning problems involving data in different modalities. Multimodal learning methods based on deep learning continue to develop and have been widely used in fields such as image caption and visual question answering, with good results.

[0003] Few-shot learning aims to learn with a limited number of labeled examples, which can reduce the burden of collecting and annotating large-scale labeled data. Multimodal data is more expensive to label than unimodal data, and the amount of data is smaller. In many cases, only a limited amount of labeled data is available for the target data or task, while there is a large amount of source data or tasks that are different from the target data. It is challenging to adapt the model trained on these rich source data to a small number of target data sets or tasks. If the difference in data distribution between the two cannot be well bridged, the model will not be well adapted.

[0004] In the financial field, multimodality plays an important role in business scenarios such as banking and insurance. For example, the text retrieval image used in the image-text video conversion of the algorithm platform (life insurance marketing) is solved by retrieving the image through the text and finally generating a marketing video. However, the current general multimodal retrieval models are all trained on general data sets, which is very different from the business scenarios in the financial field, and the retrieval recall rate is relatively low.

[0005] Therefore, how to improve the retrieval accuracy of multimodal learning models has become a technical problem that needs to be solved urgently. Summary of the invention

[0006] The present application provides a multimodal learning method, apparatus, device and medium based on a meta-weight regulator, aiming to improve the retrieval accuracy of a multimodal learning model.

[0007] In a first aspect, the present application provides a multimodal learning method based on a meta-weight regulator, and the multimodal learning method based on a meta-weight regulator comprises the following steps:

[0008] Get the source data set and the target data set;

[0009] Based on the meta-weight regulator, comparing the first gradient descent direction of each source data sample in the source data set with the second gradient descent direction of the target data sample in the target data set to obtain the direction similarity corresponding to each source data sample;

[0010] Based on the directional similarity, adjusting the source weight of each source data sample in the source data set, and calculating the first model loss corresponding to the source data set after the source weight is adjusted;

[0011] When a loss difference between the first model loss and a second model loss corresponding to the target data set is less than a preset loss difference threshold, model parameters of the multimodal learning model are determined based on the first model loss.

[0012] In a second aspect, the present application further provides a multimodal learning device based on a meta-weight regulator, wherein the multimodal learning device based on a meta-weight regulator comprises:

[0013] A data acquisition module, used to acquire a source data set and a target data set;

[0014] a direction similarity determination module, configured to compare, based on a meta-weight regulator, a first gradient descent direction of each source data sample in the source data set and a second gradient descent direction of a target data sample in the target data set to obtain a direction similarity corresponding to each source data sample;

[0015] A source weight adjustment module, used to adjust the source weight of each source data sample in the source data set based on the directional similarity, and calculate the first model loss corresponding to the source data set after the source weight is adjusted;

[0016] A model parameter determination module is used to determine the model parameters of the multimodal learning model based on the first model loss when the loss difference between the first model loss and the second model loss corresponding to the target data set is less than a preset loss difference threshold.

[0017] In a third aspect, the present application also provides a computer device, comprising a processor, a memory, and a computer program stored in the memory and executable by the processor, wherein when the computer program is executed by the processor, the steps of the multimodal learning method based on the meta-weight regulator as described above are implemented.

[0018] In a fourth aspect, the present application also provides a computer-readable storage medium, on which a computer program is stored, wherein when the computer program is executed by a processor, the steps of the multimodal learning method based on the meta-weight regulator as described above are implemented.

[0019] The present application provides a multimodal learning method, device, computer equipment and storage medium based on a meta-weight regulator. The method of the present application includes obtaining a source data set and a target data set; based on the meta-weight regulator, comparing the first gradient descent direction of each source data sample in the source data set with the second gradient descent direction of the target data sample in the target data set to obtain the directional similarity corresponding to each source data sample; based on the directional similarity, adjusting the source weight of each source data sample in the source data set, and calculating the first model loss corresponding to the source data set after the source weight is adjusted; when the loss difference between the first model loss and the second model loss corresponding to the target data set is less than a preset loss difference threshold, determining the model parameters of the multimodal learning model based on the first model loss. In the above manner, the present application compares the first gradient descent direction of the source data sample with the second gradient descent direction of the target data sample through the meta-weight regulator to calculate the directional similarity, which helps to identify and strengthen the portion of the source data that is similar to the target data in distribution. By adjusting the weights of source data samples based on directional similarity, the model pays more attention to source samples with similar distribution to the target data during training, effectively narrowing the distribution difference between the source data and the target data and improving the adaptability of the model on the target data. By continuously adjusting the source weights and calculating the loss until the difference in model loss between the source and target datasets is below the preset threshold, the performance of the model on the target data is ensured to be comparable to that of the model trained on a large amount of source data, effectively improving the retrieval accuracy of the multimodal learning model in business scenarios with less data. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0021] Figure 1 A schematic diagram of a flow chart of a first embodiment of a multimodal learning method based on a meta-weight regulator provided in the present application;

[0022] Figure 2 A schematic diagram of a process for training a multimodal learning model based on a meta-weight regulator provided in an embodiment of the present application;

[0023] Figure 3 It is a structural schematic diagram of a first embodiment of a multimodal learning device based on a meta-weight regulator provided by the present application;

[0024] Figure 4 It is a schematic block diagram of the structure of a computer device provided in an embodiment of the present application.

[0025] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0026] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0027] The flowcharts shown in the accompanying drawings are only examples and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may also be decomposed, combined or partially merged, so the actual execution order may change according to actual conditions.

[0028] In conjunction with the accompanying drawings, some embodiments of the present application are described in detail below. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0029] Please refer to Figure 1 , Figure 1 A schematic flow chart of a first embodiment of a multimodal learning method based on a meta-weight regulator provided in the present application.

[0030] like Figure 1 As shown, the multimodal learning method based on the meta-weight regulator includes steps S101 to S104.

[0031] S101, obtaining a source data set and a target data set;

[0032] In one embodiment, the sample distribution of the source data set is inconsistent with that of the target data set. There are a large number of labeled samples in the source data set, which can be used for conventional deep learning training; while there are only a small number of labeled samples in the target data set. It can be understood that the source data samples in the source data set are used to train and build the initial model, and the target data samples in the target data set are the new data sets to which the model will be applied. The knowledge and features learned in the source data set can be utilized through transfer learning to obtain good performance on the target task.

[0033] In one embodiment, the source data set may be denoted as S Train , the target data set is recorded as T F Because there are a large number of labeled samples in the source dataset, which can be used to train the deep learning model, while there are only a small number of labeled samples in the target dataset, which are not enough to train the model. Therefore, T F For STrain The trained model is adjusted so that the model can adapt well to the target distribution after training.

[0034] For example, a model can be trained on a large number of image-text pair datasets, and a small amount of image description data can be used to adjust the training process. The adaptability of the model from image-text retrieval (source data) to image description (target data) can then be evaluated. Alternatively, the imbalance problem of too few images of certain types can be solved by adjusting the weights of different types of images in image descriptions.

[0035] S102, based on the meta-weight regulator, comparing the first gradient descent direction of each source data sample in the source data set with the second gradient descent direction of the target data sample in the target data set to obtain the direction similarity corresponding to each source data sample;

[0036] Generally, in transfer learning, there is a situation where the sample distribution of the source dataset and the target dataset is inconsistent. In this case, the model parameters can be adjusted to better adapt to the target dataset.

[0037] In one embodiment, a meta weight regulator is a strategy used in transfer learning that improves the performance of a model on a target dataset by adjusting the weights of samples in a source dataset. The core idea of ​​this method is to obtain the direction similarity corresponding to the source data samples by comparing the gradient descent directions of samples in the source dataset and the target dataset, and then adjust the weight of the source samples in the loss function.

[0038] In one embodiment, during the model training process, when calculating the loss function of the model, a meta-weight adjuster can be used to compare the gradient descent direction of the model to adjust the weight of the source sample in the loss function, and finally calculate the weighted source loss.

[0039] During the conditioning process, when the gradient descent directions of the source and target samples are similar, the source weight is increased, and vice versa, so that the model trained on the source data learns to minimize the target loss. The meta-weight conditioning process can be viewed as utilizing T at the gradient level. F For S Train The process of performing soft data selection.

[0040] In another embodiment, the influence of different data sets on model training can be balanced by adjusting the weights of different parts in the loss function. For example, if there are multiple loss functions, the weights of these losses can be manually adjusted to keep them consistent in magnitude, thereby preventing a certain loss function from becoming dominant due to excessive weight setting.

[0041] Furthermore, based on the meta-weight regulator, a source weight is configured for each source data sample in the source data set to obtain a weighted source data set; the loss function corresponding to each source data sample in the weighted source data set is calculated to obtain the first gradient descent direction; the loss function of the target data sample in the target data set is calculated to obtain the second gradient descent direction; the similarity between the first gradient descent direction and the second gradient descent direction is calculated to obtain the direction similarity.

[0042] In one embodiment, if Figure 2 As shown in Figure 1, the meta-weight regulator first assigns a weight to each source sample and associates the source weight with the target loss in the computation graph. Next, the source weights are updated by gradient descent through the second-order derivative of the target loss. Through subsequent calculations, the weights of source data that are beneficial to the model's performance on the target distribution are increased. Finally, the model is trained on the weighted source samples and the model parameters are updated according to the weighted source loss.

[0043] Generally speaking, gradient descent is a key algorithm for optimizing model parameters. It iteratively updates model parameters and optimizes in the opposite direction of the gradient (i.e., the direction of fastest descent) to minimize the loss function. When training on the source dataset, a large number of labeled samples can be used to calculate the gradient and update the model parameters. On the target dataset, due to the small number of labeled samples, some special strategies such as transfer learning or meta-weight regulators may be needed to adjust the model parameters to better adapt them to the target dataset.

[0044] In one embodiment, the source weight of each sample in the source data set can be initialized according to prior knowledge or random initialization, and the weight of each sample can be dynamically adjusted according to the difference in feature distribution between the source data set and the target data set through the meta-weight regulator.

[0045] Furthermore, based on the current source weight, a first loss corresponding to each source data sample in the source data set is calculated; a weighted sum is performed on the first losses corresponding to each source data sample in the source data set to obtain the first model loss.

[0046] In one embodiment, the configured source weights are used to calculate the loss function of each sample in the weighted source data set, and the losses of all samples are weighted and summed to obtain the total loss function of the entire source data set. The total loss function is derived with respect to the model parameters to obtain the gradient. The gradient indicates how to adjust the parameters in the parameter space to minimize the loss function, i.e., the first gradient descent direction.

[0047] Exemplarily, the loss function calculation formula may be:

[0048]

[0049] Among them, Loss s (y,l) represents the source data loss, Loss t (y t ,l t ) is the target data loss, is the source weight.

[0050] In one embodiment, the loss function may be a cross entropy loss, a mean square error loss, etc., depending on the nature of the task (eg, classification, regression, etc.).

[0051] In one embodiment, because there are fewer samples in the target data set, the loss function can be directly calculated for the samples in the target data set. The loss function of the target data set is derived with respect to the model parameters to obtain a gradient. The gradient indicates how to adjust the parameters in the parameter space to minimize the loss function of the target data set, i.e., the second gradient descent direction.

[0052] In one embodiment, the directional similarity between the first gradient descent direction and the second gradient descent direction is calculated by, for example, using a dot product, cosine similarity, or Euclidean distance.

[0053] The dot product of two gradient vectors is used as a measure of their directional similarity. Alternatively, the cosine of the angle between two gradient vectors is calculated as a measure of their directional similarity. Alternatively, the Euclidean distance between two gradient vectors is calculated. The smaller the distance, the more similar the directions.

[0054] S103, adjusting the source weight of each source data sample in the source data set based on the directional similarity, and calculating the first model loss corresponding to the source data set after the source weight is adjusted;

[0055] In one embodiment, the source weights of the source data samples may be further adjusted based on the directional similarity, so that the model is more inclined to learn feature representations that are useful for the target data set, thereby improving the performance of the model on the target task.

[0056] Further, when the directional similarity is greater than a first similarity threshold, the source weight of the corresponding source data sample is increased. When the directional similarity is less than the first similarity threshold, the source weight of the corresponding source data sample is decreased.

[0057] In one embodiment, the weight of each sample in the source data set is adjusted according to the calculated directional similarity. If the gradient descent direction of the source sample is highly similar to the gradient descent direction of the target sample, the weight of the source sample is increased; if the similarity is low, the weight of the source sample is reduced. The directional similarity is measured by a first similarity threshold, which can be determined according to the similarity distribution or according to prior knowledge.

[0058] Exemplarily, it can be achieved by a linear adjustment formula, for example:

[0059] ω new =ω old ×(1+α×similarity)

[0060] Among them, ω new is the new weight, ω old is the old weight, α is the adjustment coefficient, which is used to control the magnitude of the weight adjustment, and similarity is the direction similarity.

[0061] In another embodiment, when the directional similarity is less than a second similarity threshold, the source weight of each of the source data samples in the source data set is adjusted until the directional similarity is greater than or equal to the second similarity threshold; based on the current source weight, the first model loss corresponding to the source data set is calculated; based on the first model loss, the model parameters of the multimodal learning model are determined.

[0062] In one embodiment, after calculating the directional similarity of the gradient descent directions on the source data set and the target data set, the directional similarity can be directly compared with a preset second similarity threshold. If the directional similarity is less than the second similarity threshold, the source weight of each source data sample in the source data set can be adjusted by a meta-weight adjuster or other optimization algorithm to increase the similarity of the gradient descent directions of the source sample and the target sample.

[0063] The process of adjusting the source weight may be iterative until the direction similarity is greater than or equal to the second similarity threshold.

[0064] In one embodiment, the adjusted source weights are used to calculate a first model loss corresponding to the source data set, and the first model loss is then used to update the model parameters.

[0065] Furthermore, based on the target data set, the target model loss when the target data set is used to train the multimodal learning model is calculated; based on the association formula between the target model loss and the source weight, the current source weight is calculated; based on the current source weight, the first model loss of the source data set when training the multimodal learning model is calculated.

[0066] In one embodiment, the multimodal learning model is trained using the target data set, and the loss function value of the model on the target data set is calculated, that is, the target model loss (L target ). According to the association formula between the target model loss and the source weight, update the current source weight ω old , get the updated source weight ω new .

[0067] The associated formula can be used to express the difference in loss, the similarity of the gradient descent direction, or other metrics that reflect the difference between the source and target datasets. For example, if the gradient descent directions of the source and target samples are similar, the weight of the source sample may increase, and vice versa.

[0068] Optionally, the adjusted weights can be normalized to ensure that their sum is 1 (if it is a probability distribution) or within a reasonable range. This can be achieved by the following formula:

[0069]

[0070] Among them, ω normalized represents the normalized source weight, and N is the total number of samples in the source dataset.

[0071] In one embodiment, after the source weights are adjusted, the updated source weights are used to train a multimodal learning model based on the source data set, and a first model loss of the model on the source data set is calculated.

[0072] Specifically, the calculation formula of the first model loss can be expressed as:

[0073]

[0074] Among them, L new is the first model loss, L i is the loss of the ith sample.

[0075] Understandably, in transfer learning, in addition to the meta-weight regulator, other methods can be used to improve the performance of deep learning models on the target dataset, such as fine-tuning, feature extraction, domain adaptation, multi-task learning, transfer model integration, data enhancement, model pruning, hyperparameter optimization, teacher-student learning, cross-modal transfer learning, zero-shot and few-shot learning, etc.

[0076] For example, in fine-tuning, a pre-trained model is used to initialize the model of the target task, and some or all layers of the model are frozen, and then the unfrozen layers are further trained with training data to adapt to the specific data of the target task. In this way, the features and knowledge learned by the pre-trained model on the source task can be used to improve the generalization ability of the model on the target task.

[0077] For example, multi-task learning is a method of learning multiple related tasks simultaneously in one model. By sharing the parameters and feature representations of the model, the generalization ability of the model for different tasks can be improved. Multi-task learning can effectively avoid overfitting and data scarcity problems, and can improve the performance of the model by learning the correlation between different tasks.

[0078] For example, teacher-student learning can be achieved by having a larger, more complex "teacher" model teach a smaller, simpler "student" model. The student model learns to imitate the behavior of the teacher model, thereby effectively transferring knowledge.

[0079] In one embodiment, the meta-weight mediator can be used in combination with other transfer learning techniques, such as domain adaptation, multi-task learning, etc., to further improve the performance of the model on the target dataset.

[0080] For example, in transfer learning based on deep neural networks, one of the following three methods can be used: transfer learning, feature vector extraction, and fine-tuning. These methods can be used alone or in combination with technologies such as meta-weight regulators to adapt to different transfer learning scenarios and needs.

[0081] S104. When a loss difference between the first model loss and the second model loss corresponding to the target data set is less than a preset loss difference threshold, determine model parameters of the multimodal learning model based on the first model loss.

[0082] In one embodiment, when the difference between the first model loss (loss on the source data set) and the second model loss (loss on the target data set) is less than a preset loss difference threshold, this indicates that the performance of the model on the two data sets is close enough, and the model parameters can be determined based on the first model loss.

[0083] Specifically, a loss difference threshold is set to determine whether the performance of the model on the source dataset and the target dataset is close enough. The model loss function on the source dataset and the target dataset is calculated by forward propagating the model parameters and the samples in the dataset. The difference between the two losses is calculated and compared with the preset loss difference threshold. If the loss difference is less than the threshold, it means that the performance of the model on the two datasets is close enough, and the model parameters can be determined. If the loss difference is greater than the threshold, the model parameters may need to be further adjusted, such as by fine-tuning, optimizing the network structure, or adjusting the weights in the loss function.

[0084] In one embodiment, when the loss difference between the first model loss and the second model loss corresponding to the target data set is less than a preset loss difference threshold, the model parameters are updated by a back propagation algorithm according to the first model loss. Specifically, the gradient of the loss function with respect to the model parameters is calculated, and then the parameters are updated according to the gradient descent algorithm.

[0085] The present embodiment provides a multimodal learning method based on a meta-weight regulator. The method of the present application compares the first gradient descent direction of the source data sample with the second gradient descent direction of the target data sample through the meta-weight regulator to calculate the directional similarity, which helps to identify and strengthen the part of the source data that is similar to the target data in distribution. The weights of the source data samples are adjusted based on the directional similarity, so that the model pays more attention to those source samples with similar distribution to the target data during training, effectively narrowing the distribution difference between the source data and the target data, and improving the adaptability of the model on the target data. By continuously adjusting the source weights and calculating the loss until the difference in model loss between the source data set and the target data set is lower than the preset threshold, it is ensured that the performance of the model on the target data is comparable to that of the model trained on a large amount of source data, effectively improving the retrieval accuracy of the multimodal learning model in business scenarios with less data.

[0086] See also Figure 3 , Figure 3 It is a structural schematic diagram of a first embodiment of a multimodal learning device based on a meta-weight regulator provided in the present application. The multimodal learning device based on a meta-weight regulator is used to execute the aforementioned multimodal learning method based on a meta-weight regulator.

[0087] like Figure 3 As shown, the multimodal learning device 200 based on the meta-weight regulator includes: a data acquisition module 201, a direction similarity determination module 202, a source weight adjustment module 203 and a model parameter determination module 204.

[0088] The data acquisition module 201 is used to acquire a source data set and a target data set;

[0089] A direction similarity determination module 202 is used to compare the first gradient descent direction of each source data sample in the source data set with the second gradient descent direction of the target data sample in the target data set based on a meta-weight regulator to obtain the direction similarity corresponding to each source data sample;

[0090] A source weight adjustment module 203, used to adjust the source weight of each source data sample in the source data set based on the directional similarity, and calculate the first model loss corresponding to the source data set after the source weight is adjusted;

[0091] The model parameter determination module 204 is used to determine the model parameters of the multimodal learning model based on the first model loss when the loss difference between the first model loss and the second model loss corresponding to the target data set is less than a preset loss difference threshold.

[0092] In one embodiment, the direction similarity determination module 202 includes:

[0093] A source data weighting unit, configured to configure a source weight for each source data sample in the source data set based on the meta-weight adjuster, to obtain a weighted source data set;

[0094] A first gradient descent direction calculation unit, used to calculate the loss function corresponding to each source data sample in the weighted source data set to obtain the first gradient descent direction;

[0095] A second gradient descent direction calculation unit, used to calculate the loss function of the target data sample in the target data set to obtain the second gradient descent direction;

[0096] A direction similarity calculation unit is used to calculate the similarity between the first gradient descent direction and the second gradient descent direction to obtain the direction similarity.

[0097] In one embodiment, the source weight adjustment module 203 includes:

[0098] The source weight increasing unit is used to increase the source weight of the corresponding source data sample when the directional similarity is greater than a first similarity threshold.

[0099] In one embodiment, the source weight adjustment module 203 further includes:

[0100] A source weight reducing unit is used to reduce the source weight of the corresponding source data sample when the direction similarity is less than the first similarity threshold.

[0101] In one embodiment, the source weight adjustment module 203 further includes:

[0102] A target model loss calculation unit, used to calculate the target model loss when the target data set is used to train the multimodal learning model based on the target data set;

[0103] A current source weight calculation unit, used to calculate the current source weight based on the association formula between the target model loss and the source weight;

[0104] The first model loss calculation unit is used to calculate the first model loss of the source data set for training the multimodal learning model based on the current source weight.

[0105] In one embodiment, the first model loss calculation unit includes:

[0106] A first loss calculation subunit, configured to calculate a first loss corresponding to each source data sample in the source data set based on the current source weight;

[0107] The first model loss calculation subunit is used to perform weighted summation of the first losses corresponding to each source data sample in the source data set to obtain the first model loss.

[0108] In one embodiment, the multimodal learning device 200 based on the meta-weight regulator further includes a second module for adjusting model parameters, including:

[0109] a weight adjustment unit, configured to adjust the source weight of each of the source data samples in the source data set when the directional similarity is less than a second similarity threshold, until the directional similarity is greater than or equal to the second similarity threshold;

[0110] A second model loss calculation unit, configured to calculate the first model loss corresponding to the source data set based on a current source weight;

[0111] A model parameter determination unit is used to determine the model parameters of the multimodal learning model based on the first model loss.

[0112] It should be noted that technical personnel in the relevant field can clearly understand that, for the convenience and conciseness of description, the specific working process of the above-described device and each module can refer to the corresponding process in the aforementioned embodiment of the multimodal learning method based on meta-weight regulator, and will not be repeated here.

[0113] The apparatus provided in the above embodiment may be implemented in the form of a computer program. The computer program may be Figure 4 Runs on the computer device shown.

[0114] See also Figure 4 , Figure 4 1 is a schematic block diagram of a computer device provided in an embodiment of the present application. The computer device may be a server.

[0115] See also Figure 4 The computer device includes a processor, a memory and a network interface connected through a system bus, wherein the memory may include a non-volatile storage medium and an internal memory.

[0116] The non-volatile storage medium can store an operating system and a computer program. The computer program includes program instructions, and when the program instructions are executed, the processor can execute any multimodal learning method based on a meta-weight regulator.

[0117] The processor is used to provide computing and control capabilities and support the operation of the entire computer equipment.

[0118] The internal memory provides an environment for the operation of a computer program in a non-volatile storage medium. When the computer program is executed by a processor, the processor can execute any multimodal learning method based on a meta-weight regulator.

[0119] The network interface is used for network communication, such as sending assigned tasks, etc. Those skilled in the art will appreciate that Figure 4 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0120] It should be understood that the processor may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0121] In one embodiment, the processor is used to run a computer program stored in the memory to implement the following steps:

[0122] Get the source data set and the target data set;

[0123] Based on the meta-weight regulator, comparing the first gradient descent direction of each source data sample in the source data set with the second gradient descent direction of the target data sample in the target data set to obtain the direction similarity corresponding to each source data sample;

[0124] Based on the directional similarity, adjusting the source weight of each source data sample in the source data set, and calculating the first model loss corresponding to the source data set after the source weight is adjusted;

[0125] When a loss difference between the first model loss and a second model loss corresponding to the target data set is less than a preset loss difference threshold, model parameters of the multimodal learning model are determined based on the first model loss.

[0126] In one embodiment, when the processor implements the meta-weight regulator, compares the first gradient descent direction of each source data sample in the source data set with the second gradient descent direction of the target data sample in the target data set, and obtains the direction similarity corresponding to each source data sample, it is used to implement:

[0127] Based on the meta-weight regulator, a source weight is respectively configured for each source data sample in the source data set to obtain a weighted source data set;

[0128] Calculating the loss function corresponding to each source data sample in the weighted source data set to obtain the first gradient descent direction;

[0129] Calculating the loss function of the target data sample in the target data set to obtain the second gradient descent direction;

[0130] The similarity between the first gradient descent direction and the second gradient descent direction is calculated to obtain the direction similarity.

[0131] In one embodiment, when the processor implements adjusting the source weight of each source data sample in the source data set based on the directional similarity, it is configured to implement:

[0132] When the directional similarity is greater than a first similarity threshold, the source weight of the corresponding source data sample is increased.

[0133] In one embodiment, when the processor implements adjusting the source weight of each source data sample in the source data set based on the directional similarity, the processor is further configured to implement:

[0134] When the directional similarity is less than the first similarity threshold, the source weight of the corresponding source data sample is reduced.

[0135] In one embodiment, when the processor implements the step of calculating the first model loss of the multimodal learning model obtained by training the source data set, the processor is used to implement:

[0136] Based on the target data set, calculating the target model loss when the target data set is used to train the multimodal learning model;

[0137] Calculating a current source weight based on a formula associating the target model loss with the source weight;

[0138] Based on the current source weights, a first model loss for training the multimodal learning model with the source data set is calculated.

[0139] In one embodiment, when the processor implements the step of calculating the first model loss of the source data set for training the multimodal learning model based on the current source weight, the processor is configured to implement:

[0140] Based on the current source weight, calculating a first loss corresponding to each source data sample in the source data set;

[0141] A weighted sum is performed on the first losses corresponding to each source data sample in the source data set to obtain the first model loss.

[0142] In one embodiment, after implementing the meta-weight regulator, comparing the first gradient descent direction of each source data sample in the source data set with the second gradient descent direction of the target data sample in the target data set to obtain the direction similarity corresponding to each source data sample, the processor is further configured to implement:

[0143] When the directional similarity is less than a second similarity threshold, adjusting the source weight of each of the source data samples in the source data set until the directional similarity is greater than or equal to the second similarity threshold;

[0144] Based on the current source weight, calculating the first model loss corresponding to the source data set;

[0145] Based on the first model loss, model parameters of the multimodal learning model are determined.

[0146] A computer-readable storage medium is also provided in an embodiment of the present application, wherein the computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and the processor executes the program instructions to implement any multimodal learning method based on a meta-weight regulator provided in an embodiment of the present application.

[0147] The computer-readable storage medium may be an internal storage unit of the computer device described in the foregoing embodiment, such as a hard disk or memory of the computer device. The computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, a smart memory card (SmartMedia Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc., equipped on the computer device.

[0148] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A multimodal learning method based on a meta-weight regulator, characterized in that: The method comprises: Get the source data set and the target data set; Based on the meta-weight regulator, comparing the first gradient descent direction of each source data sample in the source data set with the second gradient descent direction of the target data sample in the target data set to obtain the direction similarity corresponding to each source data sample; Based on the directional similarity, adjusting the source weight of each source data sample in the source data set, and calculating the first model loss corresponding to the source data set after the source weight is adjusted; When a loss difference between the first model loss and a second model loss corresponding to the target data set is less than a preset loss difference threshold, model parameters of the multimodal learning model are determined based on the first model loss.

2. The multimodal learning method based on meta-weight regulator according to claim 1, characterized in that: The method of comparing the first gradient descent direction of each source data sample in the source data set with the second gradient descent direction of the target data sample in the target data set based on the meta-weight regulator to obtain the direction similarity corresponding to each source data sample includes: Based on the meta-weight regulator, a source weight is respectively configured for each source data sample in the source data set to obtain a weighted source data set; Calculating the loss function corresponding to each source data sample in the weighted source data set to obtain the first gradient descent direction; Calculating the loss function of the target data sample in the target data set to obtain the second gradient descent direction; The similarity between the first gradient descent direction and the second gradient descent direction is calculated to obtain the direction similarity.

3. The multimodal learning method based on meta-weight regulator according to claim 1, characterized in that: The adjusting the source weight of each source data sample in the source data set based on the directional similarity includes: When the directional similarity is greater than a first similarity threshold, the source weight of the corresponding source data sample is increased.

4. The multimodal learning method based on meta-weight regulator according to claim 3, characterized in that: The adjusting the source weight of each source data sample in the source data set based on the directional similarity further includes: When the directional similarity is less than the first similarity threshold, the source weight of the corresponding source data sample is reduced.

5. The multimodal learning method based on meta-weight regulator according to claim 1, characterized in that: The calculating a first model loss of the multimodal learning model obtained by training the source data set includes: Based on the target data set, calculating the target model loss when the target data set is used to train the multimodal learning model; Calculating a current source weight based on a formula associating the target model loss with the source weight; Based on the current source weights, a first model loss for training the multimodal learning model with the source data set is calculated.

6. The multimodal learning method based on meta-weight regulator according to claim 5, characterized in that: The calculating, based on the current source weight, a first model loss of the source data set for training the multimodal learning model comprises: Based on the current source weight, calculating a first loss corresponding to each source data sample in the source data set; A weighted sum is performed on the first losses corresponding to each source data sample in the source data set to obtain the first model loss.

7. The multimodal learning method based on meta-weight regulator according to claim 1, characterized in that: After the first gradient descent direction of each source data sample in the source data set is compared with the second gradient descent direction of the target data sample in the target data set based on the meta-weight regulator to obtain the direction similarity corresponding to each source data sample, the method further includes: When the directional similarity is less than a second similarity threshold, adjusting the source weight of each of the source data samples in the source data set until the directional similarity is greater than or equal to the second similarity threshold; Based on the current source weight, calculating the first model loss corresponding to the source data set; Based on the first model loss, model parameters of the multimodal learning model are determined.

8. A multimodal learning device based on a meta-weight regulator, characterized in that: The multimodal learning device based on the meta-weight regulator comprises: A data acquisition module, used to acquire a source data set and a target data set; a direction similarity determination module, configured to compare, based on a meta-weight regulator, a first gradient descent direction of each source data sample in the source data set and a second gradient descent direction of a target data sample in the target data set to obtain a direction similarity corresponding to each source data sample; A source weight adjustment module, used to adjust the source weight of each source data sample in the source data set based on the directional similarity, and calculate the first model loss corresponding to the source data set after the source weight is adjusted; A model parameter determination module is used to determine the model parameters of the multimodal learning model based on the first model loss when the loss difference between the first model loss and the second model loss corresponding to the target data set is less than a preset loss difference threshold.

9. A computer device, characterized in that: The computer device includes a processor, a memory, and a computer program stored in the memory and executable by the processor, wherein when the computer program is executed by the processor, the steps of the multimodal learning method based on the meta-weight regulator as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, the steps of the multimodal learning method based on a meta-weight regulator as described in any one of claims 1 to 7 are implemented.

Images