Multimodal contrastive learning model training methods, devices, electronic equipment, and media
The multimodal contrastive learning model, constructed using a sparse hybrid expert model and a router, solves the catastrophic forgetting problem of dense models and achieves efficient training results for multi-task processing.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- TERMINUSBEIJING TECH CO LTD
- Filing Date
- 2023-03-10
- Publication Date
- 2026-05-26
AI Technical Summary
Existing multimodal contrastive learning models are typically dense models, which suffer from catastrophic forgetting problems and deteriorate in training performance on early tasks as new tasks are added, making it difficult to efficiently handle multiple tasks.
A sparse hybrid expert model is adopted, which uses routers and multilayer perceptrons to build expert layers, activates only some expert processing features, and combines a single-tower structure and a contrastive learning network to optimize the model structure through sparsification.
It achieves the effect of handling multiple tasks in a single model, while avoiding catastrophic forgetting, thus improving work efficiency and training results.
Smart Images
Figure CN116341678B_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to the field of computer technology, and in particular to a multimodal contrastive learning model training method and apparatus, electronic device, and medium. Background Technology
[0002] Pathways (a Google general AI architecture proposed by Jeff Dean in 2021) encapsulates the research vision of building a single, large-scale model capable of handling thousands of tasks and numerous data patterns. To date, considerable progress has been made in sparse unimodal models for text (Switch, Task-MoE, GLaM) and computer vision (Vision MoE).
[0003] Existing multimodal contrastive learning models are almost all dense models. However, dense models are often subject to negative perturbations. For example, when multiple tasks need to be processed, task diversity means that only one model is trained for each task. Even if the model is used for multi-task learning, there may be a catastrophic forgetting problem. Furthermore, as new tasks are added, the training performance of the model on the early tasks will become worse. Summary of the Invention
[0004] This disclosure aims to address at least one of the problems existing in the prior art by providing a method, apparatus, electronic device, and medium for training a multimodal contrastive learning model.
[0005] One aspect of this disclosure provides a method for training a multimodal contrastive learning model, the training method comprising:
[0006] Acquire multimodal data, which includes text data and image data;
[0007] The multimodal data is processed to obtain serialized marker data corresponding to the text data and image patch data corresponding to the image data;
[0008] The serialized marker data and the image patch data are input into the feature encoder to obtain initial text features and initial image features;
[0009] The initial text features and the initial image features are input into a preset routing-based expert layer to obtain multimodal features;
[0010] The multimodal features are input into the encoding layer to obtain encoded text features and encoded image features;
[0011] The contrastive learning network is trained based on the encoded text features and the encoded image features to obtain a trained multimodal contrastive learning model.
[0012] Optionally, the expert layer includes a router, multiple feedforward network modules connected to the router, and multiple multilayer perceptrons connected to each feedforward network module in a one-to-one correspondence. Each feedforward network module includes multiple parallel feedforward networks.
[0013] The step of inputting the initial text features and the initial image features into a preset expert layer to obtain multimodal features includes:
[0014] The initial text features and the initial image features are input into the router to obtain the gating weights corresponding to the initial text features and the initial image features, respectively;
[0015] The gating weights are compared with a preset threshold to determine the target feedforward network;
[0016] The initial text features and the initial image features are input into the target feedforward network, and the output of the target feedforward network is input into the corresponding multilayer perceptron to obtain the multimodal features.
[0017] Optionally, comparing the gating weights with a preset threshold to determine the target feedforward network includes:
[0018] The gating weights are compared with preset thresholds corresponding to each feedforward network, and the feedforward network whose preset threshold is less than the gating weight is selected as the target feedforward network.
[0019] Optionally, the gating weight is calculated according to the following formula (1):
[0020] g(x) = softmax(Wx) (1)
[0021] Where x represents the initial text feature or the initial image feature, g(x) represents the function by which the gating unit processes x, W represents the weight of the gating unit, and softmax() represents the softmax function.
[0022] Optionally, the multimodal features are represented by the following equation (2):
[0023]
[0024] Where f(x) represents the multimodal feature corresponding to x, h = 1, ..., K represents the index of the feedforward network included in the target feedforward network, K represents the number of feedforward networks included in the target feedforward network, and g(x) h This represents the gated unit x corresponding to the h-th feedforward network, MLP h (x) represents the multilayer perceptron corresponding to the h-th feedforward network of x.
[0025] Optionally, the loss function of the contrastive learning network is expressed as equation (3):
[0026]
[0027] Among them, L infoNCE Let T represent the temperature, n represent the number of text-image data pairs consisting of the text data and the image data, k = 1, ..., n represent the numbers of the data pairs, and f represents the loss function of the contrastive learning network. ij f represents the features obtained from the j-th batch of images. tj f represents the features obtained from the j-th batch of text. ik Let f represent the features obtained from the k-th image. tk This represents the feature obtained from the k-th text.
[0028] Optionally, the dimensions of the encoded text features and the encoded image features are smaller than the dimensions of the multimodal features.
[0029] Another aspect of this disclosure provides a multimodal contrastive learning model training apparatus, the training apparatus comprising:
[0030] An acquisition module is used to acquire multimodal data, which includes text data and image data;
[0031] The processing module is used to process the multimodal data to obtain serialized marker data corresponding to the text data and image block data corresponding to the image data;
[0032] The first input module is used to input the serialized marker data and the image block data into the feature encoder to obtain initial text features and initial image features;
[0033] The second input module is used to input the initial text features and the initial image features into a preset routing-based expert layer to obtain multimodal features;
[0034] The third input module is used to input the multimodal features into the encoding layer to obtain encoded text features and encoded image features;
[0035] The training module is used to train the contrastive learning network based on the encoded text features and the encoded image features to obtain a trained multimodal contrastive learning model.
[0036] Another aspect of this disclosure provides an electronic device comprising:
[0037] At least one processor; and a memory communicatively connected to the at least one processor; wherein,
[0038] The memory stores instructions that can be executed by at least one processor, which enables the at least one processor to perform the multimodal contrastive learning model training method described above.
[0039] Another aspect of this disclosure provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the multimodal contrastive learning model training method described above.
[0040] Compared with the prior art, the embodiments disclosed herein apply a sparse hybrid expert model to an image-text multimodal contrastive learning model. By using a sparsification method, the task can be completed by activating only some experts in the expert layer for each input through routing. This allows a single model to handle multiple unimodal and multimodal tasks simultaneously. While achieving the effect of a single model handling a single task, it also avoids the catastrophic forgetting problem, thereby improving work efficiency and training effect. Attached Figure Description
[0041] One or more embodiments are illustrated by way of example with the corresponding pictures in the accompanying drawings. These illustrations do not constitute a limitation on the embodiments. Elements with the same reference numerals in the drawings are denoted as similar elements. Unless otherwise stated, the figures in the drawings are not to be limited by scale.
[0042] Figure 1 A flowchart illustrating a multimodal contrastive learning model training method provided in one embodiment of this disclosure;
[0043] Figure 2 A schematic diagram of the structure of a multimodal contrastive learning model provided in another embodiment of this disclosure;
[0044] Figure 3 A schematic diagram of the structure of a multimodal contrastive learning model training device provided for another embodiment of this disclosure;
[0045] Figure 4 A schematic diagram of the structure of an electronic device provided in another embodiment of this disclosure. Detailed Implementation
[0046] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the various embodiments of this disclosure will be described in detail below with reference to the accompanying drawings. However, those skilled in the art will understand that many technical details have been provided in the various embodiments of this disclosure to facilitate a better understanding of the disclosure. However, the technical solutions claimed in this disclosure can be implemented even without these technical details and with various variations and modifications based on the following embodiments. The division of the various embodiments below is for ease of description and should not constitute any limitation on the specific implementation of this disclosure. The various embodiments can be combined with and referenced by each other without contradiction.
[0047] One embodiment of this disclosure relates to a method for training a multimodal contrastive learning model, the process of which is as follows: Figure 1 As shown, it includes:
[0048] Step S110: Obtain multimodal data, which includes text data and image data.
[0049] Specifically, text data and image data can have content correlations, thus forming several text-image data pairs containing text and images, which can be used to train the multimodal contrastive learning model in subsequent steps.
[0050] Step S120: Process the multimodal data to obtain serialized marker data corresponding to the text data and image block data corresponding to the image data.
[0051] Specifically, for text data, this step can use the sentence piece algorithm (an open-source natural language processing toolkit from Google, an unsupervised text tokenization tool for neural network text generation systems) to obtain serialized token data corresponding to the text data. For image data, this step can process the image into ViT (Vision Transformer)-style image patches to obtain image patch data corresponding to the image data.
[0052] Step S130: Input the serialized marker data and image patch data into the feature encoder to obtain the initial text features and initial image features.
[0053] Specifically, multimodal contrastive learning is usually performed independently, using independent encoders for each modality to encode the data for each modality. Then, the features obtained from the encoders of each modality are used for contrastive learning between images and text. This type of model is often referred to as a "dual-tower" structure.
[0054] Combined Figure 2This implementation proposes using a single-tower feature encoder to encode data from each modality, meaning all modalities share a single feature encoder. In this step, serialized marker data and image patch data can be used as input tokens, fed into the shared feature encoder. Each token is encoded using this feature encoder to obtain the initial text features corresponding to the serialized marker data and the initial image features corresponding to the image patch data. The advantage of this single-tower feature encoder is that it provides greater versatility and scalability, and may also enable cross-modal and cross-task knowledge transfer.
[0055] Step S140: Input the initial text features and initial image features into a preset routing-based expert layer to obtain multimodal features.
[0056] Sparse hybrid expert models have recently begun to be used in multimodal vision and text models with good results. The main motivation for using hybrid expert models in multimodal vision and text models is to increase model parameters while controlling computational costs. At the same time, these models can also alleviate other problems to some extent; for example, sparsity can prevent catastrophic forgetting in continuous learning and can improve the performance of multi-task learning.
[0057] By studying sparse hybrid expert models, this implementation proposes a modality-independent, route-based expert layer capable of processing both images and text simultaneously. Instead of applying the entire expert layer to all inputs, it utilizes select "experts" within the layer to focus on different tasks or data types, while still leveraging shared components. This allows the expert layer size to be increased or decreased while maintaining constant computational cost, providing an efficient and environmentally friendly model scaling method.
[0058] For example, combined Figure 2The expert layer consists of a router, multiple feedforward network modules connected to the router, and multiple multi-layer perceptrons (MLPs) connected one-to-one with each feedforward network module. Each feedforward network module comprises multiple parallel feedforward networks (FFNs). The MLPs are the input-dependent activation components of the expert layer. Each FFN is an "expert." For each input token, the expert layer can sparsely process it using F experts out of E available experts. Given a sequence of tokens containing multiple tokens, the router can learn to predict which experts should process which tokens. Activating only a small number of experts per token means that although a large number of experts significantly increases model capacity, the actual computational cost can be controlled by sparsely utilizing these experts. If only one expert is activated, the cost of the expert layer is roughly equivalent to that of a standard Transformer model.
[0059] For example, step S140 includes: inputting initial text features and initial image features into a router to obtain gating weights corresponding to the initial text features and initial image features respectively; comparing the gating weights with a preset threshold to determine the target feedforward network; inputting the initial text features and initial image features into the target feedforward network, and inputting the output of the target feedforward network into the corresponding multilayer perceptron to obtain multimodal features.
[0060] Specifically, the router can be a lightweight router that predicts the gating weight for each token. The gating weight can be calculated using the following formula (1):
[0061] g(x) = softmax(Wx) (1)
[0062] Where x represents the initial text feature or initial image feature, g(x) represents the function by which the gating unit processes x, W represents the weight of the gating unit, and softmax() represents the softmax function.
[0063] After the gating weights are determined, the gating weights are compared with preset thresholds to determine the target feedforward network. Specifically, the gating weights are compared with the preset thresholds corresponding to each feedforward network, and the feedforward network whose preset threshold is less than the gating weight is selected as the target feedforward network.
[0064] Specifically, if the gating weight corresponding to the initial text feature or initial image feature is greater than the preset threshold corresponding to a certain feedforward network, i.e., a certain expert, it indicates that the expert is activated and used to process the initial text feature or initial image feature corresponding to the gating weight.
[0065] Since a multilayer perceptron is composed of fully connected layers, after the output of the target feedforward network is input into the multilayer perceptron, the fully connected layers in the multilayer perceptron will linearly concatenate the output of the target feedforward network and use the linearly concatenated output as the output of the entire expert layer, i.e., the multimodal features.
[0066] For example, multimodal features are represented by the following equation (2):
[0067]
[0068] Where f(x) represents the multimodal feature corresponding to x, h = 1, ..., K represents the index of the feedforward network included in the target feedforward network, K represents the number of feedforward networks included in the target feedforward network, and g(x) h This represents the gated unit x corresponding to the h-th feedforward network, MLP h (x) represents the multilayer perceptron corresponding to the h-th feedforward network of x.
[0069] It's important to note that, due to computational efficiency limitations, feedforward networks, i.e., experts, have a fixed buffer capacity. The maximum number of tokens each expert can process—the capacity—is predetermined, and the capacities of all experts are roughly balanced. If the number of tokens to be processed exceeds the expert's capacity, some tokens will be "abandoned." Abandoned tokens are not processed by the expert, and the expert's output is zero for these abandoned tokens. The ratio of successfully processed (not abandoned) tokens to the total number of tokens to be processed is called the success rate. The success rate is an important indicator of a healthy and balanced routing system, typically indicating the stability of training. Therefore, during model training, observing the success rate can help determine whether relevant parameters need adjustment.
[0070] Step S150: Input the multimodal features into the coding layer to obtain coded text features and coded image features.
[0071] Specifically, combined Figure 2 After obtaining multimodal features through the expert layer, the high-dimensional multimodal features are encoded by the encoding layer to obtain relatively low-dimensional encoded text features and encoded image features. Therefore, the dimensions of the encoded text features and encoded image features are smaller than the dimensions of the multimodal features.
[0072] Step S160: Train the contrastive learning network based on the encoded text features and encoded image features to obtain a trained multimodal contrastive learning model.
[0073] Specifically, the goal of contrastive learning is to make paired text-image features closer together in the feature space, while unpaired text-image features are further apart. Therefore, the loss function of the contrastive learning network can adopt the symmetric infoNCE loss function, expressed as equation (3):
[0074]
[0075] Among them, L infoNCE Let f represent the loss function of the contrastive learning network, T represent temperature, n represent the number of text-image data pairs consisting of text and image data, k = 1, ..., n represent the pair numbers, and f ij f represents the features obtained from the j-th batch of images, i.e., the multimodal features of the j-th batch of images. tj The features obtained from the j-th batch of texts are the multimodal features of the j-th batch of texts, f ik The feature obtained from the k-th image is called the multimodal feature of the k-th image, f. tk Let e represent the feature obtained from the k-th text, which is the multimodal feature of the k-th text, and f represent the natural constant. i Referring to an image, f t Refers to the text.
[0076] It should be noted that this embodiment does not limit the specific type of contrastive learning network, and those skilled in the art can choose according to actual needs.
[0077] Compared with the prior art, the embodiments disclosed herein apply a sparse hybrid expert model to an image-text multimodal contrastive learning model. By using a sparsification method, the task can be completed by activating only some experts in the expert layer for each input through routing. This allows a single model to handle multiple unimodal and multimodal tasks simultaneously. While achieving the effect of a single model handling a single task, it also avoids the catastrophic forgetting problem, thereby improving work efficiency and training effect.
[0078] Another embodiment of this disclosure relates to a multimodal contrastive learning model training apparatus, such as Figure 3 As shown, it includes:
[0079] The acquisition module 301 is used to acquire multimodal data, which includes text data and image data;
[0080] Processing module 302 is used to process multimodal data to obtain serialized marker data corresponding to text data and image block data corresponding to image data;
[0081] The first input module 303 is used to input serialized marker data and image patch data into the feature encoder to obtain initial text features and initial image features;
[0082] The second input module 304 is used to input the initial text features and initial image features into a preset route-based expert layer to obtain multimodal features;
[0083] The third input module 305 is used to input multimodal features into the coding layer to obtain coded text features and coded image features;
[0084] Training module 306 is used to train the contrastive learning network based on the encoded text features and encoded image features to obtain a trained multimodal contrastive learning model.
[0085] For a detailed implementation of the multimodal contrastive learning model training device provided in this disclosure, please refer to the multimodal contrastive learning model training method provided in this disclosure, which will not be repeated here.
[0086] Compared with the prior art, the present invention applies a sparse hybrid expert model to an image-text multimodal contrastive learning model. This allows the task to be completed by activating only some experts in the expert layer for each input using routing. As a result, a single model can be used to process multiple unimodal and multimodal tasks simultaneously. While achieving the effect of a single model processing a single task, it can also avoid the catastrophic forgetting problem, thereby improving work efficiency and training effect.
[0087] Another embodiment of this disclosure relates to an electronic device, such as Figure 4 As shown, it includes:
[0088] At least one processor 401; and,
[0089] Memory 402 is communicatively connected to at least one processor 401; wherein,
[0090] The memory 402 stores instructions that can be executed by at least one processor 401, which enables the at least one processor 401 to perform the multimodal contrastive learning model training method described in the above embodiments.
[0091] The memory and processor are connected via a bus, which can include any number of interconnecting buses and bridges, connecting various circuits of one or more processors and memories. The bus can also connect various other circuits, such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and will not be described further herein. The bus interface provides an interface between the bus and the transceiver. The transceiver can be a single element or multiple elements, such as multiple receivers and transmitters, providing a unit for communicating with various other devices over a transmission medium. Data processed by the processor is transmitted over the wireless medium via an antenna, which further receives data and transmits it to the processor.
[0092] The processor manages the bus and general processing, and also provides various functions, including timing, peripheral interfaces, voltage regulation, power management, and other control functions. Memory is used to store data used by the processor during operation.
[0093] Another embodiment of this disclosure relates to a computer-readable storage medium storing a computer program that, when executed by a processor, implements the multimodal contrastive learning model training method described in the above embodiments.
[0094] That is, those skilled in the art will understand that all or part of the steps in the methods described in the above embodiments can be implemented by a program instructing related hardware. This program is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this disclosure. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.
[0095] Those skilled in the art will understand that the above embodiments are specific implementations of this disclosure, and in practical applications, various changes can be made to them in form and detail without departing from the spirit and scope of this disclosure.
Claims
1. A method for training a multimodal contrastive learning model, characterized in that, The training method includes: Acquire multimodal data, which includes text data and image data; The multimodal data is processed to obtain serialized marker data corresponding to the text data and image patch data corresponding to the image data; The serialized marker data and the image patch data are input into the feature encoder to obtain initial text features and initial image features; The initial text features and the initial image features are input into a preset routing-based expert layer to obtain multimodal features; The multimodal features are input into the encoding layer to obtain encoded text features and encoded image features; The contrastive learning network is trained based on the encoded text features and the encoded image features to obtain a trained multimodal contrastive learning model. The expert layer includes a router, multiple feedforward network modules connected to the router, and multiple multilayer perceptrons connected to each feedforward network module in a one-to-one correspondence. Each feedforward network module includes multiple parallel feedforward networks. The step of inputting the initial text features and the initial image features into a preset routing-based expert layer to obtain multimodal features includes: The initial text features and the initial image features are input into the router to obtain the gating weights corresponding to the initial text features and the initial image features, respectively; The gating weights are compared with a preset threshold to determine the target feedforward network; The initial text features and the initial image features are input into the target feedforward network, and the output of the target feedforward network is input into the corresponding multilayer perceptron to obtain the multimodal features.
2. The training method according to claim 1, characterized in that, The step of comparing the gating weights with a preset threshold to determine the target feedforward network includes: The gating weights are compared with preset thresholds corresponding to each feedforward network, and the feedforward network whose preset threshold is less than the gating weight is selected as the target feedforward network.
3. The training method according to claim 2, characterized in that, The gating weight is calculated according to the following formula (1): （1） Where x represents the initial text feature or the initial image feature. Let W represent the function by which the gating unit in the router processes x, and let W represent the weight of the gating unit. This represents the softmax function.
4. The training method according to claim 3, characterized in that, The multimodal features are expressed as follows (2): （2） in, This represents the multimodal feature corresponding to x. K represents the number of feedforward networks included in the target feedforward network. This represents the gated unit x corresponding to the h-th feedforward network. Let x represent the multilayer perceptron corresponding to the h-th feedforward network.
5. The training method according to claim 4, characterized in that, The loss function of the contrastive learning network is expressed as follows (3): （3） in, Let T represent the loss function of the contrastive learning network, T represent the temperature, and n represent the number of text-image data pairs composed of the text data and the image data. Indicates the number of the data pair. This represents the features obtained from the j-th batch of images. This represents the features obtained from the j-th batch of text. This represents the features obtained from the k-th image. This represents the feature obtained from the k-th text.
6. The training method according to any one of claims 1 to 5, characterized in that, The dimensions of the encoded text features and the encoded image features are smaller than the dimensions of the multimodal features.
7. A multimodal contrastive learning model training device, characterized in that, The training device includes: An acquisition module is used to acquire multimodal data, which includes text data and image data; The processing module is used to process the multimodal data to obtain serialized marker data corresponding to the text data and image block data corresponding to the image data; The first input module is used to input the serialized marker data and the image block data into the feature encoder to obtain initial text features and initial image features; The second input module is used to input the initial text features and the initial image features into a preset routing-based expert layer to obtain multimodal features; The third input module is used to input the multimodal features into the encoding layer to obtain encoded text features and encoded image features; The training module is used to train the contrastive learning network based on the encoded text features and the encoded image features to obtain a trained multimodal contrastive learning model. The expert layer includes a router, multiple feedforward network modules connected to the router, and multiple multilayer perceptrons connected to each feedforward network module in a one-to-one correspondence. Each feedforward network module includes multiple parallel feedforward networks. The step of inputting the initial text features and the initial image features into a preset routing-based expert layer to obtain multimodal features includes: The initial text features and the initial image features are input into the router to obtain the gating weights corresponding to the initial text features and the initial image features, respectively; The gating weights are compared with a preset threshold to determine the target feedforward network; The initial text features and the initial image features are input into the target feedforward network, and the output of the target feedforward network is input into the corresponding multilayer perceptron to obtain the multimodal features.
8. An electronic device, characterized in that, include: At least one processor; And, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the multimodal contrastive learning model training method according to any one of claims 1 to 6.
9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the multimodal contrastive learning model training method according to any one of claims 1 to 6.