Training of a multi-modal machine learning model

Abstract

Embodiments of the present disclosure provide a scheme for training a multi-modal machine learning model. In a method, a machine learning model is obtained, which is pre-trained to generate an output of a text modality based on an input of the text modality. A set of training samples is constructed according to a predetermined data format. The predetermined data format includes a first set of tokens for a first input of a first modality and a second set of tokens for a second input of a text modality, and the first modality is different from the text modality. Furthermore, the machine learning model is trained using the set of training samples.

Description

Technical Field

[0001] The exemplary embodiments of this disclosure generally relate to the field of computer technology, and more particularly to the training of multimodal machine learning models. Background Technology

[0002] In recent years, the rapid development of machine learning (ML) technology has sparked a surge of interest in the field of multimodal ML models. Multimodal ML models can perform tasks such as text-only tasks, visual understanding, text-guided image generation, and text-guided video generation. However, extensive training of multimodal ML models typically requires the use of large amounts of training data. The heavy cost of the training process makes developing multimodal ML models challenging. Therefore, there is a general desire to simplify the training process of multimodal ML models. Summary of the Invention

[0003] In a first aspect of this disclosure, a method for model training is provided. The method includes: acquiring a machine learning model pre-trained to generate text modal outputs based on text modal inputs; constructing a set of training samples according to a predetermined data format including a first set of labels for a first input to a first modality and a second set of labels for a second input to a text modality, the first modality being different from the text modality; and training the machine learning model using the set of training samples.

[0004] In a second aspect of this disclosure, an apparatus for model training is provided. The apparatus includes: an acquisition module configured to acquire a machine learning model pre-trained to generate text modal outputs based on text modal inputs; a construction module configured to construct a set of training samples according to a predetermined data format including a first set of labels for a first input to a first modality and a second set of labels for a second input to a text modality, the first modality being different from the text modality; and a training module configured to train the machine learning model using the set of training samples.

[0005] In a third aspect of this disclosure, an electronic device is provided. The electronic device includes: at least one processing unit; and at least one memory coupled to the at least one processing unit and storing instructions executable by the at least one processing unit, the instructions, when executed by the at least one processing unit, causing the electronic device to perform the method according to a first aspect of this disclosure.

[0006] In a fourth aspect of this disclosure, a computer-readable storage medium is provided. This computer-readable storage medium has a computer program stored thereon, which, when executed by an electronic device, causes the electronic device to perform the method according to a first aspect of this disclosure.

[0007] In a fifth aspect of this disclosure, a computer program product is provided. The computer program product is implemented on a computer-readable medium and includes computer-executable instructions that are executed by a processor to perform the method according to a first aspect of this disclosure.

[0008] It should be understood that the content described in this summary section is not intended to limit the key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0009] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. In the drawings, the same or similar reference numerals denote the same or similar elements, wherein:

[0010] Figure 1 A block diagram is shown of an example environment in which various embodiments of the present disclosure may be implemented;

[0011] Figure 2 A flowchart of a method for model training according to some example embodiments of the present disclosure is shown;

[0012] Figure 3 An example pipeline of a multimodal ML model according to some example embodiments of the present disclosure is shown;

[0013] Figure 4 Block diagrams of apparatus for model training according to some example embodiments of the present disclosure are shown; and

[0014] Figure 5 A block diagram of an electronic device in which one or more embodiments of the present disclosure may be implemented is shown. Detailed Implementation

[0015] Embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While some exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as limited to the embodiments described herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the accompanying drawings and embodiments of the present disclosure are for illustrative purposes only and are not intended to limit the scope of protection of the present disclosure.

[0016] In the description of embodiments of this disclosure, the term "comprising" and similar terms should be understood as open-ended inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "one embodiment" or "this embodiment" should be understood as "at least one embodiment". The term "some example embodiments" should be understood as "at least some example embodiments". Other explicit and implicit definitions may also be included below. As used herein, the term "model" can represent the degree of matching between various data. For example, the above degree of matching can be obtained based on various currently available and / or future developed technical solutions.

[0017] It should be understood that the data involved in this technical proposal (including but not limited to the data itself, data collection or use) shall comply with the requirements of applicable laws, regulations and relevant provisions.

[0018] It should be understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure and their authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0019] For example, in response to receiving a user's active request, a prompt message is sent to the user to explicitly inform the user that the operation (the operation requested by the user) will require the acquisition and use of the user's personal information. Therefore, the user can choose, based on the prompt message, whether to provide personal information to the software or hardware such as electronic devices, applications, servers, or storage media that perform the operations of the technical solutions disclosed herein.

[0020] As an optional, and non-limiting, implementation, a method for sending a prompt to the user in response to a user's active request may be, for example, a pop-up window, in which the prompt message may be presented as text. Additionally, the pop-up window may also include a selection control allowing the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0021] It should be understood that the above notification and user authorization acquisition process is merely illustrative and does not limit the implementation of this disclosure. Other methods that comply with applicable laws and regulations may also be applied to the implementation of this disclosure.

[0022] As used in this paper, the term "model" refers to a model that learns the relationship between inputs and outputs from training data so that, after training, it can generate corresponding outputs for a given input. Model generation can be based on machine learning techniques. Deep learning is a machine learning algorithm that processes inputs and provides corresponding outputs using multiple layers of processing units. Neural network models are examples of models based on deep learning. As used in this paper, "model" can also be referred to as "machine learning model," "learning model," "machine learning network," "learning network," etc., and these terms are used interchangeably in this paper.

[0023] A neural network is a machine learning network based on deep learning. A neural network can process inputs and provide corresponding outputs, typically consisting of input and output layers, and one or more hidden layers between them. Neural networks used in deep learning applications often include many hidden layers, thus increasing the network's depth. The layers of a neural network are connected sequentially so that the output of the previous layer is used as the input to the next layer, where the input layer receives the inputs to the neural network, and the output layer's output is used as the final output of the neural network. Each layer of a neural network includes one or more nodes (also called processing nodes or neurons), each node processing the input from the previous layer.

[0024] Generally, machine learning can be broadly divided into three phases: training, testing, and application (also known as inference). During the training phase, a given model is trained using large-scale training data, iteratively updating parameter values ​​until the model can consistently generate inference that meets the expected goals from the training data. Through training, the model can be considered to have learned the relationship between inputs and outputs (also known as the input-to-output mapping) from the training data. The parameter values ​​of the trained model are determined. In the testing phase, test inputs are applied to the trained model to test whether the model can provide the correct output, thus determining the model's performance. In the application phase, the model can be used to process actual inputs based on the parameter values ​​obtained from training and determine the corresponding output.

[0025] Figure 1 A block diagram of an example environment 100 in which various embodiments of the present disclosure may be implemented is shown. Figure 1 In environment 100, two distinct phases of the model are illustrated, including a training phase 102 and an application phase 106. After the training phase 102 is completed, a testing phase may exist. Figure 1 Not shown in the image.

[0026] In training phase 102, model training system 110 is configured to train machine learning model 105 using training dataset 112. At the start of training, machine learning model 105 may have initial parameter values. In some example embodiments, machine learning model 105 may be pre-trained to generate text modal outputs based on text modal inputs. The training process is used to update the parameter values ​​of machine learning model 105 to desired values ​​based on the training data.

[0027] In application phase 106, a machine learning model 105 with trained parameter values ​​can be provided to the model application system 130 for use. In application phase 106, the machine learning model 105 can be used to process the target input 132 and provide the corresponding target output 134.

[0028] exist Figure 1 In this context, the model training system 110 and the model application system 130 can be implemented on any computing system with computing capabilities, such as various computing devices / systems, terminal devices, servers, etc. Terminal devices can include any type of mobile terminal, fixed terminal, or portable terminal, including mobile phones, desktop computers, laptop computers, notebook computers, tablet computers, media computers, multimedia tablet computers, or any combination thereof, including accessories and peripherals of these devices or any combination thereof. Servers include, but are not limited to, mainframes, edge computing nodes, computing devices in cloud environments, etc.

[0029] It should be understood that the structure and function of each element in environment 100 are described for illustrative purposes only and do not imply any limitation on the scope of this disclosure. In the example, although shown as separate entities, model training system 110 and model application system 130 may be integrated into the same system or device. Implementations of the schemes disclosed herein are not limited thereto.

[0030] As briefly mentioned above, multimodal ML models can perform a variety of tasks, such as text-only tasks, visual understanding, text-guided image generation, and text-guided video generation. In one existing design, a pre-trained diffusion module is incorporated as a visual tokenizer into the language model to achieve multimodal generation. However, this approach of combining the visual model with the language model increases the complexity of the language model, and the pre-trained visual tokenizer may potentially become a bottleneck when scaling up the language model. In another existing design, a multimodal ML model is built based on the language model and specific modifications to its structure. In both designs, the entire multimodal ML model is trained from scratch during the training phase, which requires a large amount of training data. This leads to a cumbersome training process and a huge consumption of computational resources.

[0031] According to embodiments of this disclosure, an improved scheme for model training is proposed. According to the scheme of embodiments of this disclosure, a machine learning model is acquired and pre-trained to generate text-modal outputs based on text-modal inputs. A set of training samples is constructed according to a predetermined data format. The predetermined data format includes a first set of labels for a first input to a first modality and a second set of labels for a second input to a text-modality, wherein the first modality is different from the text-modality. Furthermore, the machine learning model is trained using a set of training samples.

[0032] Based on the scheme according to embodiments of this disclosure, a pre-trained ML model for generating text output from text input is used as the starting point of the training process, and multimodal training samples are used to train the pre-trained ML model to acquire multimodal capabilities. Thus, the structure of the pre-trained ML model remains unchanged, and the amount of training data and time consumed in the training process can be significantly reduced, thereby simplifying the training process.

[0033] The exemplary embodiments of this disclosure will now be described in detail with reference to the accompanying drawings.

[0034] Figure 2 A flowchart of a method 200 for model training according to some example embodiments of the present disclosure is shown. For example, method 200 may be provided by, for example... Figure 1 The model training system 110 shown is executed. For ease of description, refer to the following... Figure 3 Description method 200, Figure 3 An example pipeline 300 for a multimodal ML model according to some example embodiments of the present disclosure is shown. Figure 3 In the example shown, the pre-trained ML model 305 can be Figure 1 An example implementation of ML model 105 is provided. It should be understood that method 200 may also include additional boxes not shown, and / or the boxes shown may be omitted. Additionally, example pipeline 300 may also include additional components not shown, and / or the components shown may be omitted. The scope of this disclosure is not limited in this respect.

[0035] At 210, an ML model 305 is obtained, which is pre-trained to generate text-modal outputs based on text-modal inputs. In some example embodiments, the ML model 305 can be pre-trained using text-only data by means of any suitable training process, and the pre-trained ML model 305 can be able to perform text-only tasks. In this respect, the pre-trained ML model 305 can also be referred to as a language model at the outset. It should be noted that after being trained according to the proposed scheme, the language model acquires the ability to process a modality other than text (e.g., Figure 3The ability to use data (such as the images in the example shown) allows the language model to be transformed into a multimodal ML model. This pre-trained ML model 305 can be used as a starting point for the training process according to embodiments of this disclosure. This strategy ensures that the proposed model training scheme can benefit from the structure and training experience of mature language models.

[0036] At position 220, a set of training samples is constructed according to a predetermined data format. This predetermined data format includes a first set of labels for a first input to a first modality and a second set of labels for a second input to a text modality, wherein the first modality differs from the text modality. For illustrative purposes, in Figure 3 In the example shown, the first modality is the image modality. It should be understood that the first modality can be any other suitable modality, such as an audio modality, a video modality, etc. The scope of this disclosure is not limited in this respect.

[0037] like Figure 3 As shown, the training text 302 can be tokenized based on a word vocabulary using a second tokenizer 312 to obtain a second set of tokens 322. In some example embodiments, each token in the second set of tokens 322 is a discrete token. For example, the second tokenizer may be a byte-pair encoding (BPE) tokenizer, a byte-level byte-pair encoding (BBPE) tokenizer, a WordPiece tokenizer, etc. Similarly, the training image 301 can be tokenized based on an image codebook using a first tokenizer 311 to obtain a first set of tokens 321. In some example embodiments, each token in the first set of tokens 321 is a discrete token. For example, the first tokenizer 311 may be a vector quantization variational autoencoder (VQVAE) tokenizer, a vector quantization generative adversarial network (VQGAN) tokenizer, etc. For example, the first tokenizer 311 can encode an image with a resolution of 512×512 into 1024 discrete tokens in a codebook of size 8192. It should be understood that the specific values ​​listed herein are intended as examples and not to limit the scope of this disclosure.

[0038] In some example implementations, the aspect ratios of all candidate training images can be used for filtering. For example, images with aspect ratios less than a threshold (such as 2) can be retained. By center cropping, the retained images can be resized to a predetermined resolution, such as 512×512 pixels, to obtain training images. This ensures that for text-image pairs, the cropped image still matches the corresponding text, and thus training images with the predetermined resolution can be obtained.

[0039] In some example embodiments, tags in a codebook that tokenizes data for the first modality can be added to a codebook for the pre-trained ML model 305. For example, tags in a codebook for the first tokenizer 311 can be appended to a text codebook produced by the second tokenizer 312 to expand the vocabulary size of the pre-trained ML model 305. For instance, numbers from 0 to 600 can represent a text codebook, while numbers from 601 to 8792 can represent an image codebook. By combining codebooks for data from different modalities, the language space of the pre-trained ML model 305 can be extended to a multimodal space encompassing elements for all these modalities. Figure 3 In the example shown, the expanded multimodal space encompasses both visual and linguistic elements. This enables the acquisition of multimodal capabilities without modifying the structure of the pre-trained ML model 305, and therefore data from different modalities (such as images and text) can all share the same model structure and feature space, allowing for full interaction and enhancing the potential of the multimodal model.

[0040] In some example embodiments, in a predetermined data format, the first set of markers may follow the second set of markers, and the second set of markers and the first set of markers may be separated by a marker indicating the start of markers for the first modality. Furthermore, the predetermined data format may also include: a marker indicating the end of markers for the first modality, a marker indicating the start of a marker sequence, a marker indicating the end of a marker sequence, etc.

[0041] for Figure 3 The example shown illustrates that the predefined data format can be defined as follows: [bos]{second group of markers}[boi]{first group of markers}[eoi][eos] Where [bos] indicates the start of the marker sequence, [eos] indicates the end of the marker sequence, [boi] indicates the start of the marker sequence for the first mode, and [eoi] indicates the end of the marker sequence for the first mode. Figure 3 As shown, training sample 330 was constructed based on this data format.

[0042] Alternatively, in the predetermined data format, the first set of tags may precede the second set of tags, and the first and second sets of tags may be separated by a tag indicating the start of tags for the first modality. Furthermore, the predetermined data format may also include: a tag indicating the end of tags for the text modality, a tag indicating the start of a tag sequence, a tag indicating the end of a tag sequence, etc. For example, the predetermined data format may be defined as follows: [bos]{first group of markers}[bot]{second group of markers}[eot][eos] Wherein, [bos] indicates the start of a tag sequence, [eos] indicates the end of a tag sequence, [bot] indicates the start of a tag for a text modality, and [eot] indicates the end of a tag for a text modality.

[0043] By utilizing this predetermined data format, labels for data of different modalities can be concatenated in an ordered manner at the input side, and labels for data of different modalities can be easily separated from each other at the output side. Thus, labels for data of different modalities can be processed in a unified label space, enabling multimodal capabilities to be obtained without modifying the structure of the pre-trained ML model 305, and therefore allowing the pre-trained ML model 305 to quickly acquire multimodal capabilities, such as image generation and understanding capabilities. It should be noted that the predetermined data format can also be defined in any other suitable manner. For example, when more than two modalities (e.g., text, image, audio, etc.) are involved in the training samples, additional special labels can be introduced to indicate the start and / or end of labels for the additional modalities. The scope of this disclosure is not limited in this respect.

[0044] Back Figure 2 At position 230, a set of training samples is used to train the ML model 305. In some example embodiments, during the first training phase, the ML model 305 is further trained using a first subset of training samples from the set. It should be noted that the first subset of training samples may be the same as the set of training samples; alternatively, the set of training samples may include (or more) additional training samples besides the first subset. The first subset of training samples may be generated from a first training dataset. The first training dataset may include training data pairs that include data from the first modality and text data. As an example, the training data pair may be an image-text pair. With the aid of such multimodal training data pairs, the pre-trained ML model 305 can be trained to obtain multimodal capabilities.

[0045] In some example embodiments, the training data pair may include a first portion and a second portion, the first portion being for a task of generating data of a first modality from text data, and the second portion being for a task of generating text data from the first modality data. For example, in the case of image-text pairs, the first portion of the image-text pair may be text-to-image data for a visual generation task, and the second portion may be image-to-text data for a visual understanding task.

[0046] Through well-designed experiments, the inventors have observed a mutual reinforcement between visual generation and visual understanding as the feature spaces for visual understanding and generation become unified. More specifically, it has been observed that adding more visual understanding data during training not only improves the performance of visual understanding but also significantly improves the performance of the visual generation task. Similarly, adding more visual generation data also enhances the performance of both the visual understanding and visual generation tasks. This observation suggests that training for both tasks can benefit each other when the modal spaces for visual understanding and visual generation are unified. Intuitively, both visual generation and visual understanding tasks require stronger consistency constraints and sufficient interaction between linguistic and visual information when using embeddings representing images from the same space. Thus, the optimization directions for these two tasks are very similar. This further demonstrates the potential of pre-trained language models as general multimodal generators. Continuous addition of visual training data can simultaneously enhance the model's capabilities in both multimodal understanding and generation.

[0047] Thus, the proportion of the first part of the training data pairs in the first training dataset can be negatively correlated with the proportion of the second part of the training data pairs in the first training dataset, while both multimodal generation capability and multimodal understanding capability remain unchanged. For example, the proportion of text-to-image data in the first training dataset can decrease as the proportion of image-to-text data in the first training dataset increases. Similarly, the proportion of the second part of the training data pairs in the first training dataset can be negatively correlated with the proportion of the first part of the training data pairs in the first training dataset, while both multimodal generation capability and multimodal understanding capability remain unchanged. For example, the proportion of image-to-text data in the first training dataset can decrease as the proportion of text-to-image data in the first training dataset increases. Therefore, the amount of data required to train the ML model 305 can be advantageously further reduced without affecting the multimodal capability of the final model.

[0048] In some other example embodiments, the number of training iterations and / or the share of training duration using the first portion of the training data pair may be negatively correlated with the number of training iterations and / or the share of training duration using the second portion of the training data pair. Similarly, the number of training iterations and / or the share of training duration using the second portion of the training data pair may be negatively correlated with the number of training iterations and / or the share of training duration using the first portion of the training data pair.

[0049] Furthermore, the inventors have noted that if the pre-trained language model is further trained using image-text pairs, the language capabilities of the final model may be degraded or even completely lost. Thus, in some additional example embodiments, the first training dataset may also include text-only data to preserve the language capabilities of the ML model 305.

[0050] Through well-designed experiments, the inventors also noted that the proposed training scheme could negatively impact language performance when the ML model 305 was small. However, this degradation in language performance gradually disappeared as the model size increased, indicating that larger models have a better ability to learn multimodal capabilities without affecting the model's original capabilities. Furthermore, it was observed that the quality of images generated by the final acquired model improved with increasing model size. This demonstrates that the pre-trained language model has a high upper bound as a multimodal generator and exhibits the same scaling law as the language model.

[0051] Thus, the proportion of text-only data in the first training dataset can be negatively correlated with the size of the ML model 305, while the linguistic capabilities of the ML model 305 remain unchanged. For example, the proportion of text-only data in the first training dataset can decrease as the model size increases. This advantageously reduces the amount of data required to train the ML model 305 without affecting the multimodal capabilities of the final model. In some other example embodiments, the proportion of the number of training iterations and / or training duration using text-only data can be negatively correlated with the size of the ML model 305. By way of example and not limitation, the size of the ML model 305 can be measured by the number of parameters of the ML model 305.

[0052] To illustrate the details of the training process, let's take the first training sample 330 from the set of training samples used to train the ML model 305 as an example. Figure 3 As shown, the first training sample 330 can be a first label sequence, which is constructed according to the predetermined data format mentioned above. Based on the first label sequence, a second label sequence corresponding to the output sample 340 can be generated by the ML model 305. For example, the generation of the second label sequence can be performed based on the next label prediction scheme. More specifically, a first label at a first position in the second label sequence can be generated based on at least one label in the first label sequence, and the position of each of the at least one label precedes the first position. As an example, a label at the i-th position in the second label sequence can be generated based on the labels at positions 0 to (i-1) in the first label sequence, where i is an integer.

[0053] Furthermore, the loss can be determined based on the first and second label sequences. As an example, the loss can be calculated using a cross-entropy loss function, etc. Based on the determined loss, the parameter values ​​of the ML model 305 can be updated to minimize the loss. The process described above can be executed iteratively until a termination condition is met. For example, the termination condition can include: loss convergence, loss falling below a predetermined threshold, performing a predetermined number of iterations, etc. In some example embodiments, loss spikes may occur in the early stages of training, which can significantly affect training performance and convergence speed. To mitigate this problem, the maximum gradient norm can be reduced for larger models (e.g., reduced to 0.5, etc.), and the logistic values ​​(logits) can be normalized using max-z loss. This can enhance the stability of training.

[0054] The first training phase described above can also be called the "continued pre-training phase," which preserves the inherent language capabilities of the pre-trained model without requiring the addition of an additional visual model for visual understanding and generation capabilities.

[0055] In some additional example embodiments, the training process of ML model 305 may further include a second training phase following the first training phase. In the second training phase, ML model 305 may be trained using a second subset of training samples from a set of training samples. This second subset of training samples may be generated from a second training dataset, which includes text-only instruction data, multimodal instruction data, data for a task involving generating first modality data from text data, etc. Figure 3 In the example, the second training dataset may include text-only instruction data, text-to-image data, and multimodal instruction tuning data. This allows for further refinement of the ML model 305 acquired after the first training phase and improves its ability to respond to instructions. In this respect, the second training phase can be referred to as the "instruction tuning phase." Apart from the training dataset, the training details of this second training phase may be the same as or similar to those of the first training phase described above, and therefore will not be repeated here for the sake of brevity.

[0056] like Figure 3 As shown, based on the special markers [bos], [boi], [eoi], and [eos] described above, the output sample 340 can be divided into two parts: text markers and image markers. The text markers can be de-marked by the second de-marker 352 to obtain the output text 362, and the image markers can be de-marked by the first de-marker 351 to obtain the output image 361. In some example embodiments, this de-marking process can be omitted during the training phase and performed only during the application phase.

[0057] Additionally, the inventors have observed that this convenience can be applied to visual generation due to the language model's ability to output dynamically long text in the form of a next-tag prediction. As an example, by appending instructions regarding image resolution to the input text prompt, such as "length: XXX, width: YYY", the model can quickly learn to generate the corresponding code based on a specified number of rows and columns. Thus, the resulting multimodal ML model 305 can advantageously support different image resolutions, thereby expanding the model's application scenarios.

[0058] It should be noted that the structure of the pre-trained ML model 305 remains unchanged throughout the entire training process. Experiments have demonstrated that the amount of training data required to continue training the pre-trained ML model 305 according to the proposed model training scheme can be significantly reduced compared to training the entire model from scratch. For example, the amount of training data can be reduced by approximately three orders of magnitude, from billions to millions. Consequently, the time consumption for the training process can be greatly reduced, thus improving training speed. Additionally, the model trained according to the proposed scheme exhibits the ability to match or even outperform task-specific models across language tasks, image generation, and image understanding. Moreover, due to the rich variety of existing language models, the ML model can be easily scaled up, bringing this effective training scheme to larger-scale models.

[0059] In the application phase, the input label sequence can be used as a condition, and a corresponding output label sequence can be generated according to the next label prediction scheme. For example, the first label in the output label sequence can be generated based on the entire input label sequence; the second label in the output label sequence can be generated based on the entire input label sequence and the generated first label; the third label in the output label sequence can be generated based on the entire input label sequence and the generated first and second labels; and so on. It can be seen that each next label is generated in an autoregressive manner, based on the input label sequence and the (multiple) labels already generated. This generation process is performed until a label indicating the end of the label sequence (e.g., label [eos]) is generated.

[0060] In summary, a pre-trained ML model used as a language model is employed as the starting point for the training process. This pre-trained ML model is then trained using multimodal training samples to acquire multimodal capabilities. This approach preserves the structure of the pre-trained ML model and significantly reduces the amount of training data and time required for the training process. Consequently, it enables the rapid acquisition of multimodal capabilities (such as visual generation and visual understanding) while maintaining language proficiency.

[0061] Those skilled in the art will understand that the operations of the methods disclosed herein are not necessarily presented in any particular order, and that it is possible and contemplated to perform some or all of the operations in an alternative order. For ease of description and illustration, the operations have been presented in the order shown. Operations may be added, omitted, performed together, and / or performed simultaneously without departing from the scope of the appended claims. It should also be understood that the illustrated methods may end at any time and need not be performed as a whole.

[0062] Figure 4 A block diagram of an apparatus 400 for model training according to some example embodiments of the present disclosure is shown. The apparatus 400 may be, for example, in... Figure 1 The model training system 110 shown is implemented or included. Various modules / components in device 400 can be implemented by hardware, software, firmware, or any combination thereof.

[0063] like Figure 4 As shown, the apparatus 400 includes an acquisition module 410, a construction module 420, and a training module 430. The acquisition module 410 is configured to acquire a machine learning model pre-trained to generate text modal outputs based on text modal inputs. The construction module 420 is configured to construct a set of training samples according to a predetermined data format, which includes a first set of labels for a first input to a first modality and a second set of labels for a second input to a text modality, wherein the first modality is different from the text modality. The training module 430 is configured to train the machine learning model using the set of training samples.

[0064] In some example embodiments, in a predetermined data format, the first set of markers follows the second set of markers, and the second set of markers and the first set of markers are separated by a marker that indicates the start of markers for the first modality.

[0065] In some example embodiments, the predetermined data format further includes at least one of the following: a marker indicating the end of a marker for a first modality, a marker indicating the beginning of a marker sequence, or a marker indicating the end of a marker sequence.

[0066] In some example embodiments, the tags in the codebook that tokenizes the data of the first modality are added to the codebook that tokenizes the pre-trained machine learning model.

[0067] In some example embodiments, the training module 430 is further configured to train a machine learning model in a first training phase using a first subset of training samples from a set of training samples, the first subset of training samples being generated from a first training dataset, the first training dataset including at least one of the following: text data only, or training data pairs including data of the first modality and text data.

[0068] In some example implementations, the proportion of text-only data in the first training dataset is negatively correlated with the size of the machine learning model.

[0069] In some example embodiments, the training data pair includes a first part and a second part, the first part being for the task of generating data of a first modality from text data, the second part being for the task of generating text data from the data of the first modality, and the share of the first part in the first training dataset is negatively correlated with the share of the second part in the first training dataset.

[0070] In some example embodiments, the training module 430 is further configured to train a machine learning model in a second training phase following the first training phase using a second subset of training samples from a set of training samples. The second subset of training samples is generated from a second training dataset, which includes at least one of the following: text-only instruction data, multimodal instruction data, or data for a task that generates first modality data from text data.

[0071] In some example embodiments, the first training sample in a set of training samples includes a first label sequence, and the training module 430 is further configured to: generate a second label sequence by a machine learning model based on the first label sequence, wherein a first label at a first position in the second label sequence is generated based on at least one label in the first label sequence, the position of each of the at least one label preceding the first position; determine a loss based on the first label sequence and the second label sequence; and update the values ​​of the parameters of the machine learning model based on the determined loss.

[0072] In some example embodiments, each of the first group of tags and the second group of tags is a discrete tag.

[0073] In some example embodiments, the first modality includes an image modality.

[0074] The units and / or modules included in device 400 can be implemented in various forms, including software, hardware, firmware, or any combination thereof. In some example embodiments, one or more units and / or modules may be implemented using software and / or firmware (such as machine-executable instructions stored on a storage medium). In addition to or as a substitute for machine-executable instructions, some or all of the units and / or modules in device 400 may be implemented at least partially by one or more hardware logic components. By way of example and not limitation, example types of hardware logic components that may be used include field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard circuits (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), etc.

[0075] Figure 5 A block diagram of an electronic device 500 in which one or more embodiments of the present disclosure may be implemented is shown. It should be understood that... Figure 5 The electronic device 500 shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments described herein. The electronic device 500 can, for example, be used to implement... Figure 1 The model training system 110.

[0076] like Figure 5 As shown, electronic device 500 is in the form of a general-purpose computing device. Components of electronic device 500 may include, but are not limited to, one or more processors or processing units 510, memory 520, storage device 530, one or more communication units 540, one or more input devices 550, and one or more output devices 560. Processing unit 510 may be a physical or virtual processor and can perform various processes according to a program stored in memory 520. In a multiprocessor system, multiple processing units execute computer-executable instructions in parallel to improve the parallel processing capability of electronic device 500.

[0077] Electronic device 500 typically includes various computer storage media. Such media can be any available media accessible to electronic device 500, including but not limited to volatile and non-volatile media, removable and non-removable media. Memory 520 can be volatile memory (e.g., registers, cache, random access memory (RAM)), non-volatile memory (e.g., read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory), or any combination thereof. Storage device 530 can be any removable or non-removable media and can include machine-readable media, such as flash drives, disks, or any other media that can be used to store information and / or data (such as training data for training) and can be accessed within electronic device 500.

[0078] Electronic device 500 may also include additional removable / non-removable, volatile / non-volatile storage media. Although not explicitly stated... Figure 5 As shown, disk drives for reading from or writing to removable, non-volatile disks (e.g., "floppy disks") and optical disk drives for reading from or writing to removable, non-volatile optical disks can be provided. In these cases, each drive can be connected to a bus (not shown) via one or more data media interfaces. Memory 520 may include computer program product 525 having one or more program modules configured to perform various methods or actions of various embodiments of this disclosure.

[0079] Communication unit 540 communicates with other computing devices via a communication medium. Additionally, the functionality of the components in electronic device 500 can be implemented using a single computing cluster or multiple computing machines that can communicate via communication connections. Therefore, electronic device 500 can operate in a networked environment using logical connections to one or more other servers, network personal computers (PCs), or other network nodes.

[0080] Input device 550 can be one or more input devices, such as a mouse, keyboard, trackball, etc. Output device 560 can be one or more output devices, such as a monitor, speaker, printer, etc. Electronic device 500 can also communicate with one or more external devices (not shown) as needed via communication unit 540. External devices (such as storage devices, display devices, etc.) communicate with one or more devices that enable user interaction with electronic device 500, or with any device that enables electronic device 500 to communicate with one or more other computing devices (e.g., network card, modem, etc.). Such communication can be performed via input / output (I / O) interfaces (not shown).

[0081] According to an exemplary implementation of this disclosure, a computer-readable storage medium is provided that stores computer-executable instructions or a computer program thereon, wherein the computer-executable instructions or computer program are executed by a processor to implement the methods described above. According to an exemplary implementation of this disclosure, a computer program product is also provided. This computer program product is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions that are executed by a processor to implement the methods described above.

[0082] This document describes various aspects of the disclosure with reference to flowchart illustrations and / or block diagrams of methods, apparatuses, devices, and computer program products implemented according to the disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of each block in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0083] These computer-readable program instructions may be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine that, when executed by the computer or other programmable data processing apparatus, performs the functions / actions specified in one or more boxes of a flowchart and / or block diagram. These computer-readable program instructions may also be stored in a computer-readable storage medium. These instructions enable a computer, programmable data processing apparatus, and / or other apparatus to operate in a particular manner. Thus, a computer-readable medium containing the instructions includes a product that includes the instructions to implement aspects of the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0084] Computer-readable program instructions may be loaded onto a computer, other programmable data processing apparatus or other device to perform a series of operational steps on the computer, other programmable data processing apparatus or other device to produce a computer-implemented process, such that the instructions, which execute on the computer, other programmable data processing apparatus or other device, perform the function / action specified in one or more boxes in a flowchart and / or block diagram.

[0085] The flowcharts and block diagrams in the accompanying drawings illustrate possible architectures, functions, and operations of systems, methods, and computer program products implemented according to this disclosure. In this regard, each block in a flowchart or block diagram may represent a portion of a module, program segment, or instruction, which contains one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented by a dedicated hardware-based system performing the specified function or action, or by a combination of dedicated hardware and computer instructions.

[0086] Implementations of this disclosure have been described above. The above description is exemplary and not exhaustive, and is not limited to the disclosed implementations. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described implementations. The terminology used herein is chosen to best explain the principles, practical applications, or improvements to technology in the market, or to enable others skilled in the art to understand the various implementations disclosed herein.

Claims

1. A method for model training, comprising: Obtain a machine learning model, which is pre-trained to generate the output of the text modality based on the input of the text modality; A set of training samples is constructed according to a predetermined data format, the predetermined data format including a first set of labels for a first input of a first modality and a second set of labels for a second input of a text modality, wherein the first modality is different from the text modality; and The machine learning model is trained using the set of training samples.

2. The method of claim 1, wherein in the predetermined data format the first set of markers follows the second set of markers, and the second set of markers and the first set of markers are separated by markers indicating the start of markers for the first modality.

3. The method of claim 2, wherein the predetermined data format further comprises at least one of the following: A marker indicating the end of the marker for the first modality. A marker indicating the start of a marker sequence, or A marker indicating the end of the marker sequence.

4. The method according to any one of claims 1-3, wherein the tags in the codebook for tagging the data of the first modality are added to the codebook for the pre-trained machine learning model.

5. The method according to any one of claims 1-4, wherein training the machine learning model using the set of training samples comprises: In the first training phase, the machine learning model is trained using a first subset of training samples from the set of training samples, the first subset of training samples being generated from a first training dataset, the first training dataset comprising at least one of the following: Text data only, or The training data includes the data from the first modality and the text data.

6. The method of claim 5, wherein the proportion of text-only data in the first training dataset is negatively correlated with the size of the machine learning model.

7. The method according to any one of claims 5-6, wherein the training data pair comprises a first portion and a second portion, the first portion being for the task of generating data of the first modality from text data, the second portion being for the task of generating text data from the data of the first modality, and the share of the first portion in the first training dataset is negatively correlated with the share of the second portion in the first training dataset.

8. The method according to any one of claims 5-7, wherein training the machine learning model using the set of training samples further comprises: In a second training phase following the first training phase, the machine learning model is trained using a second subset of training samples from the set of training samples, the second subset of training samples being generated from a second training dataset, the second training dataset including at least one of the following: Text instruction data only. Multimodal command data, or Data for the task of generating the first modality from text data.

9. The method according to any one of claims 1-8, wherein the first training sample in the set of training samples comprises a first labeled sequence, and training the machine learning model using the set of training samples comprises: Based on the first label sequence, the machine learning model generates a second label sequence, wherein a first label at a first position in the second label sequence is generated based on at least one label in the first label sequence, and the position of each of the at least one label is before the first position; The loss is determined based on the first and second label sequences; as well as The values ​​of the parameters of the machine learning model are updated based on the determined loss.

10. The method according to any one of claims 1-9, wherein each of the first set of marks and the second set of marks is a discrete mark.

11. The method according to any one of claims 1-10, wherein the first modality comprises an image modality.

12. An apparatus for model training, comprising: The acquisition module is configured to acquire a machine learning model, which is pre-trained to generate the output of the text modality based on the input of the text modality; A construction module is configured to construct a set of training samples according to a predetermined data format, the predetermined data format including a first set of labels for a first input to a first modality and a second set of labels for a second input to a text modality, the first modality being different from the text modality; as well as The training module is configured to train the machine learning model using the set of training samples.

13. An electronic device, comprising: At least one processing unit; as well as At least one memory, coupled to the at least one processing unit and storing instructions executable by the at least one processing unit, which, when executed by the at least one processing unit, cause the electronic device to perform the method according to any one of claims 1-11.

14. A computer-readable storage medium having a computer program stored thereon, the computer program, when executed by an electronic device, causing the electronic device to perform the method according to any one of claims 1-11.

15. A computer program product implemented on a computer-readable medium and comprising computer-executable instructions which are executed by a processor to perform the method according to any one of claims 1-11.

Images