Multi-modal large model training method and device and storage medium

Abstract

The invention discloses a multi-modal large model training method, which is applied to the training of a multi-modal large model, a plurality of single modals are mounted in the model architecture of the multi-modal large model, and the method comprises the following steps: obtaining a training sample, and determining the data type of the training sample; a single mode corresponding to the data type is determined from a model framework of the multi-mode large model, time sequence alignment training is conducted on the training sample through the single mode, and after the single mode is pre-trained based on the corresponding single-mode training sample, the single mode is mounted to the model framework of the multi-mode large model. The invention further discloses a multi-modal large model training device and a storage medium, the model architecture of the applied multi-modal large model integrates multiple pieces of modal information, efficient feature alignment and time synchronization are achieved, the efficiency of a processing mechanism is greatly improved, and therefore the technical effect of improving the performance of the multi-modal large model is achieved.

Description

Technical Field

[0001] The present application relates to the field of big data, and in particular to a multimodal large model training method, device and storage medium. Background Art

[0002] In the current multimodal large model architecture, a large language model is usually used as the core, supplemented by other modalities such as images, videos, and voices. These modal information is first converted into Tokens through an encoder, and then aligned with the semantic space of the large language model in the feature space. However, this approach has an essential flaw: since the features of different modalities are distributed differently in the feature space, it is difficult to optimize the model performance if the alignment is mechanically based on the feature space distribution of the language modality.

[0003] In addition, existing multimodal models often use simple linear splicing or alternating presentation when processing different modalities. This processing mode ignores the fact that in actual application scenarios, multiple modal information often appears in parallel. Take video as an example, its content often contains multiple information such as images, text, sound, and human voice. In this case, the existing model cannot generate multimodal information that is accurately aligned on the timeline, resulting in discontinuity and distortion in information processing.

[0004] Furthermore, the current multimodal processing methods usually require multiple processing processes in series during implementation, such as voice-to-text conversion, text generation through a large model, and then text conversion back to voice, etc. Such process design not only increases the accumulation of errors, but also prolongs the processing time, thus affecting the user experience.

[0005] The above contents are only used to assist in understanding the technical solution of the present application and do not constitute an admission that the above contents are prior art. Summary of the invention

[0006] The main purpose of this application is to provide a multimodal large model training method, device and storage medium, aiming to solve the technical problem of insufficient performance of existing multimodal large models.

[0007] To achieve the above objectives, the present application proposes a multimodal large model training method, which is applied to the training of a multimodal large model. In the model architecture of the multimodal large model, multiple single modalities are mounted. The method includes:

[0008] Acquire training samples and determine the data type of the training samples;

[0009] A single modality corresponding to the data type is determined from the model architecture of the multimodal large model, and the training samples are subjected to time alignment training using the single modality, wherein the single modality is pre-trained based on the corresponding single modality training samples and then mounted to the model architecture of the multimodal large model.

[0010] In one embodiment, the step of pre-training the single modality based on the corresponding single modality training sample and then mounting it to the model architecture of the multimodal large model includes:

[0011] Extracting training data of the single-modality training sample to generate an input sequence, and inputting the input sequence into the single-modality for data processing;

[0012] The output sequences of the data processing are mapped to the same feature space.

[0013] In one embodiment, the multimodal large model training method further includes:

[0014] A new hybrid expert model is created, and the model architecture of the multimodal large model is constructed using the hybrid expert model;

[0015] The pre-trained single modality is mounted on the multimodal large model.

[0016] In one embodiment, the step of creating a new hybrid expert model and constructing a model architecture of the multimodal large model with the hybrid expert model includes:

[0017] Create a new input and output encoder according to the model architecture;

[0018] The input-output encoder is arranged in the multimodal large model.

[0019] In one embodiment, the step of creating a new hybrid expert model and constructing a model architecture of the multimodal large model with the hybrid expert model includes:

[0020] A new internal attention mechanism is created, where the internal attention mechanism includes a modality internal attention mechanism and a cross-modality attention mechanism;

[0021] The internal attention mechanism is fed into the mixture of experts model.

[0022] In one embodiment, after the step of mounting the pre-trained single modality to the multimodal large model, the method further includes:

[0023] Determining the processing data type of the single modality;

[0024] An activation mode based on the processed data type is newly created, and the activation mode is stored in the gating layer of the hybrid expert model.

[0025] In one embodiment, the step of determining a single modality corresponding to the data type from the model architecture of the multimodal large model and performing time alignment training on the training samples using the single modality includes:

[0026] Determining an activation mode corresponding to the single mode;

[0027] Before the step of performing time alignment training on the training samples in the single mode, the method further includes:

[0028] The single mode is activated in the activation mode.

[0029] In one embodiment, the step of determining a single modality corresponding to the data type from the model architecture of the multimodal large model and performing time alignment training on the training samples using the single modality includes:

[0030] The unimodal output sequence is subjected to GPT function evaluation, and a training result of the timing alignment training is generated according to the evaluation result.

[0031] In addition, to achieve the above-mentioned objectives, the present application also proposes a multimodal large model training device, which includes: a memory, a processor, and a computer program stored on the memory and executable on the processor, and the computer program is configured to implement the steps of the multimodal large model training method as described above.

[0032] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by the processor, the steps of the multimodal large model training method described above are implemented.

[0033] One or more technical solutions proposed in this application have at least the following technical effects:

[0034] The technical solution of the present application is applied to the training of a large multimodal model. In the model architecture of the large multimodal model, multiple single modalities are mounted, and training samples are obtained and the data type of the training samples is determined; the single modality corresponding to the data type is determined from the model architecture of the large multimodal model, and the training samples are subjected to time alignment training with the single modality, wherein the single modality is pre-trained based on the corresponding single modal training samples and then mounted to the model architecture of the large multimodal model. The model architecture of the large multimodal model used in the present application integrates multiple modal information to achieve efficient feature alignment and time synchronization, greatly improving the efficiency of the processing mechanism and thus achieving the technical effect of improving the performance of the large multimodal model. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0036] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0037] Figure 1 A flowchart of the first embodiment of the multimodal large model training method of the present application is provided;

[0038] Figure 2 This is a schematic diagram of the model architecture of a multimodal large model;

[0039] Figure 3 This is a schematic diagram of the device structure of the hardware operating environment involved in the multimodal large model training method in the embodiment of the present application.

[0040] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0041] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0042] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0043] The main solution of the embodiment of the present application is: applied to the training of a large multimodal model, wherein a plurality of single modalities are mounted in the model architecture of the large multimodal model, by obtaining training samples and determining the data type of the training samples; determining the single modality corresponding to the data type from the model architecture of the large multimodal model, and performing time alignment training on the training samples with the single modality, wherein the single modality is pre-trained based on the corresponding single modality training samples and then mounted to the model architecture of the large multimodal model.

[0044] The obvious deficiencies in the existing processing mechanism of large multimodal models lead to the technical problem of low application performance.

[0045] The present application provides a solution. The model architecture of the multimodal large model used in the present application integrates multiple modal information, realizes efficient feature alignment and time synchronization, greatly improves the efficiency of the processing mechanism, and thus achieves the technical effect of improving the performance of the multimodal large model.

[0046] Based on this, the present application embodiment provides a multi-modal large model training method, referring to Figure 1 , Figure 1 The flowchart of the first embodiment of the multimodal large model training method of the present application is shown in FIG. The multimodal large model training method is applied to the training of the multimodal large model. In the model architecture of the multimodal large model, multiple single modalities are mounted. In this embodiment, the multimodal large model training method includes steps S10 to S20:

[0047] Step S10, obtaining training samples and determining the data type of the training samples;

[0048] Step S20, determining a single modality corresponding to the data type from the model architecture of the multimodal large model, and performing time alignment training on the training samples with the single modality, wherein the single modality is pre-trained based on the corresponding single modality training samples and then mounted to the model architecture of the multimodal large model.

[0049] In this embodiment, the model architecture of the pre-built multimodal large model is used to perform time alignment training on the training samples. In the model architecture of the multimodal large model, the hybrid expert model is the backbone structure, and multiple single modalities are mounted. By performing end-to-end time alignment training on the training samples with the mounted single modalities, it can be viewed Figure 2 , Figure 2 Schematic diagram of the model architecture of the multimodal large model.

[0050] Optionally, the number of single modalities mounted in the multimodal large model is not fixed, and the mounted single modalities can be increased and / or decreased based on the needs of the current timing alignment training. Alternatively, the number of mounted single modalities currently mounted can be increased and / or decreased according to the number of mounts that the model architecture of the multimodal large model can carry.

[0051] Among them, the single modality refers to a method or model for processing or analyzing a single type of data, and the single modality can only be used as a data model for processing a specific type of data (such as text, image, audio, etc.). In general, the single modality can be designed and optimized for data of a certain modality, thereby optimizing the best performance achieved by a specific task. In this way, after obtaining the training sample, the corresponding single modality is determined based on the data type of the training sample. The training sample may contain one or more data types. For this reason, the single modality determined for the training sample should also be one or more.

[0052] When the single modality is mounted on the model structure of the multimodal large model, it is possible that one or more single modalities will perform the training process at the same time. For this purpose, the feature information of the single modality needs to be unified, that is, pre-training is performed on the newly created single modality to map the features of multiple single modalities to the same feature space. This mapping process is essentially a feature fusion technology that combines the features of different modalities to form a unified representation. Therefore, after the single modality is pre-trained based on the corresponding single modality training sample, the step of mounting it on the model architecture of the multimodal large model includes:

[0053] Extracting training data of the single-modality training sample to generate an input sequence, and inputting the input sequence into the single-modality for data processing;

[0054] The output sequences of the data processing are mapped to the same feature space.

[0055] When creating a single modality, it is necessary to clarify the concept of the single modality. In this embodiment, the single modality refers to a machine learning or deep learning model that specifically processes a specific type of data (such as text, images, audio, etc.). Its new benchmark is usually designed and optimized for data of a certain modality to achieve the best performance on a specific task. In this way, when creating the single modality, it is necessary to design its data structure based on the data type to be processed and the best performance as the benchmark to generate the single modality.

[0056] Therefore, based on the fact that the data type being processed is single, when creating the data structure of a single-modality model, it is usually relatively simple and highly targeted. The process of creating the single-modality model is as follows:

[0057] Clarify tasks and data types: Clarify the specific problems that the unimodal model needs to solve, such as image classification, text generation, or speech recognition, determine the corresponding data type based on the specific problem, and select the corresponding data type for processing, such as text, image, or audio;

[0058] Select model architecture: You need to select the corresponding model structure according to the type of data being processed. For example, for text data, choose a model architecture such as recurrent neural network (RNN) and Transformer; for shower image data, choose a model architecture such as convolutional neural network (CNN); for audio data, choose a deep neural network (DNN) combined with signal processing technology, etc. In addition, select the appropriate model depth and complexity based on the complexity and performance requirements of the data being processed.

[0059] Model training and optimization: first prepare the training data, divide the preprocessed data into training set, validation set and test set; then set the training parameters, such as learning rate, batch size, number of iterations, etc.; then train the model based on the training model using the training data, and update the model parameters through back propagation algorithm, gradient descent and other methods.

[0060] Model evaluation and optimization: Use the validation set to evaluate model performance and adjust model parameters or architecture based on the evaluation results to improve model performance.

[0061] Model deployment and application: Export the trained model to a deployable format, such as TensorFlow's SavedModel, PyTorch's .pth file, etc.

[0062] Deployment environment configuration: configure the necessary software and hardware resources in the target environment, that is, pre-build a multimodal end-to-end model structure with a hybrid expert model as a deep learning framework and dependency library for mounting the single modality.

[0063] Model integration and application: The generated single modality is mounted into the model architecture of the multimodal large model to implement specific data type processing.

[0064] As shown above, after creating a corresponding single modality based on the data type, pre-training is required for the single modality to clarify the feature information of the single modality.

[0065] To this end, a unimodal training sample is obtained, so as to perform a pre-training operation on the unimodal. The unimodal training sample is essentially a collection and arrangement of data types that can be processed by the unimodal, or in other words, when there are multiple unimodalities, the corresponding pre-training data can be generated into data texts of one or more separate unimodal training samples based on the processed data types of each unimodal. To this end, the unimodal is pre-trained based on the text data of the unimodal training sample.

[0066] During pre-training, the input and output data need to be processed through the encoder and decoder. The text data of the unimodal training samples can be processed based on a pre-built Encoder-Decoder model, or the text data can be processed using the Encoder-Decoder model included in the unimodal data structure.

[0067] Among them, the Encoder-Decoder model can be indicated as an encoder (Encoder) and a decoder (Decoder), the encoder converts the input data into a feature vector of fixed dimension, and the decoder reconstructs the original data from the feature vector or performs other tasks (such as classification, generation, etc.). Afterwards, according to the training task requirements of the current pre-training of the single modality, a suitable loss function is selected, for example, for the reconstruction task, the mean square error (MSE) or cross entropy loss (Cross-Entropy Loss) is used. In practical applications, for different pre-training, multiple different loss functions can be set for selection during the pre-training process.

[0068] In the above, based on the setting of the Encoder-Decoder model and the loss function, the single modality is pre-trained. The pre-training training process can be performed separately for the data of the single modality, or multiple single modalities can be trained at the same time. The pre-training status of each single modality is determined according to the output data of the pre-training and the output data is mapped to the corresponding feature space. In this way, the single modality is end-to-end pre-trained by using an encoder-decoder, and the output data of each single modality is mapped to the feature space of the same dimension. The purpose of the pre-training is to realize the fusion and interaction of multimodal data, improve the generalization ability of each single model, and realize cross-modal information retrieval and recommendation.

[0069] In the above, based on the multiple single modes mounted in the model architecture of the multimodal large model, in actual use, it is also necessary to perform relevant execution steps based on the data structure of the model architecture of the multimodal large model. Generally, the model architecture of the multimodal large model is a composite data structure with a backbone structure. Multiple functional modules are set based on the backbone structure and multiple single modes are mounted to realize the application. Based on the creation of the multimodal large model, that is, the multimodal large model training method, it also includes:

[0070] A new hybrid expert model is created, and the model architecture of the multimodal large model is constructed using the hybrid expert model;

[0071] The pre-trained single modality is mounted on the multimodal large model.

[0072] In one embodiment, the mixed expert model can be defined as Transformer MoE, which is essentially a combination of a neural network architecture based on a mixed expert model (Mixture of Experts, MoE for short) and a Transformer model, wherein the basic structure of the Transformer model consists of two parts: an encoder and a decoder.

[0073] In another embodiment, the hybrid expert model can also be a deep learning architecture that combines multiple expert models (usually neural networks) with a gating network, which can decompose complex tasks into multiple subtasks and assign them to different expert models for processing. Based on the application of the hybrid expert model, higher computing efficiency and better performance can be achieved. In this embodiment, the gating network is characterized as a gating layer of the hybrid expert model. The gating layer is a key component based on the hybrid expert model architecture, plays the role of a router, and is responsible for dynamically selecting and assigning tasks to appropriate single modalities based on the characteristics of the input data. In this embodiment, the gating layer is also used to store activation modes for activating each single modality.

[0074] Moreover, based on the working principle of the hybrid expert model, it can be summarized as a process of "task decomposition-expert processing-result aggregation". Specifically, when the hybrid expert model receives data input, the gating layer analyzes the characteristics of the input data and determines the corresponding expert network based on the characteristics to activate the expert network to process the input data. That is, each of the activated expert networks independently processes a part of the input data and generates corresponding outputs. Finally, the outputs of all expert networks are aggregated to form the final model output. With respect to the working principle shown above, the expert network is the mounted single modality, that is, the input data is processed by mounting multiple single modalities that process input data separately on the hybrid expert network.

[0075] In addition, the application of the single modality mounted in the hybrid expert network involves the analysis of input data and output data. Therefore, in a feasible implementation manner, the step of creating a new hybrid expert model and constructing the model architecture of the multimodal large model with the hybrid expert model includes:

[0076] Create a new input and output encoder according to the model architecture;

[0077] The input-output encoder is arranged in the multimodal large model.

[0078] In this embodiment, based on the hybrid expert model as the backbone model, when the multiple single modalities mounted thereon process different types of input data respectively, they need to perform input and output data processing based on the data processing type to which each single modality belongs. To this end, a new input and output encoder based on the single modality is created to serialize the processed data of the single modality and then input it into the single modality for calculation and corresponding output.

[0079] In practical applications, the input and output realized by the input-output encoder are defined as the encoder and decoder of the previous step, respectively. The encoder is responsible for encoding the input sequence, that is, the training data of the training sample is serialized and input, and the encoder processes the input sequence and outputs the serialized data after processing. In addition, one or more hidden states are generated according to the data information generated by the input and output process (indicating states that cannot be directly observed in the model. These states correspond to "hidden" states inside a system, which are the real state sequences of the system, usually expressed as S = {S1, S2, ..., ST}, where each state St independently takes a value in a discrete state set at a certain time t, specifically characterized by non-direct observability and time dependence). In this embodiment, the hidden state usually contains the complete information of the input sequence and can be used by the decoder to generate the output sequence.

[0080] The detailed analysis of the encoder and the decoder is as follows:

[0081] The encoder is an encoder that can compile and convert input signals (such as bit streams) or input data (i.e., data information recorded in training samples) into signal forms that can be used for communication, transmission and storage. Therefore, its main function is to process input data and convert it into an internal representation that captures the important features of the input data, defined as context vectors (context vectors are representations of words or phrases in a specific position or range in a text, which are used to represent the semantic information of the text and help the model understand and process text data).

[0082] The Decoder is a decoder that can convert the byte sequence or signal encoded by the encoder into original data or a set of character programs. Its conversion process is related to the output data. Therefore, its main function is to receive the encoded data processed by the encoder and decode the encoded data to restore the original data or signal. Its work content involves operations such as deserialization, parsing and conversion of data.

[0083] As mentioned above, the application of the input-output encoding can actually be characterized as a sequence-based arrangement and docking process, which is defined as Timestamp embedding. The introduction of Timestamp embedding is to solve the problem of the order of elements in the sequence. This is because the neural network cannot directly understand the order of elements in the sequence. Therefore, the Timestamp embedding is represented as a vector associated with each element in the input sequence, indicating the position of the element in the sequence. This application can solve the problem of the inability to understand the position of elements in the sequence derived from the position independence of the neural network. Therefore, it is used as an introduction position embedding so that the model can understand the order represented by the elements in the sequence.

[0084] In the above, based on the newly built input-output encoder, the data processor as the backbone structure is combined with the hybrid expert model to build the model architecture of the multimodal large model, which can also be considered as deploying the input-output encoder to the hybrid expert model, or the model architecture of the multimodal large model. Specifically, it can be determined based on the actual data processing logic of the hybrid expert model or the multimodal end-to-model architecture. In a feasible implementation, the step of building a new hybrid expert model and building the model architecture of the multimodal large model with the hybrid expert model includes:

[0085] A new internal attention mechanism is created, where the internal attention mechanism includes a modality internal attention mechanism and a cross-modality attention mechanism;

[0086] The internal attention mechanism is fed into the mixture of experts model.

[0087] In this embodiment, an internal attention mechanism is provided for the hybrid expert model, which is defined as the Attention mechanism inside the MOE. The core of the Attention is to perform a weighted summation on each element in the input sequence. The size of the weight depends on the correlation or similarity between the current task goal (or Query) and each element, so that when applied, the hybrid expert model can selectively focus on important parts of the input sequence to improve the accuracy and efficiency of task processing. To this end, the newly created internal attention mechanism is filled into the hybrid expert model for application.

[0088] In practical applications, the Attention mechanism includes the intra-modal attention mechanism (Intra-modal Attention) and the cross-modal attention mechanism (Cross-modal Attention); the intra-modal attention mechanism means that within the same modality, the hybrid expert model can focus on the most important part of the input data. For example, when processing image data, the hybrid expert model gives priority to key areas or features in the image; when processing text data, the hybrid expert model gives priority to keywords or sentences in the text. The cross-modal attention mechanism means that between different single modalities, the hybrid expert model establishes associations and pays attention to the corresponding or complementary parts of different single modal data. For example, in the scenario of joint processing of images and texts, the hybrid expert model gives priority to the correspondence between a certain area in the image and a certain description in the text.

[0089] Furthermore, in the model architecture of the multimodal large model with the hybrid expert model as the backbone structure, the cross-modal attention mechanism can be realized through interaction or collaboration between different single modalities. For example, one single modality specializes in processing image data, and the other single modality specializes in processing text data. The two single modalities exchange information or share attention weights through a certain mechanism (such as a gating layer), thereby realizing the application of the cross-modal attention mechanism. Moreover, the multiple single modalities possessed in the model architecture of the multimodal large model all have the ability to process specific modal data, and the ability to focus on important parts of the modal data through an internal attention mechanism.

[0090] Furthermore, when constructing the model architecture of the multimodal large model through the hybrid expert model, when the multiple single modalities mounted thereon process the training samples, since the data types contained in the training samples may include one or more, for this reason, the multimodalities processed for each data type are also one or more. In order to realize the application of the training sample time alignment training, multiple single modalities need to process data simultaneously. Therefore, after the step of mounting the pre-trained single modality to the multimodal large model, it also includes:

[0091] Determining the processing data type of the single modality;

[0092] An activation mode based on the processed data type is newly created, and the activation mode is stored in the gating layer of the hybrid expert model.

[0093] According to the single mode to be mounted currently, the data type that the single mode can process is determined. The processed data type is the data type of a certain type of data that it can process, such as multimedia data, text, images and other data types with data forms. A corresponding activation mode is newly created for the data type, and the activation mode is stored in the gating layer of the hybrid professional model. When storing the activation mode, an association relationship is created between the activation mode and the corresponding single mode, which is used to instantly call and activate the corresponding single mode.

[0094] In this embodiment, the activation mode created for the processing data type of the unimodal actually means that the unimodal can be correctly activated and generate corresponding output when processing a specific type of input data. Therefore, by setting the corresponding activation function as the activation mode of the unimodal, the activation function is the key factor in determining whether the neuron should be activated (i.e., output a non-zero value). For the activation task of the unimodal, by selecting a suitable activation function to adapt the performance of the unimodal processing data, for example, when the unimodal is an image processing type, ReLU and its variants (such as Leaky ReLU, PReLU, etc.) are used as the activation function of the unimodal. The specific setting of the activation function is related to the processing data type of each unimodal. The activation function that best meets the performance of the unimodal is selected as the activation mode, and is stored in the gating layer of the hybrid expert model.

[0095] The above describes that when there is a need for timing alignment training, a model architecture of a multimodal large model is constructed based on a hybrid expert model, and multiple single modalities are mounted into the model architecture of the multimodal large model so that the training samples can be time-aligned through the application of each single modality.

[0096] In the process of performing time alignment training on training samples based on the model architecture of the constructed multimodal large model, it is also necessary to activate the single mode corresponding to the data type of the training sample, that is, the step of determining the single mode corresponding to the data type from the model architecture of the multimodal large model and performing time alignment training on the training samples with the single mode includes:

[0097] Determining an activation mode corresponding to the single mode;

[0098] Before the step of performing time alignment training on the training samples in the single mode, the method further includes:

[0099] The single mode is activated in the activation mode.

[0100] According to the determined single mode, the activation mode of the single mode is determined from the gating layer. Since the gating layer stores all activation modes of the currently mounted single mode, and the activation mode is stored in a manner that uses the unique information of the single mode as an identifier to create an association relationship between the single mode and the corresponding activation mode, the confirmation method for determining the corresponding activation mode based on the single mode can be defined as obtaining the corresponding activation mode from the association relationship stored in the gating layer using the unique information of the single mode as an identifier.

[0101] Furthermore, for the acquired activation mode, before the training samples are subjected to timing alignment training, the corresponding single modality is activated with the activation mode, wherein, since the number of the single modalities is one or more, the single modalities activated simultaneously are one or more, that is, one or more of the single modalities can be activated simultaneously, thereby performing alignment training on the training samples with the single modality.

[0102] Furthermore, according to the analysis of the activation mode shown above, the activation process for the single modality can also be defined as processing the sample data in the training samples, and splitting the sample data into different types of data, that is, separating the sample data of the same type from the training samples as input data of the corresponding single modality, thereby performing timing alignment training on the input data through the encoder and decoder of the single modality.

[0103] Regarding the processing of input data by the unimodal encoder and decoder, in the above definition, the input data is sequenced by the encoder to obtain an input sequence, and an output sequence is generated by the decoder, wherein Timestamp embedding is applied to process the order of elements in the output sequence, that is, the vector associated with each element in the input sequence indicated by the Timestamp embedding is used to indicate the position of the element in the sequence, and the output sequence is sequentially processed to obtain the final output data.

[0104] In addition, considering the training results of the timing alignment training, the processed output data can be evaluated based on the GPT function, so as to obtain the performance of the model architecture of the constructed multimodal large model based on the training results. That is, the step of determining a single mode corresponding to the data type from the model architecture of the multimodal large model, and performing timing alignment training on the training samples with the single mode, includes:

[0105] The single-modal output sequence is subjected to GPT function evaluation, and a training result of the timing alignment training is generated according to the evaluation result.

[0106] In this embodiment, after processing the input data through each of the single modalities and outputting the data, the output data of the multiple single modalities are sequentially processed to obtain the final output data based on the current training sample, and the loss function of GPT (Generative Pre-trained Transformer) is applied to the performance evaluation of the model architecture training process of the multimodal large model. The loss function, called the loss function or cost function, is used to evaluate the gap between the prediction result of the model on the training data and the true value. In the process of O-type training, the selection and optimization of the loss function has an important impact on the performance and effect of the model. In practical applications, the commonly used loss functions of the GPT include the cross entropy loss function and other loss functions, wherein the cross entropy loss function is a common method for measuring the difference between the probability distribution generated by the model and the true label. Specifically in the embodiment of the present application, the cross entropy loss function is used to calculate the difference between the probability distribution of the predicted next word and the probability distribution of the real next word during the model architecture training process of the multimodal large model. By minimizing the cross entropy loss, the model architecture of the multimodal large model can learn how to generate predictions that are closer to the real text, thereby improving the performance of the model architecture of the multimodal large model.

[0107] The other Loss functions are that in addition to the cross entropy loss function, GPT may also use other types of loss functions, such as the mean square error (MSE) loss function or the perplexity (Perplexity) loss function. The choice of the loss function depends on the application scenario and requirements of the model architecture of the multimodal large model. For example, the mean square error loss function is generally used to measure the degree of difference between the prediction result of the regression model and the true value. Specifically, it is set according to the actual application scenario and training requirements of the model architecture of the multimodal large model.

[0108] In addition, in order to avoid overfitting the model architecture of the multimodal large model to the training data, it is usually necessary to regularize the Loss function. Regularization is used to reduce the complexity of the model architecture of the multimodal large model and improve the generalization ability. Among them, common regularization methods include L1, L2 regularization, as well as Dropout, Batch Normalization, etc. Moreover, during the training process of the model architecture of the multimodal large model, the Loss function needs to be evaluated and iterated, wherein, by calculating the loss value of the model architecture of the multimodal large model, the training progress of the model architecture of the multimodal large model can be understood, and timely adjustments can be made.

[0109] In addition, other indicators, such as accuracy, F1 value, etc., can also be used to measure the performance of the model architecture of the multimodal large model based on the output data, and to iteratively optimize the Loss function.

[0110] From the above, it is clear that after constructing the model architecture of the multimodal large model by using the hybrid expert model as the backbone model, multiple single modalities need to be mounted on the multimodal end-to-model architecture in order to realize training based on the multimodal large model. To this end, multiple single modalities need to be newly created to realize the mounting of the model architecture of the multimodal large model.

[0111] In this embodiment, the model architecture of the multimodal large model is applied to integrate multiple single-modal information, realize efficient feature alignment and time synchronization, greatly improve the efficiency of the processing mechanism, and thus achieve the technical effect of improving the performance of the multimodal large model.

[0112] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the multimodal large model training method of the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.

[0113] The present application provides a multimodal large model training device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the multimodal large model training method in the above-mentioned embodiment one.

[0114] Reference below Figure 3 , which shows a schematic diagram of the structure of a multimodal large model training device suitable for implementing the embodiment of the present application. The multimodal large model training device in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 3 The multimodal large model training device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0115] like Figure 3As shown, the multimodal large model training device may include a processing device 1001 (such as a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the multimodal large model training device are also stored. The processing device 1001, ROM1002 and RAM1004 are connected to each other via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the I / O interface 1006: input devices 1007 including, for example, a touch screen, a touchpad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 1003 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 1009. The communication device 1009 can allow the multimodal large model training device to communicate wirelessly or wired with other devices to exchange data. Although the figure shows a multimodal large model training device with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems may be implemented or have instead.

[0116] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0117] The multimodal large model training device provided by the present application adopts the multimodal large model training method in the above embodiment, which can solve the technical problem that the existing multimodal large model has low performance due to the shortcomings in the processing mechanism. Compared with the prior art, the beneficial effects of the multimodal large model training device provided by the present application are the same as the beneficial effects of the multimodal large model training method provided by the above embodiment, and the other technical features in the multimodal large model training device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0118] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0119] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0120] The present application provides a storage medium, which is a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, and the computer-readable program instructions are used to execute the multimodal large model training method in the above-mentioned embodiment.

[0121] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination of the above.

[0122] The above-mentioned computer-readable storage medium may be included in the multimodal large model training device; or it may exist independently without being assembled into the multimodal large model training device.

[0123] The above-mentioned computer-readable storage medium carries one or more programs. When the above-mentioned one or more programs are executed by the multimodal large model training device, the multimodal large model training device implements the technical content of the multimodal large model training method embodiment shown above.

[0124] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0125] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0126] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0127] The readable storage medium provided in this application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned multimodal large model training method, and can solve the technical problem that the existing multimodal large model has low performance due to deficiencies in the processing mechanism. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as the beneficial effects of the multimodal large model training method provided in the above-mentioned embodiment, and will not be repeated here.

Claims

1. A multimodal large model training method, characterized in that: Applied to the training of a multimodal large model, in which a plurality of single modalities are mounted in the model architecture of the multimodal large model, the method includes: Acquire training samples and determine the data type of the training samples; A single modality corresponding to the data type is determined from the model architecture of the multimodal large model, and the training samples are subjected to time alignment training using the single modality, wherein the single modality is pre-trained based on the corresponding single modality training samples and then mounted to the model architecture of the multimodal large model.

2. The multimodal large model training method according to claim 1, characterized in that: The step of pre-training the single modality based on the corresponding single modality training sample and then mounting it to the model architecture of the multimodal large model includes: Extracting training data of the single-modality training sample to generate an input sequence, and inputting the input sequence into the single-modality for data processing; The output sequences of the data processing are mapped to the same feature space.

3. The multimodal large model training method according to claim 1, characterized in that: The multimodal large model training method further includes: A new hybrid expert model is created, and the model architecture of the multimodal large model is constructed using the hybrid expert model; The pre-trained single modality is mounted on the multimodal large model.

4. The multimodal large model training method according to claim 3, characterized in that: The step of creating a new hybrid expert model and constructing the model architecture of the multimodal large model with the hybrid expert model includes: Create a new input and output encoder according to the model architecture; The input-output encoder is arranged in the multimodal large model.

5. The multimodal large model training method according to claim 3, characterized in that: The step of creating a new hybrid expert model and constructing the model architecture of the multimodal large model with the hybrid expert model includes: A new internal attention mechanism is created, where the internal attention mechanism includes a modality internal attention mechanism and a cross-modality attention mechanism; The internal attention mechanism is fed into the mixture of experts model.

6. The multimodal large model training method according to claim 3, characterized in that: After the step of mounting the pre-trained single modality onto the multimodal large model, the method further includes: Determining the processing data type of the single modality; An activation mode based on the processed data type is newly created, and the activation mode is stored in the gating layer of the hybrid expert model.

7. The multimodal large model training method according to claim 1, characterized in that: The step of determining a single modality corresponding to the data type from the model architecture of the multimodal large model, and performing time alignment training on the training samples using the single modality, comprises: Determining an activation mode corresponding to the single mode; Before the step of performing time alignment training on the training samples in the single mode, the method further includes: The single mode is activated in the activation mode.

8. The multimodal large model training method according to claim 1, characterized in that: The step of determining a single modality corresponding to the data type from the model architecture of the multimodal large model, and performing time alignment training on the training samples using the single modality, comprises: The unimodal output sequence is subjected to GPT function evaluation, and a training result of the timing alignment training is generated according to the evaluation result.

9. A multimodal large model training device, characterized in that: The device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the multimodal large model training method as described in any one of claims 1 to 8.

10. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the multimodal large model training method as described in any one of claims 1 to 8 are implemented.

Images