Multi-modal model training method and device

Abstract

The invention provides a multi-modal model training method and device. The method comprises the following steps: training each decomposition matrix in a multi-modal model by using training data until the loss value of a preset loss function is smaller than a first preset threshold value, and obtaining a trained multi-modal model; wherein the parameter matrix of the multi-modal model comprises a basic matrix and an increment matrix of each layer, the decomposition matrix is obtained by decomposing the increment matrix, one decomposition matrix corresponding to the increment matrix is a same-order matrix, other decomposition matrixes are non-same-order matrixes, and the total parameter quantity of the decomposition matrixes is smaller than the parameter quantity of the increment matrix. According to the method, the decomposition matrix with a small parameter quantity in the multi-modal model is trained, the fine tuning training time is shortened, the training efficiency of the model is improved, the demand of hardware video memory resources is reduced, the training cost is reduced, the precision improvement of downstream tasks of the multi-modal model is completed by using a small amount of video memory resources and training time, and the training efficiency of the multi-modal model is improved. And the method has relatively strong generalization ability.

Description

Technical Field

[0001] This application belongs to the technical field of multimodality, and particularly relates to a multimodal model training method and apparatus. Background Art

[0002] A multimodal large model refers to a deep learning model that can process multiple modalities of data simultaneously. It can process and fuse various types of data such as images, texts, and audios to achieve a more accurate understanding and description of complex scenarios.

[0003] Generally, a large amount of publicly available datasets are used to train multimodal large models, but these multimodal large models are general large models. When directly applied to specific fields, their performance cannot meet business requirements, and specific data needs to be used for fine-tuning so that the adjusted large model can be applied to specific fields.

[0004] However, the existing fine-tuning parameter technology fine-tunes and learns all parameters, which is costly and inefficient. Summary of the Invention

[0005] Embodiments of this application provide a multimodal model training method, apparatus, electronic device, readable storage medium, and computer program product, which can solve the problems of high cost and inefficiency of the existing fine-tuning parameter technology.

[0006] In a first aspect, embodiments of this application provide a multimodal model training method, including:

[0007] Obtain training data;

[0008] Use the training data to train each decomposition matrix in the multimodal model until the model loss value of the preset loss function is less than a first preset threshold to obtain a trained multimodal model;

[0009] Wherein, the parameter matrix of the multimodal model includes a base matrix and an increment matrix for each layer, the decomposition matrix is obtained by decomposing the increment matrix, one of the decomposition matrices corresponding to the increment matrix is a matrix of the same order, and the other decomposition matrices are matrices of different orders, and the total number of parameters of the decomposition matrix is less than the number of parameters of the increment matrix.

[0010] In a second aspect, embodiments of this application provide a multimodal model training apparatus, including:

[0011] An obtaining module, configured to obtain training data;

[0012] A training module, configured to use the training data to train each decomposition matrix in the multimodal model until the model loss value of the multimodal model is less than a preset threshold to obtain a trained multimodal model;

[0013] Among them, the parameter matrix of the multimodal model includes the basic matrix and the incremental matrix of each layer. The decomposition matrix is obtained by decomposing the incremental matrix. One of the decomposition matrices corresponding to the incremental matrix is a matrix of the same order, and the other decomposition matrices are matrices of different orders. The total number of parameters of the decomposition matrix is less than the number of parameters of the incremental matrix.

[0014] In a third aspect, an embodiment of the present application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the method described in any one of the above first aspects is implemented.

[0015] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium storing a computer program, and when the computer program is executed by a processor, the method described in any one of the above first aspects is implemented.

[0016] In a fifth aspect, an embodiment of the present application provides a computer program product. When the computer program product runs on an electronic device, the electronic device is caused to execute the method described in any one of the above first aspects.

[0017] The beneficial effects of the embodiments of the present application compared with the prior art are as follows:

[0018] In the embodiments of the present application, training data is obtained; each decomposition matrix in the multimodal model is trained using the training data until the model loss value of the preset loss function is less than the first preset threshold, and a trained multimodal model is obtained. Among them, the parameter matrix of the multimodal model includes the basic matrix and the incremental matrix of each layer. The decomposition matrix is obtained by decomposing the incremental matrix. One of the decomposition matrices corresponding to the incremental matrix is a matrix of the same order, and the other decomposition matrices are matrices of different orders. The total number of parameters of the decomposition matrix is less than the number of parameters of the incremental matrix, so as to train the decomposition matrix with a small number of parameters in the multimodal model, reduce the fine-tuning training time, improve the training efficiency of the model, reduce the demand for hardware video memory resources, and at the same time reduce the training cost, so as to achieve the improvement of the accuracy of the downstream tasks of the multimodal model using a small amount of video memory resources and training time, and has strong generalization ability.

[0019] It can be understood that the beneficial effects of the above second aspect to fifth aspect can refer to the relevant descriptions in the above first aspect, and will not be repeated here. Description of the Drawings

[0020] To more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the accompanying drawings required for use in the embodiments or the description of the prior art. Obviously, the accompanying drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other accompanying drawings can also be obtained based on these drawings.

[0021] Figure 1 is a schematic diagram of a system provided by an embodiment of the present application;

[0022] Figure 2 is the first flowchart of a multi-modal model training method provided by an embodiment of the present application;

[0023] Figure 3 is the second flowchart of a multi-modal model training method provided by an embodiment of the present application;

[0024] Figure 4 is the third flowchart of a multi-modal model training method provided by an embodiment of the present application;

[0025] Figure 5 is a schematic diagram of the structure of a multi-modal model training device provided by an embodiment of the present application;

[0026] Figure 6 is a schematic diagram of the structure of an electronic device provided by an embodiment of the present application. Detailed implementation manners

[0027] In the following description, for the purpose of illustration rather than limitation, specific details such as specific system structures and technologies are proposed to thoroughly understand the embodiments of the present application. However, those skilled in the art should clearly understand that the present application can also be implemented in other embodiments without these specific details. In other cases, the detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid unnecessary details from interfering with the description of the present application.

[0028] It should be understood that when used in the specification of the present application and the appended claims, the term "comprising" indicates the presence of the described features, wholes, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components, and / or their combinations.

[0029] It should also be understood that the term " / and / or" used in the specification of the present application and the appended claims refers to any combination and all possible combinations of one or more of the related listed items, and includes these combinations.

[0030] As used in the specification of this application and the appended claims, the term "if" may be construed, depending on the context, as "when", "once", "in response to determining", or "in response to detecting". Similarly, the phrase "if determined" or "if [the described condition or event] is detected" may be construed, depending on the context, to mean "once determined", "in response to determining", "once [the described condition or event] is detected", or "in response to detecting [the described condition or event]".

[0031] In addition, in the description of the specification of this application and the appended claims, the terms "first", "second", "third", etc. are used only for distinguishing descriptions and cannot be construed as indicating or implying relative importance.

[0032] Reference to "one embodiment" or "some embodiments" or the like described in the specification of this application means that a specific feature, structure, or characteristic described in connection with that embodiment is included in one or more embodiments of this application. Thus, statements such as "in one embodiment", "in some embodiments", "in other some embodiments", "in still other embodiments", etc. that appear in different places in this specification do not necessarily all refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "comprising", "including", "having", and their variants all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0033] Multimodal large models involve the fusion of multiple perception modalities (text and images) in a huge neural network, that is, deep learning models that can process multiple modalities of data simultaneously. They can process and fuse various types of data such as images, text, and audio, and achieve a more accurate understanding and description of complex scenarios.

[0034] Usually, a large number of publicly available datasets are used to train multimodal large models, but these multimodal large models are general large models. This makes it impossible to meet the business requirements when directly applied to specific fields, and specific data needs to be used for fine-tuning so that the adjusted large model can be applied to specific fields.

[0035] However, the existing fine-tuning parameter technology fine-tunes and learns all parameters, which is costly and inefficient.

[0036] Among them, the efficient parameter fine-tuning training technology is to use smaller labeled data to adjust the model parameters on the basis of a pre-trained model, so that the model is more suitable for specific tasks and the performance is improved. Fine-tuning is a more efficient and economical way.

[0037] Figure 1 This is a system example diagram provided by an embodiment of this application. As Figure 1As shown in the figure, the multimodal model includes a large vision model, an image-to-text adapter, and a large language model.

[0038] The image is input into the large vision model to obtain an image classification result. Then, the image classification result is input into the image-to-text adapter to obtain text features. Then, the text features and the text are input into the large language model to obtain the output text.

[0039] This multimodal model can integrate, understand, and apply image and text modal data.

[0040] Among them, the large vision model and the large language model can adopt open-source structures, such as: ViT (Vision Transformer), GPT (Generative Pre-Training), GLM (Generalized Linear Model), etc.

[0041] The multimodal model of the embodiment of the present application is applicable to applications where the input is two modalities of text and image, and the output is a single modality of text. The application scenarios include talking about pictures and generating stories based on picture-text pairs. When the multimodal model is trained with parameter-efficient fine-tuning, the multimodal model learns rich semantic representations and image concepts to adapt to different tasks.

[0042] Figure 2 It is the first process schematic diagram of the multimodal model training method provided by an embodiment of the present application.

[0043] As Figure 2 shown, the method includes:

[0044] S11: Obtain training data.

[0045] In the application, since the input of the multimodal model includes two modalities of image and text, the training data includes training images and training texts.

[0046] S12: Use the training data to train each decomposition matrix in the multimodal model until the model loss value of the preset loss function is less than the first preset threshold.

[0047] Among them, the parameter matrix of the multimodal model includes the basic matrix and the incremental matrix of each layer. The decomposition matrix is obtained by decomposing the incremental matrix. One of the decomposition matrices corresponding to the incremental matrix is a matrix of the same order, and the other decomposition matrices are matrices of different orders. The total number of parameters of the decomposition matrix is less than the number of parameters of the incremental matrix.

[0048] In the application, in order to reduce the training computing power of the large language model, it is considered to stack and train the matrices of each layer, that is, train the incremental matrices of each layer, and do not train the basic matrices of each layer, so as to realize fine-tuning of some parameters of the multimodal model.

[0049] Obtain the matrices of each layer of the language large model. For the matrices of each layer, represent the matrices of each layer as a base matrix and an incremental matrix. Specifically, the matrices of each layer are obtained by adding the base matrix and the incremental matrix.

[0050] Exemplarily, the matrices of each layer include W q S , W k S , W v S , W q S , W k S , W v S which are the matrices of the language large model based on the multi-head attention structure. s represents the s-th layer in the model.

[0051] W q S , W k S , W v S are represented by the base matrix and the incremental matrix.

[0052] Specifically, W q S = W` q S + △ W q S , W k S = W` k S + △ W k S , W v S = W` v S + △ W v S . The base matrix includes W` q S , W` k S , W` v S . The incremental matrix includes △ W q S , △ W k S , △ W v S .

[0053] To reduce the number of parameters, after obtaining the incremental matrix, the incremental matrix can be decomposed into at least two decomposition matrices.

[0054] In a possible implementation, according to the information of the incremental matrix, a decomposition method of the incremental matrix is determined. The incremental matrix is decomposed according to the decomposition method to obtain at least two decomposition matrices.

[0055] Due to △ W q S 、 △ W k S 、 △ W v S the structural similarity of, for △ W q S 、 △ W k S 、 △ W v S the same decomposition method is adopted.

[0056] Among them, since the order of the decomposition matrix can be set, usually the order of the decomposition matrix is set to be less than the number of parameters of the incremental matrix, so that the total number of parameters of the decomposition matrix is less than the number of parameters of the incremental matrix.

[0057] After the parameter matrix of the multimodal model includes the decomposition matrix, the multimodal model is trained using the training images and training texts, and each decomposition matrix is fine-tuned until the model loss value is less than the first preset threshold.

[0058] When the model loss value is less than the first preset threshold, a fine-tuned multimodal model is obtained so that the fine-tuned multimodal model can be applied to a specific field.

[0059] In a possible implementation, the preset loss function is:

[0060]

[0061] Among them, T is the length of the input text and output text of the multimodal model, y is the output text, t is the position from the model input to the model output, and p is the probability function.

[0062] In this embodiment, training data is obtained; each decomposition matrix in the multi-modal model is trained using the training data until the model loss value of the preset loss function is less than the first preset threshold, and a trained multi-modal model is obtained; wherein, the parameter matrix of the multi-modal model includes the base matrix and the increment matrix of each layer, the decomposition matrix is obtained by decomposing the increment matrix, one of the decomposition matrices corresponding to the increment matrix is a matrix of the same order and the other decomposition matrices are matrices of different orders, and the total number of parameters of the decomposition matrix is less than the number of parameters of the increment matrix, so as to realize training the decomposition matrix with a small number of parameters in the multi-modal model, reduce the fine-tuning training time, improve the training efficiency of the model, reduce the demand for hardware video memory resources, and at the same time reduce the training cost, so as to realize improving the accuracy of the downstream tasks of the multi-modal model using a small amount of video memory resources and training time, and has strong generalization ability.

[0063] In one embodiment, when the increment matrix is a matrix of the same order, the increment matrix includes a first decomposition matrix and a second decomposition matrix, the first decomposition matrix is a matrix of the same order, the second decomposition matrix is a matrix of different orders, and the number of parameters of the first decomposition matrix is less than the number of parameters of the second decomposition matrix.

[0064] In application, the corresponding decomposition method is determined according to the increment matrix being a matrix of the same order. The increment matrix can be decomposed into a matrix of the same order and a matrix of different orders to obtain the first decomposition matrix and the second decomposition matrix.

[0065] Correspondingly, the increment matrix can be obtained based on matrix operations on the first decomposition matrix and the second decomposition matrix.

[0066] In a possible implementation manner, the matrix of the same order to be decomposed is an n×n matrix of the same order. The n×n matrix of the same order is obtained by multiplying the transposed matrix of the r×n matrix of different orders, the r×r matrix of the same order, and the r×n matrix of different orders in sequence.

[0067] Specifically, the increment matrix is an n×n matrix of the same order, the r×r matrix of the same order is the first decomposition matrix, and the r×n matrix of different orders is the second decomposition matrix.

[0068] Wherein, r < n, the number of parameters of the r×r matrix of the same order is less than that of the r×n matrix of different orders, and the total number of parameters of the r×r matrix of the same order and the r×n matrix of different orders is less than that of the n×n matrix of the same order.

[0069] Exemplarily, the matrix of the same order to be decomposed: △ W q S , △ W q S =Q T DQ, △ W q S ∈R n×n, Q ∈ R r×n , D ∈ R r×r , r < n.

[0070] It should be noted that other decomposition methods only need to follow r < n.

[0071] In this embodiment, the incremental matrix includes a first decomposition matrix and a second decomposition matrix. The first decomposition matrix is a matrix of the same order, the second decomposition matrix is a matrix of different orders, and the number of parameters of the first decomposition matrix is less than that of the second decomposition matrix, so as to reduce the number of parameters participating in the training, and reduce the demand for hardware video memory resources and training time.

[0072] In one embodiment, when the incremental matrix is a matrix of different orders, the incremental matrix includes a third decomposition matrix and a fourth decomposition matrix. The third decomposition matrix includes a fifth decomposition matrix and a sixth decomposition matrix.

[0073] Among them, the third decomposition matrix is a matrix of the same order, the fourth decomposition matrix is a matrix of different orders, and the third decomposition matrix and the fourth decomposition matrix are sparse matrices.

[0074] In application, when the incremental matrix is a matrix of different orders, the incremental matrix needs to be divided into a matrix of the same order and a matrix of different orders to obtain the third decomposition matrix and the fourth decomposition matrix.

[0075] Correspondingly, the incremental matrix can be obtained by performing matrix operations based on the third decomposition matrix and the fourth decomposition matrix.

[0076] In one possible implementation, the incremental matrix is an n×m matrix of different orders, and the n×m matrix of different orders is obtained by multiplying an n×n matrix of the same order by an n×m matrix of different orders. The n×n matrix of the same order and the n×m matrix of different orders for the decomposition matrix are sparse matrices.

[0077] For example, the incremental matrix: △ W q S , △ W q S = A × B, △ W q S ∈ R n×m , A ∈ R n×n , B ∈ R n×m .

[0078] In application, to reduce the number of parameters participating in the training, the third decomposition matrix can be decomposed into a matrix of the same order and a matrix of different orders to obtain the fifth decomposition matrix and the sixth decomposition matrix.

[0079] The fifth decomposition matrix is a matrix of the same order, the sixth decomposition matrix is a matrix of different orders, the number of parameters of the fifth decomposition matrix is less than that of the sixth decomposition matrix, and the total number of parameters of the fifth decomposition matrix and the sixth decomposition matrix is less than that of the third decomposition matrix.

[0080] In a possible implementation, the matrix of the same order to be decomposed is an n×n matrix of the same order, and the n×n matrix of the same order is obtained by multiplying the transposed matrix of an r×n matrix of different orders, an r×r matrix of the same order, and an r×n matrix of different orders in sequence.

[0081] Specifically, the third decomposition matrix is an n×n matrix of the same order, the r×r matrix of the same order is the fifth decomposition matrix, and the r×n matrix of different orders is the sixth decomposition matrix.

[0082] Among them, r < n, the number of parameters of the r×r matrix of the same order is less than that of the r×n matrix of different orders, and the total number of parameters of the r×r matrix of the same order and the r×n matrix of different orders is less than that of the n×n matrix of the same order.

[0083] Exemplarily, the matrix of the same order to be decomposed: W1q S , △ W1q S = Q T DQ, △ W1q S ∈R n×n Q ∈ R r×n D ∈ R r×r r < n.

[0084] It should be noted that other decomposition methods only need to follow r < n.

[0085] In this embodiment, the incremental matrix includes a third decomposition matrix and a fourth decomposition matrix. The third decomposition matrix is a matrix of the same order, the fourth decomposition matrix is a matrix of different orders, and the third decomposition matrix and the fourth decomposition matrix are sparse matrices, which can reduce the number of parameters.

[0086] And by the third decomposition matrix including a fifth decomposition matrix and a sixth decomposition matrix, the fifth decomposition matrix is a matrix of the same order, the sixth decomposition matrix is a matrix of different orders, the number of parameters of the fifth decomposition matrix is less than that of the sixth decomposition matrix, and the total number of parameters of the fifth decomposition matrix and the sixth decomposition matrix is less than that of the third decomposition matrix, further reducing the number of parameters participating in training, reducing the demand for hardware video memory resources and training time.

[0087] It can be understood that through the decomposition of the incremental matrix described in the above embodiment, if n = 100 and r = 10, the number of parameters for fine-tuning can be reduced by 90%, saving about 90% of the hardware resources, and also reducing the training time.

[0088] Figure 3This is the second process schematic diagram of the multimodal model training method provided by an embodiment of the present application. As Figure 3 shown, training each decomposition matrix in the multimodal model using training data until the model loss value of the preset loss function is less than the first preset threshold includes:

[0089] S31: Input the training image into the visual large model in the multimodal model to obtain the image classification result output by the visual large model.

[0090] Among them, the training data includes training images and training texts.

[0091] In the application, the visual large model analyzes the training image to obtain the image classification result of the training image.

[0092] S32: Input the image classification result into the image-to-text adapter in the multimodal model, so that the image-to-text adapter multiplies the image classification result by the dimensionality reduction matrix to obtain the dimensionality-reduced feature, and multiplies the dimensionality-reduced feature by the dimensionality-increasing matrix to obtain the text feature, and then outputs the text feature.

[0093] Exemplarily, the image classification result output by the visual large model is x v with a dimension of j, the dimension of the dimensionality-reduced feature is l, and the dimensionality reduction matrix is W d ∈R j×l . The dimension of the dimensionality-increased feature is k, and the dimensionality-increasing matrix is W u ∈R l×k . The expression of the text feature is: x n =W u W d x v , x v =b(x1), where x1 is the training image and b is the algorithm of the visual large model.

[0094] Among them, the values of l and k follow k < j and l < k. Generally, l takes a smaller value, and k is half the length of j.

[0095] S33: Use the text feature and the training text to train each decomposition matrix, the dimensionality reduction matrix, and the dimensionality-increasing matrix until the model loss value of the preset loss function is less than the first preset threshold.

[0096] In the application, since there is an image-to-text adapter in the multimodal model, based on the preset loss function and the supervised training data, each decomposition matrix, the dimensionality reduction matrix, and the dimensionality-increasing matrix are trained simultaneously. When the model loss value is less than the first preset threshold, a fine-tuned multimodal model is obtained, so that the fine-tuned multimodal model can better fuse multimodal data related to a specific field and better understand and process different types of data.

[0097] In a possible implementation, the language large model adopts a pre-trained large language model, and the output text of the pre-trained large language model is y, y = g(x n , x2) = g(W u W d x v , x2).

[0098] Meanwhile, the pre-trained large language model can be used to align text and image data.

[0099] In this embodiment, the image classification result is multiplied by the dimensionality reduction matrix through the image-to-text adapter to obtain the dimensionality-reduced features, and the dimensionality-reduced features are multiplied by the dimensionality increase matrix to obtain the text features, and then the text features are output, so that the text features have better feature expression.

[0100] And by using the text features and the training text to train each decomposition matrix, dimensionality reduction matrix, and dimensionality increase matrix until the model loss value of the preset loss function is less than the first preset threshold, and at the same time fine-tuning each decomposition matrix, dimensionality reduction matrix, and dimensionality increase matrix of the multimodal model, so that the fine-tuned multimodal can better fuse the multimodal data related to the specific field, and reduce the demand for hardware video memory resources and training time.

[0101] After obtaining the trained multimodal model, the trained multimodal model can be applied to the application scenario to which the training data belongs. The image and text instructions are input into the trained multimodal model, and the trained multimodal model determines the corresponding text description according to the input image and text instructions.

[0102] Exemplarily, the image shows a person holding a shovel in his hand, the shovel is inserted into the soil, and there is a tree in the soil. The text instruction is to describe the image, and the text description output by the trained multimodal model is that there is a person planting a tree.

[0103] Figure 4 is the third process schematic diagram of the multimodal model training method provided by an embodiment of the present application. As Figure 4 shown, before the image classification result is input into the image-to-text adapter of the multimodal model, it further includes:

[0104] S34: Using the image classification result and the training text to train the adapter until the model loss value of the preset loss function is less than the second preset threshold, and obtaining the image-to-text adapter.

[0105] In the application, based on the preset loss function, the adapter is trained by using the image classification result and the training text. When the model loss value is less than the second preset threshold, the image-to-text adapter is obtained, so that the multimodal model can establish a unified data representation and can understand and process different types of data.

[0106] In one embodiment, the method further includes:

[0107] S41: Normalize and perform dropout processing on the dimension-reduced features to obtain first transformed features.

[0108] Among them, dropout is a method for optimizing deep learning model parameters. Specifically, in each training batch, by ignoring half of the feature detectors (setting the values of half of the hidden layer nodes to 0), the overfitting phenomenon is reduced to reduce the interaction between feature detectors (hidden layer nodes).

[0109] Normalization is to normalize the data to the interval [-1, 1] to avoid differences in the order of magnitude of the data.

[0110] S42: Multiply the first transformed features by the dimension-raising matrix to obtain dimension-ascended features.

[0111] S43: Normalize and perform dropout processing on the dimension-ascended features to obtain second transformed features.

[0112] In application, use the second transformed features and the training text to train each decomposition matrix, dimension-reduction matrix, and dimension-raising matrix until the model loss value of the preset loss function is less than the first preset threshold.

[0113] Exemplarily, the expression of the second transformed features is: x n ` = f2(W u f1(W d x v )), where f2 includes the transformation of normalization and dropout after dimension-ascension, and f1 includes the transformation of normalization and dropout after dimension-reduction.

[0114] In this embodiment, by normalizing and performing dropout processing on the dimension-reduced features to obtain first transformed features, and normalizing and performing dropout processing on the dimension-ascended features to obtain second transformed features, the parameters of the image-to-text adapter are optimized to improve the training effect.

[0115] It should be understood that the magnitudes of the sequence numbers of the steps in the above embodiments do not mean the order of execution. The execution order of each process should be determined according to its function and internal logic, and should not constitute any limitation to the implementation process of the embodiments of the present application.

[0116] Corresponding to the method described in the above embodiments, for the sake of convenience of description, only the parts related to the embodiments of the present application are shown.

[0117] Figure 5 It is a schematic structural diagram of a multi-modal model training device provided by an embodiment of the present application. As Figure 5 shown, the device includes:

[0118] An acquisition module 10, configured to acquire training data;

[0119] A training module 11, configured to train each decomposition matrix in the multi-modal model by using the training data until the model loss value of the multi-modal model is less than a preset threshold.

[0120] Wherein, the parameter matrix of the multi-modal model includes a base matrix and an increment matrix of each layer, the decomposition matrix is obtained by decomposing the increment matrix, one of the decomposition matrices corresponding to the increment matrix is a matrix of the same order and the other decomposition matrices are matrices of different orders, and the total number of parameters of the decomposition matrix is less than the number of parameters of the increment matrix.

[0121] In one embodiment, the training module is specifically configured to input a training image into a vision large model in the multi-modal model to obtain an image classification result output by the vision large model. The training data includes the training image and the training text; the image classification result is input into an image-to-text adapter in the multi-modal model, so that the image-to-text adapter multiplies the image classification result by a dimensionality reduction matrix to obtain a dimensionality-reduced feature, and multiplies the dimensionality-reduced feature by a dimensionality-increasing matrix to obtain a text feature and then outputs the text feature; use the text feature and the training text to train each decomposition matrix, the dimensionality reduction matrix, and the dimensionality-increasing matrix until the model loss value of a preset loss function is less than a first preset threshold.

[0122] In one embodiment, the training module is further configured to use the image classification result and the training text to train the adapter until the model loss value of a preset loss function is less than a second preset threshold to obtain an image-to-text adapter.

[0123] In one embodiment, the training module is further configured to perform normalization and dropout processing on the dimensionality-reduced feature to obtain a first transformed feature; multiply the first transformed feature by the dimensionality-increasing matrix to obtain a dimensionality-increased feature; perform normalization and dropout processing on the dimensionality-increased feature to obtain a second transformed feature.

[0124] Figure 6 It is a schematic structural diagram of an electronic device provided in an embodiment of the present application. As Figure 6 shown, the electronic device 2 in this embodiment includes: at least one processor 20 ( Figure 6 only one is shown), a memory 21, and a computer program 22 stored in the memory 21 and executable on the at least one processor 20. When the processor 20 executes the computer program 22, the steps in any of the above method embodiments are implemented.

[0125] The electronic device 2 may be a computing device such as a desktop computer, a notebook, a palm computer, and a cloud server. The electronic device 2 may include, but is not limited to, a processor 20 and a memory 21. Those skilled in the art can understand,Figure 6 The electronic device 2 is only an example and does not constitute a limitation to the electronic device 2. It may include more or fewer components than those shown in the figure, or combine certain components, or different components. For example, it may also include input / output devices, network access devices, etc.

[0126] The so-called processor 20 may be a central processing unit (CPU). The processor 20 may also be other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc.

[0127] In some embodiments, the memory 21 may be an internal storage unit of the electronic device 2, such as the hard disk or memory of the electronic device 2. In some other embodiments, the memory 21 may also be an external storage device of the electronic device 2, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the electronic device 2. Further, the memory 21 may also include both the internal storage unit and the external storage device of the electronic device 2. The memory 21 is used to store an operating system, application programs, a boot loader, data, and other programs, such as the program code of the computer program, etc. The memory 21 may also be used to temporarily store the data that has been output or will be output.

[0128] It should be noted that the content such as the information interaction and execution process between the above-mentioned device / unit, due to being based on the same concept as the method embodiment of the present application, for its specific functions and the technical effects brought, please refer to the method embodiment part for details, and will not be elaborated here.

[0129] Those skilled in the art can clearly understand that, for the convenience and conciseness of description, only the division of the above functional units and modules is used as an example. In actual applications, the above functions can be allocated to different functional units and modules according to needs, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. Each functional unit and module in the embodiments can be integrated into a processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above integrated unit can be implemented in the form of hardware or in the form of a software functional unit. In addition, the specific names of the functional units and modules are only for the convenience of mutual distinction and do not limit the protection scope of this application. The specific working processes of the units and modules in the above system can refer to the corresponding processes in the foregoing method embodiments and will not be elaborated here.

[0130] An embodiment of this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the foregoing method embodiments can be implemented.

[0131] An embodiment of this application provides a computer program product. When the computer program product runs on an electronic device, the electronic device can be caused to execute the steps in the foregoing method embodiments.

[0132] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, to implement all or part of the processes in the above method embodiments of this application, a computer program can be used to instruct relevant hardware to complete. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by a processor, the steps in the foregoing method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in the form of source code, object code, executable file or some intermediate form, etc. The computer-readable medium can at least include: any entity or device that can carry the computer program code to the photographing device / terminal device, recording medium, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, RandomAccess Memory), electrical carrier signal, telecommunication signal, and software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disc, etc. In some cases, the computer-readable medium cannot be an electrical carrier signal and a telecommunication signal.

[0133] In the above embodiments, the descriptions of the respective embodiments have their own focuses. For parts not described or recorded in a certain embodiment, reference may be made to the relevant descriptions of other embodiments.

[0134] Those of ordinary skill in the art can realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. A professional technician can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of this application.

[0135] In the embodiments provided in this application, it should be understood that the disclosed device / network device and method can be implemented in other ways. For example, the device / network device embodiments described above are merely illustrative. For example, the division of the modules or units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed couplings or direct couplings or communication connections to each other can be through some interfaces. The indirect couplings or communication connections of the device or unit can be in electrical, mechanical or other forms.

[0136] The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they can be located in one place, or distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0137] The above-described embodiments are only used to illustrate the technical solutions of this application, rather than to limit them; although this application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements for some of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of each embodiment of this application, and should all be included within the protection scope of this application.

Claims

1. A multi-modal model training method, characterized in that, Including: Obtaining training data; Training each decomposition matrix in the multi-modal model using the training data until the model loss value of a preset loss function is less than a first preset threshold to obtain a trained multi-modal model; Wherein, the parameter matrix of the multi-modal model includes a basic matrix and an incremental matrix for each layer, the decomposition matrix is obtained by decomposing the incremental matrix, one of the decomposition matrices corresponding to the incremental matrix is a matrix of the same order and the other decomposition matrices are matrices of different orders, and the total number of parameters of the decomposition matrix is less than the number of parameters of the incremental matrix.

2. The method according to claim 1, wherein When the incremental matrix is a matrix of the same order, the incremental matrix includes a first decomposition matrix and a second decomposition matrix, the first decomposition matrix is a matrix of the same order, the second decomposition matrix is a matrix of different orders, and the number of parameters of the first decomposition matrix is less than the number of parameters of the second decomposition matrix.

3. The method according to claim 1, wherein When the incremental matrix is a matrix of different orders, the incremental matrix includes a third decomposition matrix and a fourth decomposition matrix, the third decomposition matrix is a matrix of the same order, the fourth decomposition matrix is a matrix of different orders, and the third decomposition matrix and the fourth decomposition matrix are sparse matrices; The third decomposition matrix includes a fifth decomposition matrix and a sixth decomposition matrix, the fifth decomposition matrix is a matrix of the same order, the sixth decomposition matrix is a matrix of different orders, the number of parameters of the fifth decomposition matrix is less than the number of parameters of the sixth decomposition matrix, and the total number of parameters of the fifth decomposition matrix and the sixth decomposition matrix is less than the third decomposition matrix.

4. The method according to claim 2 or 3, characterized in that, The matrix of the same order to be decomposed is an n×n matrix of the same order, and the n×n matrix of the same order is obtained by sequentially multiplying the transposed matrix of an r×n matrix of different orders, an r×r matrix of the same order, and an r×n matrix of different orders; wherein, r < n, the number of parameters of the r×r matrix of the same order is less than the number of parameters of the r×n matrix of different orders, and the total number of parameters of the r×r matrix of the same order and the r×n matrix of different orders is less than the n×n matrix of the same order.

5. The method according to claim 1, characterized in that, The training each decomposition matrix in the multi-modal model using the training data until the model loss value of a preset loss function is less than a first preset threshold includes: Inputting a training image into a visual large model in the multi-modal model to obtain an image classification result output by the visual large model, and the training data includes the training image and training text; Inputting the image classification result into an image-to-text adapter in the multi-modal model, so that the image-to-text adapter multiplies the image classification result by a dimensionality reduction matrix to obtain dimensionality-reduced features, and multiplies the dimensionality-reduced features by a dimensionality increase matrix to obtain text features and then outputs the text features; Training each decomposition matrix, the dimensionality reduction matrix, and the dimensionality increase matrix using the text features and the training text until the model loss value of a preset loss function is less than a first preset threshold.

6. The method according to claim 5, wherein Before inputting the image classification result into the image-to-text adapter in the multi-modal model, it further includes: Training the adapter using the image classification result and the training text until the model loss value of the preset loss function is less than a second preset threshold to obtain the image-to-text adapter.

7. The method according to claim 5 or 6, characterized in that, Further including: Normalize and perform dropout processing on the dimension-reduced features to obtain first transformed features; Multiply the first transformed features by the dimension-ascending matrix to obtain dimension-ascended features; Normalize and perform dropout processing on the dimension-ascended features to obtain second transformed features.

8. A multimodal model training device, characterized in that, It includes: An acquisition module for acquiring training data; A training module for training each decomposition matrix in the multi-modal model using the training data until the model loss value of the multi-modal model is less than a preset threshold to obtain a trained multi-modal model; Wherein, the parameter matrix of the multi-modal model includes the basic matrix and the incremental matrix of each layer, the decomposition matrix is obtained by decomposing the incremental matrix, one of the decomposition matrices corresponding to the incremental matrix is a matrix of the same order and the other decomposition matrices are matrices of different orders, and the total number of parameters of the decomposition matrix is less than the number of parameters of the incremental matrix.

9. An electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, the method according to any one of claims 1 to 7 is implemented.

Images