Multi-modal large language model training method based on indication data set optimization

Abstract

The invention discloses a multi-modal large language model training method based on indication data set optimization. The method comprises the following steps: S1, initial training: training a multi-modal large language model by using an initial multi-modal data set; s2, updating the indication data set; after the multi-modal data set is changed, inputting the multi-modal data set into a multi-modal dynamic diffusion network to extract dynamic characteristics, and then dynamically screening and adjusting an indication data set according to the importance and the change condition of the dynamic characteristics; s3, updating a feature fusion mechanism: extracting static features from the multi-modal data set by using a traditional multi-modal language feature model, and fusing the dynamic features and the static features to obtain fused features; s4, joint training: performing joint training on the multi-modal large language model according to the indication data set and the fusion features; and repeating the steps S2-S4 until the training is completed. According to the scheme, the trained model can have better generalization ability and performance.

Description

Technical Field

[0001] The present invention relates to the field of deep learning and multimodal data processing, and in particular to a multimodal large language model training method based on indicator data set optimization. Background Art

[0002] When processing multimodal data, traditional models often have difficulty fully capturing the complex relationships between different data modalities, resulting in degraded model performance. Existing diffusion models have limitations in processing dynamic data features and multimodal information, and cannot achieve real-time adaptation and optimization. Summary of the invention

[0003] The present invention mainly solves the technical problem that the prior art cannot adapt in real time when the data set used for training undergoes dynamic changes, and provides a multimodal large language model training method based on optimization of an indicator data set.

[0004] The present invention aims to solve the above technical problems mainly through the following technical solutions: a multimodal large language model training method based on indicator data set optimization, comprising the following steps:

[0005] S1. Initial training: Use the initial multimodal dataset (i.e., training dataset) to train the multimodal large language model;

[0006] S2, update the indicator data set; after the multimodal data set changes, the multimodal data set is input into the multimodal dynamic diffusion network (MM-DDN) to extract dynamic features, and then the indicator data set is dynamically screened and adjusted according to the importance and change of the dynamic features; the initial indicator data set is an empty set;

[0007] S3. Update feature fusion mechanism: Use the traditional multimodal language feature model to extract static features from the multimodal data set, fuse the dynamic features with the static features, and obtain fused features;

[0008] S4, joint training: Jointly train the multimodal large language model based on the indicator data set and the fusion features to optimize the performance of the multimodal large language model;

[0009] Repeat steps S2-S4 until the training is completed.

[0010] Preferably, in step S2, the multimodal dynamic diffusion network includes a dynamic diffusion layer and an interaction layer, and the dynamic diffusion layer has an adaptive dynamic weight mechanism, and its calculation process is:

[0011] H (t) =σ(W (t) ·H (t-1) +b (t) )

[0012] Where H(t) represents the hidden state at time step t, W(t) and b(t) represent the weight and bias at step t respectively, σ represents the activation function, and this scheme uses the Sigmoid function; the hidden state refers to the state used to store the information learned by the network in the neural network. At each time step t, the hidden state H (t) It will be updated according to the changes of input data and network parameters, and is used to transmit information and learn the relationship between data;

[0013] The interaction layer includes the attention mechanism, and the attention weight calculation formula is:

[0014]

[0015] Among them, e i represents the correlation between feature i and other features, α i represents the attention weight of feature i.

[0016] The output of the dynamic diffusion layer contains feature representations adjusted by dynamic weights, which can reflect the relationship between data features at different time steps. The output of the dynamic diffusion layer is used as the input of the interaction layer, so that the interaction layer can further strengthen the interaction between features based on the dynamic weights, thereby better representing the complex relationship between multimodal data.

[0017] The dynamic diffusion layer has an adaptive dynamic weight mechanism, which means that the model can adjust the intensity of information transmission in real time according to the changes in data features. This dynamicity enables the model to better capture the spatiotemporal correlation between multimodal data, thereby improving the performance and generalization ability of the model. On the other hand, the interaction layer introduces an attention mechanism, which helps the model to focus more on important feature interactions, thereby further improving the model's representation ability and learning efficiency. These two layers work together to enable the model to better process multimodal data. The models in this solution refer to the trained multimodal large language models.

[0018] e i It represents the correlation between feature i and other features, which is obtained by calculating the similarity or correlation between features. The similarity between general features can be calculated using dot-product attention.

[0019] The correlation calculation formula of dot product attention is as follows:

[0020] e ij =q i ·k j

[0021] Among them, e ij represents the correlation between feature i and feature j, qi and k j Denote the query vector and key vector of feature i and feature j respectively. By calculating the dot product of the query vector and the key vector, the correlation score between the features can be obtained.

[0022] Preferably, step S2 is specifically as follows:

[0023] S201, data feature extraction: For the input multimodal data set X, first use the multimodal dynamic diffusion network to extract data features. The extracted features are expressed as F = {f1, f2, ..., f n}, where f i Represents the feature vector of the i-th data sample;

[0024] S202, for each data feature f i , use the attention mechanism to calculate its importance score S i ;

[0025] S203, if the importance score of a data feature is greater than the screening threshold θ, then add this data feature to the indicated data set, otherwise do not add it; if this data feature is already in the indicated data set, then delete it from the indicated data set. The dynamic data screening process can be expressed as:

[0026] Indicator data set = {f i |S i >θ}

[0027] The key idea of ​​the dynamic indicator dataset construction algorithm is to update the indicator dataset in real time according to the dynamic changes of the model output results and data features. The dynamic indicator dataset construction algorithm is designed to match the characteristics of the MM-DDN model. The algorithm combines the output results of the model with the dynamic changes of data features to achieve real-time update and adjustment of the indicator dataset. By dynamically screening and adjusting the indicator dataset, the model can better adapt to data changes, thereby improving the robustness and generalization ability of the model. This real-time update mechanism enables the model to maintain efficient performance in the face of constantly changing data.

[0028] An indicator dataset is a special training set that indicates which data features the model should focus on during training. In the dynamic indicator dataset construction algorithm, the indicator dataset is dynamically updated based on the importance and changes of data features to help the model better adapt to data changes. The construction process of the indicator dataset involves dynamic importance evaluation and dynamic data screening, which aims to ensure that the model can focus on important features during training and make real-time adjustments as data features change.

[0029] Preferably, the importance score calculation formula of the data feature is:

[0030] S i =Softmax(W·f i )

[0031] Where W is the learned weight parameter and the Softmax function is used to normalize the score. i Indicates the importance of the feature in the current task.

[0032] The parameter W is learned through the back propagation algorithm during the model training process. During the training process, the optimization algorithm adjusts the value of parameter W according to the model's loss function and training data, so that the model can better fit the data and learn the importance of the features. Through the back propagation algorithm, the model can update the value of parameter W so that the model can more accurately calculate the importance score of the feature in the dynamic importance evaluation.

[0033] Preferably, the step S3 is specifically as follows:

[0034] F dynamic =MM-DDN(X)

[0035] F static =Traditional MM Language Model(X)

[0036] F fusion =α·F dynamic +(1-α)·F static

[0037] Among them, X represents the input multimodal dataset, F dynamic represents dynamic features, that is, the data features extracted from the multimodal dynamic diffusion network, F static represents static features, that is, the data features extracted by the traditional multimodal language model, F fusion represents the fused features, and α represents the fusion weight.

[0038] The dynamic feature fusion mechanism used in this solution is used to combine the dynamic features extracted by MM-DDN with the traditional multimodal language model. This fusion mechanism can adjust the weights and methods of feature fusion according to the dynamic changes of the data, thereby improving the model's understanding and utilization efficiency of multimodal information. Through the fusion and updating of dynamic features, the model can better adapt to the complex relationship between different data modalities, further improving the performance and generalization ability of the model. The fused features contain the combined information of dynamic features and static features, which can better represent the complex relationship of multimodal data and can be used as the input of the model for training the model or inferring downstream tasks.

[0039] The substantial effect brought about by the present invention is that it combines the dynamic indication data set construction algorithm and the dynamic feature fusion mechanism, and gradually introduces new dynamic indication data sets and dynamic feature fusion mechanisms during the training process, so as to realize the real-time adaptation and optimization of the model to the dynamic changes of data; by dynamically adjusting the training strategy, the model can better adapt to the requirements of complex multimodal tasks, improve performance and reduce the risk of overfitting; this joint training strategy enables the model to conduct more in-depth learning and optimization of the characteristics of the data at different stages, thereby achieving better model generalization ability and performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 It is a flow chart of a multimodal large language model training method based on indicator data set optimization of the present invention. DETAILED DESCRIPTION

[0041] The technical solution of the present invention is further specifically described below through embodiments and in conjunction with the accompanying drawings.

[0042] Embodiment: A multimodal large language model training method based on indicator data set optimization, such as Figure 1 As shown, the following steps are included:

[0043] S1. Initial training: Use the initial multimodal dataset (i.e., training dataset) to train the multimodal large language model;

[0044] S2, update the indicator data set; after the multimodal data set changes, the multimodal data set is input into the multimodal dynamic diffusion network (MM-DDN) to extract dynamic features, and then the indicator data set is dynamically screened and adjusted according to the importance and change of the dynamic features; the initial indicator data set is an empty set;

[0045] S3. Update feature fusion mechanism: Use the traditional multimodal language feature model to extract static features from the multimodal data set, fuse the dynamic features with the static features, and obtain fused features;

[0046] S4, joint training: Jointly train the multimodal large language model based on the indicator data set and the fusion features to optimize the performance of the multimodal large language model;

[0047] Repeat steps S2-S4 until the training is completed.

[0048] In step S2, the multimodal dynamic diffusion network includes a dynamic diffusion layer and an interaction layer. The dynamic diffusion layer has an adaptive dynamic weight mechanism, and its operation process is:

[0049] H (t) =σ(W (t) ·H (t-1) +b (t) )

[0050] Where H(t) represents the hidden state at time step t, W(t) and b(t) represent the weight and bias at step t respectively, σ represents the activation function, and this scheme uses the Sigmoid function; the hidden state refers to the state used to store the information learned by the network in the neural network. At each time step t, the hidden state H (t) It will be updated according to the changes of input data and network parameters, and is used to transmit information and learn the relationship between data;

[0051] The interaction layer includes the attention mechanism, and the attention weight calculation formula is:

[0052]

[0053] Among them, e i represents the correlation between feature i and other features, α i represents the attention weight of feature i.

[0054] The output of the dynamic diffusion layer contains feature representations adjusted by dynamic weights, which can reflect the relationship between data features at different time steps. The output of the dynamic diffusion layer is used as the input of the interaction layer, so that the interaction layer can further strengthen the interaction between features based on the dynamic weights, thereby better representing the complex relationship between multimodal data.

[0055] The dynamic diffusion layer has an adaptive dynamic weight mechanism, which means that the model can adjust the intensity of information transmission in real time according to the changes in data features. This dynamicity enables the model to better capture the spatiotemporal correlation between multimodal data, thereby improving the performance and generalization ability of the model. On the other hand, the interaction layer introduces an attention mechanism, which helps the model focus more on important feature interactions, thereby further improving the model's representation ability and learning efficiency. These two layers work together to enable the model to better handle multimodal data.

[0056] e i It represents the correlation between feature i and other features, which is obtained by calculating the similarity or correlation between features. The similarity between general features can be calculated using dot-product attention.

[0057] The correlation calculation formula of dot product attention is as follows:

[0058] e ij =q i ·k j

[0059] Among them, e ij represents the correlation between feature i and feature j, q iand k j Denote the query vector and key vector of feature i and feature j respectively. By calculating the dot product of the query vector and the key vector, the correlation score between the features can be obtained.

[0060] Step S2 is specifically as follows:

[0061] S201, data feature extraction: For the input multimodal data set X, first use the multimodal dynamic diffusion network to extract data features. The extracted features are expressed as F = {f1, f2, ..., f n}, where f i Represents the feature vector of the i-th data sample;

[0062] S202, for each data feature f i , use the attention mechanism to calculate its importance score S i ;

[0063] S203, if the importance score of a data feature is greater than the screening threshold θ, then add this data feature to the indicated data set, otherwise do not add it; if this data feature is already in the indicated data set, then delete it from the indicated data set. The dynamic data screening process can be expressed as:

[0064] Indicator data set = {f i |S i >θ}

[0065] The key idea of ​​the dynamic indicator dataset construction algorithm is to update the indicator dataset in real time according to the dynamic changes of the model output results and data features. The dynamic indicator dataset construction algorithm is designed to match the characteristics of the MM-DDN model. The algorithm combines the output results of the model with the dynamic changes of data features to achieve real-time update and adjustment of the indicator dataset. By dynamically screening and adjusting the indicator dataset, the model can better adapt to data changes, thereby improving the robustness and generalization ability of the model. This real-time update mechanism enables the model to maintain efficient performance in the face of constantly changing data.

[0066] An indicator dataset is a special training set that indicates which data features the model should focus on during training. In the dynamic indicator dataset construction algorithm, the indicator dataset is dynamically updated based on the importance and changes of data features to help the model better adapt to data changes. The construction process of the indicator dataset involves dynamic importance evaluation and dynamic data screening, which aims to ensure that the model can focus on important features during training and make real-time adjustments as data features change.

[0067] The importance score calculation formula of data features is:

[0068] Si =Softmax(W·f i )

[0069] Where W is the learned weight parameter and the Softmax function is used to normalize the score. i Indicates the importance of the feature in the current task.

[0070] The parameter W is learned through the back propagation algorithm during the model training process. During the training process, the optimization algorithm adjusts the value of parameter W according to the model's loss function and training data, so that the model can better fit the data and learn the importance of the features. Through the back propagation algorithm, the model can update the value of parameter W so that the model can more accurately calculate the importance score of the feature in the dynamic importance evaluation.

[0071] Step S3 is specifically as follows:

[0072] F dynamic =MM-DDN(X)

[0073] F static =Traditional MM Language Model(X)

[0074] F fusion =α·F dynamic +(1-α)·F static

[0075] Among them, X represents the input multimodal dataset, F dynamic represents dynamic features, that is, the data features extracted from the multimodal dynamic diffusion network, F static represents static features, that is, the data features extracted by the traditional multimodal language model, F fusion represents the fused features, and α represents the fusion weight.

[0076] The dynamic feature fusion mechanism used in this solution is used to combine the dynamic features extracted by MM-DDN with the traditional multimodal language model. This fusion mechanism can adjust the weights and methods of feature fusion according to the dynamic changes of the data, thereby improving the model's understanding and utilization efficiency of multimodal information. Through the fusion and updating of dynamic features, the model can better adapt to the complex relationship between different data modalities, further improving the performance and generalization ability of the model. The fused features contain the combined information of dynamic features and static features, which can better represent the complex relationship of multimodal data and can be used as the input of the model for training the model or inferring downstream tasks.

[0077] The input data X here may include multiple parts, which are used in the feature fusion process of the model. Specifically, it includes the following:

[0078] 1. The hidden state H(t) of the previous time step: In the model, the hidden state H(t) of the previous time step may contain the state information of the model in the previous time step, which can help the model capture the dynamic changes in the time series. This hidden state can be used as part of the input data for the feature fusion process in the dynamic feature fusion mechanism.

[0079] 2. The correlation between feature i and other features: In the dynamic feature fusion mechanism, the correlation between feature i and other features may be considered. This correlation information can help the model better understand the relationship between different features, thereby more effectively fusing dynamic and static features.

[0080] 3. Other feature information: information about each piece of data in the training set, original features of multimodal data, and static features extracted by traditional multimodal language models.

[0081] The specific embodiments described herein are merely examples of the spirit of the present invention. Those skilled in the art may make various modifications or additions to the specific embodiments described or replace them in similar ways, but they will not deviate from the spirit of the present invention or exceed the scope defined by the appended claims.

[0082] Although the terms such as multimodal data set, indicator data set, dynamic feature, etc. are used more frequently in this article, the possibility of using other terms is not excluded. These terms are used only to more conveniently describe and explain the essence of the present invention; interpreting them as any additional restrictions is contrary to the spirit of the present invention.

Claims

1. A multimodal large language model training method based on indicator data set optimization, characterized in that: The following steps are involved: S1, initial training: use the initial multimodal dataset to train the multimodal large language model; S2, updating the indication data set; after the multimodal data set changes, the multimodal data set is input into the multimodal dynamic diffusion network to extract dynamic features, and then the indication data set is dynamically screened and adjusted according to the importance and changes of the dynamic features; S3. Update feature fusion mechanism: Use the traditional multimodal language feature model to extract static features from the multimodal data set, fuse the dynamic features with the static features, and obtain fused features; S4, joint training: Jointly train the multimodal large language model based on the indicator data set and the fusion features to optimize the performance of the multimodal large language model; Repeat steps S2-S4 until the training is completed.

2. According to claim 1, a multimodal large language model training method based on indicator data set optimization is characterized in that: In step S2, the multimodal dynamic diffusion network includes a dynamic diffusion layer and an interaction layer, and the operation process of the dynamic diffusion layer is: H (t) =σ(W (t) ·H (t-1) +b (t) ) Where H(t) represents the hidden state at time step t, W(t) and b(t) represent the weight and bias at step t, respectively, and σ represents the activation function; The interaction layer includes the attention mechanism, and the attention weight calculation formula is: Among them, e i represents the correlation between feature i and other features, α i represents the attention weight of feature i.

3. The multimodal large language model training method based on indicator data set optimization according to claim 1 or 2, characterized in that: Step S2 is specifically as follows: S201, data feature extraction: For the input multimodal data set X, first use the multimodal dynamic diffusion network to extract data features. The extracted features are expressed as F = {f1, f2, ..., f n }, where f i Represents the feature vector of the i-th data sample; S202, for each data feature f i , use the attention mechanism to calculate its importance score S i ; S203. If the importance score of a data feature is greater than the screening threshold θ, then this data feature is added to the indication data set, otherwise it is not added; if this data feature is already in the indication data set, then it is deleted from the indication data set.

4. The multimodal large language model training method based on indicator data set optimization according to claim 3, characterized in that: The importance score calculation formula of the data feature is: S i =Softmax(W·f i ) Among them, W is the learned weight parameter, and the Softmax function is used to normalize the score.

5. The multimodal large language model training method based on indicator data set optimization according to claim 1 or 4, characterized in that: The step S3 is specifically as follows: F dynamic =MM-DDN(X) FOR static NTraditional MM Language Model(X) F fusion =α·F dynamic +(1-α)·F static Among them, X represents the input multimodal dataset, F dynamic represents dynamic features, that is, the data features extracted from the multimodal dynamic diffusion network, F static represents static features, that is, the data features extracted by the traditional multimodal language model, F fusion represents the fused features, and α represents the fusion weight.

Images