Model training method and device, storage medium and program product

Abstract

The embodiment of the invention provides a model training method and device, a storage medium and a program product, and is applied to the technical field of data analysis. The method comprises the steps that a plurality of training samples are obtained, each training sample comprises sample time sequence data and a corresponding sample text, and the sample text comprises sample description cues and sample interpretive answers for the sample time sequence data; performing model training according to the plurality of training samples to obtain a comparative learning model; constructing a first initial model based on the query generator, the LLM and a time sequence encoder and a text encoder included in the contrast learning model; and training the first initial model according to the plurality of training samples to obtain a multi-modal time sequence large model. The method is used for achieving the effect of improving the time sequence data analysis efficiency.

Description

Technical Field

[0001] The present application relates to the field of data analysis technology, and in particular to a model training method, device, storage medium and program product. Background Art

[0002] With the development of information technology and data science, time series analysis has gradually occupied an important position in various fields. Time series data is a kind of data that is ubiquitous in natural and social phenomena. It can reflect the dynamic characteristics of changes over time. Therefore, studying and effectively predicting time series data can not only help decision makers make more accurate strategies, but also improve the efficiency of enterprise operations and provide effective reference for field fluctuations.

[0003] In the prior art, the main methods for predicting time series data are: using a suitable time series model for modeling and prediction, determining the prediction model required for model training, predicting the time series data based on the prediction model, and generating corresponding prediction results; in order to improve the usability of the prediction results, a natural language processing model or a feature analysis model is used to convert the prediction results into an explanatory text that can be directly understood, thereby achieving complete time series data analysis.

[0004] Therefore, there is a technical problem in the prior art that the efficiency of time series data analysis is low. Summary of the invention

[0005] The embodiments of the present application provide a model training method, device, storage medium and program product to achieve the technical effect of improving the efficiency of time series data analysis.

[0006] In a first aspect, an embodiment of the present application provides a model training method, comprising:

[0007] Acquire multiple training samples, each training sample includes sample time series data and corresponding sample text, and the sample text includes sample description prompt words and sample explanatory answers for the sample time series data;

[0008] Perform model training based on multiple training samples to obtain a comparative learning model;

[0009] Constructing a first initial model based on the query generator, the LLM, and the temporal encoder and the text encoder included in the contrastive learning model;

[0010] The first initial model is trained according to multiple training samples to obtain a multimodal time series large model.

[0011] In a possible implementation, performing model training according to a plurality of training samples to obtain a comparative learning model includes:

[0012] Step a: Encode the sample time series data of each training sample through the initial time series encoder of the second initial model to generate a first sample time series vector;

[0013] Step b: encoding the sample text of each training sample through the initial text encoder of the second initial model to generate a first sample text vector;

[0014] Step c: for each training sample, calculating the similarity between the first sample time series vector and the first sample text vector corresponding to the training sample;

[0015] Step d: Calculate the cross entropy loss according to the similarity corresponding to each training sample;

[0016] Step e: updating the model parameters in the second initial model according to the cross entropy loss;

[0017] Repeat steps a to e to iteratively train the second initial model until a training cutoff condition is met to obtain a comparative learning model.

[0018] In a possible implementation, the first initial model is trained according to a plurality of training samples to obtain a multimodal time series large model, including:

[0019] Input multiple training samples into the contrastive learning model, and obtain the similarity corresponding to each training sample output by the contrastive learning model;

[0020] Determine the training samples with similarity higher than the preset similarity as target training samples;

[0021] Based on the target training samples, the first initial model is trained to obtain a large multimodal time series model.

[0022] In a possible implementation, the first initial model is trained based on the target training sample to obtain a multimodal time series large model, including:

[0023] Step f, performing mask processing on the sample explanatory answers in the sample text of the target training sample to obtain the processed sample text;

[0024] Step g: encoding the sample time series data of the target training sample through a time series encoder to generate a second sample time series vector;

[0025] Step h: encoding the processed sample text of the target training sample through a text encoder to generate a second sample text vector;

[0026] Step i: Processing the second sample time series vector by a query generator to generate a third sample time series vector;

[0027] Step g, concatenating the third sample time series vector and the second sample text vector to generate a fusion vector;

[0028] Step k: Process the fusion vector through LLM to generate a predictive explanatory answer;

[0029] Step 1: updating the model parameters in the first initial model according to the predicted explanatory answer and the sample explanatory answer of the target training sample;

[0030] Repeat steps f to l to iteratively train the first initial model until the training cutoff condition is met to obtain a large multimodal time series model.

[0031] In a possible implementation, updating the model parameters in the first initial model includes:

[0032] updating model parameters of the query generator in the first initial model;

[0033] or,

[0034] Update all model parameters in the first initial model.

[0035] In a possible implementation, the training cutoff condition is that the number of iterations reaches a preset number of iterations.

[0036] In a second aspect, an embodiment of the present application provides a model training device, comprising:

[0037] A first acquisition module is used to acquire multiple training samples, each training sample includes sample time series data and corresponding sample text, and the sample text includes sample description prompt words and sample explanatory answers for the sample time series data;

[0038] A second acquisition module is used to perform model training according to multiple training samples to obtain a comparative learning model;

[0039] A first processing module, configured to construct a first initial model based on the query generator, the LLM, and the temporal encoder and the text encoder included in the contrastive learning model;

[0040] The second processing module is used to train the first initial model according to multiple training samples to obtain a large multimodal time series model.

[0041] In a possible implementation manner, the second acquisition module is further used for:

[0042] Step a: Encode the sample time series data of each training sample through the initial time series encoder of the second initial model to generate a first sample time series vector;

[0043] Step b: encoding the sample text of each training sample through the initial text encoder of the second initial model to generate a first sample text vector;

[0044] Step c: for each training sample, calculating the similarity between the first sample time series vector and the first sample text vector corresponding to the training sample;

[0045] Step d: Calculate the cross entropy loss according to the similarity corresponding to each training sample;

[0046] Step e: updating the model parameters in the second initial model according to the cross entropy loss;

[0047] Repeat steps a to e to iteratively train the second initial model until a training cutoff condition is met to obtain a comparative learning model.

[0048] In a possible implementation manner, the second processing module is further used for:

[0049] Input multiple training samples into the contrastive learning model, and obtain the similarity corresponding to each training sample output by the contrastive learning model;

[0050] Determine the training samples with similarity higher than the preset similarity as target training samples;

[0051] Based on the target training samples, the first initial model is trained to obtain a large multimodal time series model.

[0052] In a possible implementation manner, the second processing module is further used for:

[0053] Step f, performing mask processing on the sample explanatory answers in the sample text of the target training sample to obtain the processed sample text;

[0054] Step g: encoding the sample time series data of the target training sample through a time series encoder to generate a second sample time series vector;

[0055] Step h: encoding the processed sample text of the target training sample through a text encoder to generate a second sample text vector;

[0056] Step i: Processing the second sample time series vector by a query generator to generate a third sample time series vector;

[0057] Step g, concatenating the third sample time series vector and the second sample text vector to generate a fusion vector;

[0058] Step k: Process the fusion vector through LLM to generate a predictive explanatory answer;

[0059] Step 1: updating the model parameters in the first initial model according to the predicted explanatory answer and the sample explanatory answer of the target training sample;

[0060] Repeat steps f to l to iteratively train the first initial model until the training cutoff condition is met to obtain a large multimodal time series model.

[0061] In a possible implementation manner, the second processing module is further used for:

[0062] updating model parameters of the query generator in the first initial model;

[0063] or,

[0064] Update all model parameters in the first initial model.

[0065] In a possible implementation, the training cutoff condition is that the number of iterations reaches a preset number of iterations.

[0066] In a third aspect, an embodiment of the present application provides an electronic device, including: a memory, a processor;

[0067] Memory stores computer-executable instructions;

[0068] The processor executes the computer-executable instructions stored in the memory, so that the processor executes the above first aspect and / or various possible implementations of the first aspect.

[0069] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the first aspect above and / or various possible implementations of the first aspect.

[0070] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the above first aspect and / or various possible implementation methods of the first aspect.

[0071] The model training method, device, storage medium and program product provided by the embodiment of the present application, by obtaining multiple training samples, and performing model training based on multiple training samples, realizes the association modeling between the sample time series data and the sample text semantics, thereby obtaining a contrastive learning model; constructs a first initial model based on a query generator, LLM and a contrastive learning model; trains the first initial model using multiple training samples, thereby obtaining a multimodal time series large model. In the present application, the time series encoder and text encoder in the contrastive learning model are used to realize the alignment of the time series and text in the training sample, and the fusion vector is generated using the time series and text vectors of the modal alignment, thereby obtaining a fusion vector that can characterize the multimodal features of a training sample; the fused vector is predicted based on LLM, thereby obtaining a predictive explanatory text that can correspond to the time series data. Compared with the prior art, the present application finally generates an independent multimodal time series large model, which can directly generate corresponding predictive explanatory text based on the time series data input; there is no need to further interpret the prediction results generated after the time series data analysis; thereby achieving the technical effect of improving the efficiency of time series data analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0073] Figure 1 A flowchart of the model training method provided in this application;

[0074] Figure 2 A data transmission diagram of a comparative learning model provided in an embodiment of the present application;

[0075] Figure 3 A flowchart of a training method for a comparative learning model provided in an embodiment of the present application;

[0076] Figure 4 A flowchart of a method for obtaining target training samples based on a contrastive learning model provided in an embodiment of the present application;

[0077] Figure 5 A schematic diagram of data transmission of a multimodal large model based on the Q-Former architecture provided in an embodiment of the present application;

[0078] Figure 6 A schematic diagram of a multimodal large model training method based on a Q-Former architecture provided in an embodiment of the present application;

[0079] Figure 7 A schematic diagram of the structure of the model training device provided in this application;

[0080] Figure 8 A schematic diagram of the structure of the electronic device provided in this application.

[0081] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0082] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0083] First, the terms involved in this application are explained:

[0084] Large Language Model (LLM): is a natural language processing model based on deep learning technology, usually composed of multi-layer neural networks and has a large number of parameters. The core idea of ​​this type of model is to use large-scale text data for pre-training, so that the model can learn rich language knowledge and semantic representation. Subsequently, fine-tuning on specific tasks can make the model adapt to different natural language processing tasks, such as text generation, sentiment analysis, question-answering systems, etc.

[0085] Contrastive Language-Image Pretraining (CLIP): refers to a multimodal pre-training model that aims to jointly train images and texts through contrastive learning methods.

[0086] Q-Former: is an architecture for multimodal tasks, usually used to fuse and process information from different modalities. Although the specific implementation may vary depending on the application scenario, the core idea of ​​Q-Former is to effectively extract relevant information from the input data through a query mechanism.

[0087] LoRa (Low Rank Adaptation): is a parameter optimization method, usually used for model fine-tuning or parameter update in the process of transfer learning. The LoRa algorithm balances the trade-off between low-rank and high-rank parameter updates by maintaining the stability of the gradient direction to achieve more efficient network fine-tuning.

[0088] Low-rank matrix: refers to the low-rank representation obtained by low-rank decomposition or approximation of model parameters, in order to reduce the number of parameters, improve generalization ability and accelerate calculation.

[0089] In the prior art, the main methods for performing data analysis on time series data are to perform predictive analysis on the time series data through modeling, determine a large time series model that can be used for time series analysis; generate predictive results corresponding to the time series data based on the large time series model; in order to improve the availability of the predictive results, perform human-assisted interpretation of the predictive results, or use classification processing and language processing to generate explanatory text corresponding to the predictive results.

[0090] However, in actual time series data analysis, if you want to generate explanatory text corresponding to time series data, you first need to perform data analysis and processing on the time series data, and use machine learning and deep learning to predict the time series data. The second is to match the explanatory text for the prediction results of the time series data, and use natural language processing or database matching to generate the explanatory text corresponding to each prediction result. For example: if you need to predict the changes in meteorological data within the next week based on the currently provided meteorological data, you first need to train a large time series model that can be used for meteorological data prediction. The trained large time series model can be used to predict meteorological data within the next week and obtain a prediction result of meteorological data; the result is only a piece of data or feature value; in order to improve usability, it is necessary to generate explanatory text for the result. As a result, the data analysis process for meteorological data is lengthened. Therefore, there is a technical problem of low efficiency in time series data analysis in the prior art.

[0091] In view of the above technical problems, the present application proposes the following technical concept: There is a technical problem of low efficiency in time series data analysis in the prior art. In practical applications, a large language model can form a good text generation effect, so the analysis of time series data can be combined with a large language model. Specifically: the processing mechanism of the contrastive learning model is combined with the training sample to perform model training, and a contrastive learning model that can achieve time series and text alignment is obtained. Based on the trained contrastive learning model combined with the large language model and the query generator, the first initial model is constructed, the aligned time series vector and text vector are generated by the contrastive learning model, and the time series vector is further processed by the query generator to obtain the processed time series vector, and the time series vector and the text vector are fused to obtain the fused multimodal vector; based on the large language model, the multimodal fusion vector is processed to generate a predictive explanatory answer, and finally the model training is stopped and the multimodal time series large model is determined when the iteration condition is met. The method of the present application performs model training by vector fusion, thereby obtaining a multimodal time series large model that can directly generate corresponding predictive explanatory answers based on time series data, and achieves the technical effect of improving the efficiency of time series data analysis.

[0092] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0093] Figure 1 A flow chart of the model training method provided in this application, such as Figure 1 As shown, the method includes:

[0094] S101. Acquire multiple training samples, each training sample including sample time series data and corresponding sample text.

[0095] In this step, the sample text includes sample description prompt words and sample explanatory answers for the sample time series data.

[0096] The sample description prompt words and sample explanatory answers refer to the explanations from any angle corresponding to the sample time series data.

[0097] Exemplarily, the source types of multiple training samples can be public time series-text pair datasets, public time series prediction or classification datasets, and private domain datasets in vertical fields. Among them, public time series-text pair datasets can be: open source datasets in the medical field; public time series prediction or classification datasets can be: datasets in the fields of energy, transportation, climate, cloud, Internet, transactions, nature, finance, and medical health; private domain datasets in vertical fields can be: manually collected "time series-text" mainly for industrial equipment fault diagnosis and predictive maintenance, the corresponding time series data is composed of vibration, current, and voltage sensor detection data, and the corresponding text is mainly industry experts' diagnosis reports or maintenance reports on equipment.

[0098] In this step, obtaining multiple training samples requires normalizing the sample time series data and sample text in the training samples to ensure that the composition structure of each training sample is: a section of sample time series data, a section of sample text consisting of sample description prompt words and sample explanatory answers.

[0099] Exemplarily, there is a segment of sample time series data, which is a waveform diagram of a vibration sensor in the x-axis direction of a bearing; in different description angles, the sample text corresponding to the sample segment of time series data is different. Here are four examples of different sample text descriptions for the segment of sample time series data in four angles. In the first item, the sample description prompt corresponding to the segment of sample time series data is: "This is a waveform diagram of a vibration sensor in the x-axis direction of a bearing, please perform fault diagnosis"; the corresponding sample explanatory answer is: "The waveform has an abnormal frequency band signal, which is a fatigue peeling failure caused by the bearing exceeding the fatigue limit." In the second item, the sample description prompt corresponding to the segment of sample time series data is: "This is a waveform diagram of a vibration sensor in the x-axis direction of a bearing, please predict the vibration value in the next seven steps"; the corresponding sample explanatory answer is: "50.233, 62.347, 70.742, 80.772, 40.189, 78.109, 33.383". In the third item, the sample description prompt corresponding to this section of sample time series data is: "This is a waveform of the vibration sensor of a bearing in the x-axis direction. Please predict the service life of the bearing"; the corresponding sample explanatory answer is: "The service life of the bearing is 3.7 million revolutions". In the fourth item, the sample description prompt corresponding to this section of sample time series data is: "This is a waveform of the vibration sensor of a bearing in the x-axis direction. Please give a predictive maintenance plan"; the corresponding sample explanatory answer is: "At the 3000th step, the lubricant will be exhausted and the temperature will be too high. The lubricant should be updated or added at the 2500th step".

[0100] It should be noted that the examples in this step and the text and numbers involved in the examples are only for reference and do not limit the specific numerical values ​​and text descriptions of the sample data used in actual applications.

[0101] S102: Perform model training according to multiple training samples to obtain a comparative learning model.

[0102] In this step, the contrastive learning model refers to a multimodal model that can link time series data and text together. The contrastive learning model includes a time series encoder and a text encoder. The input of the contrastive learning model is time series data input and text input, and the output is a time series vector and a text vector.

[0103] Optionally, a possible implementation method of obtaining the contrastive learning model in this step is:

[0104] S1021. Encode the sample time series data of each training sample through the initial time series encoder of the second initial model to generate a first sample time series vector.

[0105] In this step, the initial time series encoder is used to process and analyze the time series data and extract meaningful features and patterns from the time series data. The types of initial time series encoders include: recurrent neural networks, convolutional neural networks, transformers, and autoregressive models.

[0106] Among them, the recurrent neural network can process time series data of varying lengths and maintain the transmission and memory of information through a cyclic structure; the convolutional neural network can capture important features in time series data through convolution and pooling operations; the transformer is a deep learning model based on the self-attention mechanism, which processes time series data in a parallel manner and can effectively capture long-distance dependencies; the autoregressive model is a statistical model used for time series prediction.

[0107] S1022. Encode the sample text of each training sample through the initial text encoder of the second initial model to generate a first sample text vector.

[0108] In this step, the initial text encoder is used to process and analyze the sample text, convert the text data into numerical representation, and extract the feature information of the text. The initial text encoder includes: a bag-of-words model, a word embedding model, an encoder based on a recurrent neural network, a convolutional neural network, and a transformer.

[0109] Among them, the word embedding model can generate low-dimensional dense word vectors, and combine the word vectors to obtain text vectors that can represent the semantic features of the text; the bag-of-words model can generate text vectors based on the frequency of different words in the sample text and the binary encoding of the words; the encoder based on the recurrent neural network can generate text vectors with semantic features based on the context of the words in the text; the convolutional neural network can extract local features of the text and is suitable for processing short texts.

[0110] S1023. For each training sample, calculate the similarity between the first sample time series vector and the first sample text vector corresponding to the training sample.

[0111] In this step, the similarity calculation method between the first text vector and the first time series vector includes but is not limited to: calculating the cosine similarity between the two vectors, calculating the Euclidean distance between the two vectors, and calculating the Manhattan distance between the two vectors.

[0112] If the similarity between the first text vector and the first time series vector is close to 1, it indicates that the time series vector and the text vector are similar in the feature space; otherwise, it indicates that the two are not similar in the feature space. The calculation of similarity is used to align the first time series vector obtained by the time series encoder and the first text vector obtained by the text encoder in the feature space, which facilitates the subsequent modal fusion between the time series and text vectors.

[0113] S1024. Calculate the cross entropy loss according to the similarity corresponding to each training sample.

[0114] In this step, the cross entropy loss is used to measure the gap between the vector expression of sample time series data and sample text.

[0115] The method for calculating the cross entropy loss can be: obtain a fixed batch size of training samples, where there are N training samples in the batch, where N is a positive integer greater than 0. Calculate the first time series vector and the first text vector corresponding to the N training samples in the batch. Based on the similarities corresponding to the N training samples, construct a similarity matrix corresponding to the batch. Calculate the cross entropy loss based on the similarity matrix.

[0116] S1025. Update the model parameters in the second initial model according to the cross entropy loss.

[0117] In this step, the method of updating the model parameters in the second initial model according to the cross entropy loss is: based on the calculated cross entropy loss, the model parameters are updated by back propagation optimization, and the cross entropy loss is continuously optimized.

[0118] S1026. Repeat steps S1021 to S1025 to iteratively train the second initial model until a training cutoff condition is met, thereby obtaining a comparative learning model.

[0119] In this step, the training cutoff condition is that the number of iterative training reaches a preset number of iterations.

[0120] S103: construct a first initial model based on the query generator, the LLM, and the temporal encoder and the text encoder included in the contrastive learning model.

[0121] In this step, the query generator is used to map the input data into a high-dimensional space, generate a corresponding learnable query vector, and enrich another vector expressing the characteristics of the data based on the learnable query vector, so that the vector of the data is more characteristic.

[0122] In the first initial model, the time series encoder is used to encode the time series data in the input data; the text encoder is used to process the text in the input data; the query generator is used to further optimize the time series features processed by the time series encoder to obtain optimized time series features, and at the same time realize multimodal fusion between time series features and text features; LLM is used to process the fused multimodal features and generate corresponding text information.

[0123] S104: Train the first initial model according to multiple training samples to obtain a large multimodal time series model.

[0124] Optionally, a possible implementation method of obtaining a large multimodal time series model is:

[0125] S1041. Input multiple training samples into a contrastive learning model, and obtain the similarity corresponding to each training sample output by the contrastive learning model.

[0126] S1042: Determine training samples with a similarity higher than a preset similarity as target training samples.

[0127] In this step, the contrastive learning model is used to screen the training samples.

[0128] S1043. Mask the sample explanatory answer in the sample text of the target training sample to obtain the processed sample text.

[0129] In this step, the sample explanatory answers may be masked by using special tags to replace the masked text fields.

[0130] For example, there is a sample text: "This is the time series data of the oil temperature of the power transformer for the past 7*24 hours, sampled once per hour. Please predict the oil temperature in the next 7 hours: 30.53, 27.79, 27.79, 25.04, 21.95,21.17, 22.79." The explanatory bold part "0.53, 27.79, 27.79, 25.04, 21.95,21.17, 22.79" in the text is masked, and the processed sample text can be: "This is the time series data of the oil temperature of the power transformer for the past 7*24 hours, sampled once per hour. Please predict the oil temperature in the next 7 hours: [MASK], [MASK], [MASK], [MASK], [MASK], [MASK]."

[0131] S1044. Encode the sample time series data of the target training sample through a time series encoder to generate a second sample time series vector.

[0132] In this step, the temporal encoder refers to the temporal encoder in the trained contrastive learning model.

[0133] The input sample time series data type can be: text, video, audio.

[0134] S1045. Encode the processed sample text of the target training sample through a text encoder to generate a second sample text vector.

[0135] S1046: Process the second sample time series vector through the query generator to generate a third sample time series vector.

[0136] In this step, a possible implementation method of generating the third sample time series vector is:

[0137] A learnable query vector corresponding to the second sample time series vector is generated based on the query generator; and the second sample time series vector and the learnable query vector are fused to generate a third sample time series vector.

[0138] S205: Concatenate the third sample time series vector and the second sample text vector to generate a fusion vector.

[0139] In this step, the vector splicing method can be: direct splicing or weighted splicing.

[0140] S1047. Process the fused vector through LLM to generate a predictive explanatory answer.

[0141] S1048. Update the model parameters in the first initial model according to the predicted explanatory answer and the sample explanatory answer of the target training sample.

[0142] In this step, a possible implementation method of updating the model parameters in the first initial model is:

[0143] A loss value of the first initial model is calculated based on the predicted explanatory answer and the sample explanatory answer in the target training sample; and a model parameter in the first initial model is adjusted based on the loss value.

[0144] Optionally, the method of updating the model parameters in the first initial model may be:

[0145] Update the model parameters of the query generator in the first initial model; or update all the model parameters in the first initial model.

[0146] Among them, the purpose of updating the model parameters of the query generator is to focus on the learning of the parameters in the query generator during the model training process, while reducing the computational complexity of model training; updating all parameters refers to fine-tuning the overall model parameters so that the generation results of the first initial model are more accurate.

[0147] S1049. Repeat steps S1043 to S1048 to iteratively train the first initial model until the training cutoff condition is met, thereby obtaining a large multimodal time series model.

[0148] In this step, the training cutoff condition may be reaching a preset number of iterative training times or being lower than a preset loss value of the model. In this step, the input data of the obtained multimodal time series large model is the time series data and the description prompt words of the time series data, and the output data is text information containing the explanatory answers predicted by the time series data.

[0149] The model training method provided in the embodiment of the present application obtains multiple training samples, performs model training based on multiple training samples, realizes the association modeling between the sample time series data and the sample text semantics, and thus obtains the contrastive learning model; constructs the first initial model based on the query generator, LLM and the contrastive learning model; trains the first initial model using multiple training samples, thereby obtaining a multimodal time series large model. In the present application, the time series encoder and text encoder in the contrastive learning model are used to realize the alignment of the time series and text in the training sample, and the fusion vector is generated using the time series and text vectors of the modal alignment, thereby obtaining a fusion vector that can characterize the multimodal features of a training sample; the fused vector is predicted based on the large language model, thereby obtaining a predictive explanatory text that can correspond to the time series data. Compared with the prior art, the present application finally generates an independent multimodal time series large model, which can directly generate corresponding predictive explanatory text based on the input of time series data and time series data description; there is no need to further interpret the prediction results generated after the time series data analysis; thereby achieving the technical effect of improving the efficiency of time series data analysis.

[0150] On the basis of the above embodiments, the present application proposes a contrastive learning model training method based on the CLIP model. Figure 2 Schematic diagram of data transmission of the contrastive learning model provided in the embodiment of the present application. Figure 2 The contrastive learning model shown in FIG. 1 includes a time series encoder: Time Series Encoder and a text encoder: Text Encoder. The sample time series data in the input training sample is encoded by the time series encoder to obtain a time series vector TS Embeddings; the sample text is encoded by the text encoder to obtain a text vector Text Embeddings. The structure of the time series vector TS Embeddings is {S 1 , S 2 , S 3 , ..., S N}, the structure of the text vector Text Embeddings is {T 1 , T 2 , T 3 , ..., T N}. The similarity between the two vectors is calculated as Figure 2 As shown in the matrix, the similarity corresponding to each position is calculated, specifically {S 1 T 1 , S 2 T 2 , S 3 T 3 , ..., S N T N}. N represents the length of the time series vector and the text vector, and the length of the time series vector and the text vector are the same.

[0151] Figure 3 A flow chart of a training method for a contrastive learning model provided in an embodiment of the present application. Figure 3 As shown, the training method of the contrastive learning model includes:

[0152] A1. Initialize the model parameters of the second initial model corresponding to the contrastive learning model. Use the gradient optimization algorithm and set the model's learning rate, the number of iterations, and the batch size of the training samples.

[0153] Specifically:

[0154] A11. Use the AdamW gradient update algorithm with the following parameters: β 1 =0.9,β 2 =0.98,eps=1e−6.

[0155] In this step, AdamW is a gradient-based optimization algorithm that can dynamically adjust the learning rate of each parameter, effectively handle problems such as sparse gradients and non-stationary targets, and accelerate the convergence process of the model. 1 and β 2 There are two hyperparameters in the AdamW algorithm, which control the exponential decay rate of the first-order moment estimate and the second-order moment estimate respectively. eps represents a very small value used to adjust the numerical stability during the calculation process.

[0156] A12. Set the learning rate to 0.00002.

[0157] In this step, the learning rate is an important hyperparameter that controls the update amplitude of model parameters and affects the convergence speed and effect of the model.

[0158] A13. Set the maximum epoch number to 20 and the batch size to 70.

[0159] In this step, epoch refers to the process of forward propagating and backward propagating the entire training dataset in the neural network once. In one epoch, the model will traverse the entire training dataset and update the parameters. The maximum number of epochs in the training process is defined, that is, the model will be trained on the entire dataset 20 times. The entire dataset refers to all the training samples used for model training.

[0160] Batch refers to the way of dividing a large amount of training data into small blocks for training. In each epoch, the data set is divided into several batches, and the model updates the parameters by processing each batch instead of processing the entire data set separately.

[0161] A14. Set the first two epochs as the warm-up phase, and use the cosine decay learning rate for the next 18 epochs.

[0162] In this step, the first two epochs are called the warm-up phase, and a smaller learning rate is usually used to help the model stabilize faster and avoid instability during training. In the subsequent 18 epochs, the learning rate will gradually decrease in a cosine decay manner, so that the learning rate can be adjusted more finely in the later stage of training to help the model converge better and obtain better generalization ability.

[0163] A2. Load multiple training samples.

[0164] A3. Use the time series encoder in the second initial model to encode the sample time series data in the training sample to obtain a first sample time series vector; use the text encoder to encode the sample text in the training sample to obtain a second sample text vector.

[0165] A4. Perform similarity calculation on each training sample in a batch, and respectively calculate the cosine similarity of the first sample time series vector and the first sample text vector in each training sample; construct a similarity matrix corresponding to each batch. Based on the similarity matrix corresponding to each batch, calculate the cross entropy loss corresponding to the batch. Update the model parameters of the first initial model based on the cross entropy loss.

[0166] A5. Calculate whether the current number of iterations reaches the preset number of iterations.

[0167] A6. If the number of iterations reaches the preset number of iterations, the training is stopped and a comparative learning model is generated.

[0168] A7. If the number of iterations does not reach the preset number of iterations, repeat A3 to A5.

[0169] Based on the above embodiments, the embodiments of the present application also provide a method for screening training samples based on a contrastive learning model. Figure 4 A flow chart of a method for obtaining target training samples based on a contrastive learning model provided in an embodiment of the present application. Figure 4 As shown, the method includes:

[0170] B1. Load multiple training samples.

[0171] B2. Perform forward inference for each training sample based on the contrastive learning model.

[0172] B3. Calculate the similarity of each training sample.

[0173] B4. Determine whether the similarity is lower than a preset threshold.

[0174] B5. If the similarity is lower than the preset threshold, determine whether to delete directly.

[0175] B6. If you are sure not to delete, make manual adjustments and repeat step B1.

[0176] B7. If you are sure to delete, delete the training sample directly.

[0177] B8. If it is not lower than the preset threshold, the training sample is determined as the target training sample.

[0178] Based on the above embodiments, the embodiments of the present application also provide a training method for a multimodal large model based on the Q-Former architecture. Figure 5 Schematic diagram of data transmission of a multi-modal large model based on the Q-Former architecture provided in the embodiment of the present application. Figure 5As shown, the multimodal large model includes: a large language model: LLM, a time series encoder: Time SeriesEncoder; a text encoder: Text Encoder; a query generator: Q-Former. The sample time series data in the input target training sample is encoded by the time series encoder to obtain the time series vector TS Embeddings; the sample text is encoded by the text encoder to obtain the text vector Text Embeddings. The time series vector is input into the Q-Former, the learnable query vector corresponding to the time series vector is calculated, and the time series vector and the learnable query vector are fused to obtain a new time series vector. The new time series vector and the text vector are fused and input into the LLM to calculate the predictive explanatory answer for the training sample.

[0179] Figure 6 The flowchart of the multi-modal large model training method based on the Q-Former architecture provided in the embodiment of the present application is as follows. Figure 6 As shown, the method includes:

[0180] C1. Initialize model parameters, load the timing encoder, text encoder, randomly initialize Q-Former parameters, and load LLM.

[0181] Exemplarily, this step may be implemented as follows:

[0182] Using the AdamW gradient update algorithm, β 1 =0.9,β 2 =0.98,eps=1e−6, set the learning rate to 0.0002, and the maximum epoch number to 16. The first two epochs are warm-up phases, and the learning rate in the last 14 epochs uses the cosine decay method.

[0183] C2. Load the filtered target training data.

[0184] C3. Mask the explanatory answers of the sample text part in the target training data.

[0185] C4, fix the parameters corresponding to the temporal encoder, text encoder and LLM module, and only allow Q-Former to participate in gradient updates.

[0186] C5: Push the model forward based on gradient update, calculate the loss value between the predicted explanatory answer and the original explanatory answer corresponding to each target training sample, and update the parameters of Q-Former based on the loss value.

[0187] In this step, the loss value can be calculated as follows: assuming that the number of fields in the masked original explanatory answer is 5, the number of fields in the predicted explanatory answer is also 5; respectively calculate the loss values ​​between the fields with the same position in the original explanatory answer and the predicted explanatory answer, thereby obtaining 5 loss values, and calculate the sum based on these 5 loss values ​​to obtain the loss value between the original explanatory answer and the predicted explanatory answer.

[0188] C6. Repeat steps C2-C5 until the iteration stop condition is met.

[0189] D1. Repeat steps C1-C3 to determine whether to use the LoRa algorithm to adjust the parameters of the timing encoder, text encoder and LLM.

[0190] D2. If it is determined to use the LoRa algorithm, the parameters of the timing encoder, text encoder and LLM are set to the corresponding low-rank matrix, initialized, and the tensor parameters in the matrix are set to participate in the gradient calculation.

[0191] D3. If the LoRa algorithm is not used, all parameters of the timing encoder, text encoder and LLM are set to participate in the gradient calculation.

[0192] D4. Drive the model forward based on gradient update, calculate the loss value between the predicted explanatory answer and the original explanatory answer corresponding to each target training sample, and update the parameters of the temporal encoder, text encoder and LLM based on the loss value.

[0193] D5, perform iterative calculations in the same way as C6 until the iteration stop condition is met.

[0194] D6. After the iteration stop condition is met, the low-rank matrix and the parameters of the temporal encoder, text encoder and LLM can be merged.

[0195] Figure 7 A schematic diagram of the structure of the model training device provided in this application, such as Figure 7 As shown, the model training device provided in this embodiment includes:

[0196] A first acquisition module 701 is used to acquire multiple training samples, each of which includes sample time series data and corresponding sample text, and the sample text includes sample description prompt words and sample explanatory answers for the sample time series data;

[0197] A second acquisition module 702 is used to perform model training according to multiple training samples to obtain a comparative learning model;

[0198] A first processing module 703, configured to construct a first initial model based on the query generator, the LLM, and the temporal encoder and the text encoder included in the contrastive learning model;

[0199] The second processing module 704 is used to train the first initial model according to multiple training samples to obtain a multimodal time series large model.

[0200] In a possible implementation, the second obtaining module 702 is further configured to:

[0201] Step a: Encode the sample time series data of each training sample through the initial time series encoder of the second initial model to generate a first sample time series vector;

[0202] Step b: encoding the sample text of each training sample through the initial text encoder of the second initial model to generate a first sample text vector;

[0203] Step c: for each training sample, calculating the similarity between the first sample time series vector and the first sample text vector corresponding to the training sample;

[0204] Step d: Calculate the cross entropy loss according to the similarity corresponding to each training sample;

[0205] Step e: updating the model parameters in the second initial model according to the cross entropy loss;

[0206] Repeat steps a to e to iteratively train the second initial model until a training cutoff condition is met to obtain a comparative learning model.

[0207] In a possible implementation, the second processing module 704 is further configured to:

[0208] Input multiple training samples into the contrastive learning model, and obtain the similarity corresponding to each training sample output by the contrastive learning model;

[0209] Determine the training samples with similarity higher than the preset similarity as target training samples;

[0210] Based on the target training samples, the first initial model is trained to obtain a large multimodal time series model.

[0211] In a possible implementation, the second processing module 704 is further configured to:

[0212] Step f, performing mask processing on the sample explanatory answers in the sample text of the target training sample to obtain the processed sample text;

[0213] Step g: encoding the sample time series data of the target training sample through a time series encoder to generate a second sample time series vector;

[0214] Step h: encoding the processed sample text of the target training sample through a text encoder to generate a second sample text vector;

[0215] Step i: Processing the second sample time series vector by a query generator to generate a third sample time series vector;

[0216] Step g, concatenating the third sample time series vector and the second sample text vector to generate a fusion vector;

[0217] Step k: Process the fusion vector through LLM to generate a predictive explanatory answer;

[0218] Step 1: updating the model parameters in the first initial model according to the predicted explanatory answer and the sample explanatory answer of the target training sample;

[0219] Repeat steps f to l to iteratively train the first initial model until the training cutoff condition is met to obtain a large multimodal time series model.

[0220] In a possible implementation, the second processing module 704 is further configured to:

[0221] updating model parameters of the query generator in the first initial model;

[0222] or,

[0223] Update all model parameters in the first initial model.

[0224] In a possible implementation, the training cutoff condition is that the number of iterations reaches a preset number of iterations.

[0225] The model training device provided in this embodiment can execute the method provided in the above method embodiment. Its implementation principle and technical effects are similar, and will not be described in detail in this embodiment.

[0226] Figure 8 This is a schematic diagram of the structure of the electronic device provided in this application. Figure 8 As shown, the electronic device provided in this embodiment includes: at least one processor 801 and a memory 802. Optionally, the device further includes a communication component 803. The processor 801, the memory 802 and the communication component 803 are connected via a bus 804.

[0227] In a specific implementation process, at least one processor 801 executes the computer-executable instructions stored in the memory 802, so that at least one processor 801 executes the above method.

[0228] The specific implementation process of the processor 801 can be found in the above method embodiment, and its implementation principle and technical effect are similar, so this embodiment will not be repeated here.

[0229] In the above embodiments, it should be understood that the processor can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. A general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the invention can be directly implemented as a hardware processor, or can be implemented by a combination of hardware and software modules in the processor.

[0230] The memory may include a high-speed memory (Random Access Memory, RAM), and may also include a non-volatile memory (NVM), such as at least one disk storage.

[0231] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus in the drawings of this application is not limited to only one bus or one type of bus.

[0232] The present application also provides a computer program product, including a computer program, which implements the above method when executed by a processor.

[0233] The present application also provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the above method is implemented.

[0234] The above-mentioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The readable storage medium can be any available medium that can be accessed by a general or special-purpose computer.

[0235] An exemplary readable storage medium is coupled to a processor so that the processor can read information from the readable storage medium and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can be located in an application specific integrated circuit (Application Specific Integrated Circuits, referred to as: ASIC). Of course, the processor and the readable storage medium can also exist in the device as discrete components.

[0236] The division of units is only a logical function division, and there may be other divisions in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interface, device or unit, which can be electrical, mechanical or other forms.

[0237] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0238] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0239] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0240] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk and other media that can store program codes.

[0241] Finally, it should be noted that those skilled in the art will readily conceive of other embodiments of the present invention after considering the specification and practicing the invention disclosed herein. The present invention is intended to cover any variations, uses or adaptations of the present invention, which follow the general principles of the present invention and include common knowledge or customary technical means in the art not disclosed by the present invention, are not limited to the precise structure described above and shown in the drawings, and may be modified and changed in various ways without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.

Claims

1. A model training method, characterized in that: include: Acquire multiple training samples, each training sample includes sample time series data and corresponding sample text, and the sample text includes sample description prompt words and sample explanatory answers for the sample time series data; Performing model training according to the multiple training samples to obtain a comparative learning model; Constructing a first initial model based on a query generator, a large language model LLM, and a temporal encoder and a text encoder included in the contrastive learning model; The first initial model is trained according to the multiple training samples to obtain a large multimodal time series model.

2. The method according to claim 1, characterized in that The performing model training according to the plurality of training samples to obtain a comparative learning model includes: Step a: Encode the sample time series data of each training sample through the initial time series encoder of the second initial model to generate a first sample time series vector; Step b: encoding the sample text of each training sample through the initial text encoder of the second initial model to generate a first sample text vector; Step c: for each training sample, calculating the similarity between the first sample time series vector and the first sample text vector corresponding to the training sample; Step d: Calculate the cross entropy loss according to the similarity corresponding to each training sample; Step e: updating the model parameters in the second initial model according to the cross entropy loss; Repeat step a to step e to iteratively train the second initial model until a training cutoff condition is met, thereby obtaining the comparative learning model.

3. The method according to claim 2, characterized in that The step of training the first initial model according to the plurality of training samples to obtain a multimodal time series large model includes: Input the multiple training samples into the contrastive learning model, and obtain the similarity corresponding to each training sample output by the contrastive learning model; Determine the training samples whose similarity is higher than the preset similarity as target training samples; Based on the target training sample, the first initial model is trained to obtain the multimodal time series large model.

4. The method according to claim 3, characterized in that The step of training the first initial model based on the target training sample to obtain the multimodal time series large model includes: Step f, performing mask processing on the sample explanatory answers in the sample text of the target training sample to obtain the processed sample text; Step g: encoding the sample time series data of the target training sample by the time series encoder to generate a second sample time series vector; Step h: encoding the processed sample text of the target training sample by the text encoder to generate a second sample text vector; Step i: Processing the second sample time series vector by the query generator to generate a third sample time series vector; Step g, concatenating the third sample time series vector and the second sample text vector to generate a fusion vector; Step k, processing the fusion vector through the LLM to generate a predicted explanatory answer; Step 1: updating the model parameters in the first initial model according to the predicted explanatory answer and the sample explanatory answer of the target training sample; Repeat step f to step l to iteratively train the first initial model until a training cutoff condition is met, thereby obtaining the multimodal time series large model.

5. The method according to claim 4, characterized in that The updating of the model parameters in the first initial model comprises: updating model parameters of the query generator in the first initial model; or, Update all model parameters in the first initial model.

6. The method according to claim 2 or 4, characterized in that: The training cutoff condition is that the number of iterations reaches a preset number of iterations.

7. A model training device, characterized in that: include: A first acquisition module is used to acquire a plurality of training samples, each of which includes sample time series data and corresponding sample text, wherein the sample text includes sample description prompt words and sample explanatory answers for the sample time series data; A second acquisition module, used to perform model training according to the multiple training samples to obtain a comparative learning model; A first processing module, configured to construct a first initial model based on a query generator, an LLM, and a temporal encoder and a text encoder included in the contrastive learning model; The second processing module is used to train the first initial model according to the multiple training samples to obtain a multimodal time series large model.

8. An electronic device, characterized in that: include: Memory, processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory, so that the processor performs the method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 6 when executed by a processor.

10. A computer program product, characterized in that The invention comprises a computer program, which implements the method according to any one of claims 1 to 6 when being executed by a processor.

Images