Cross-modal water volume prediction method based on large model

Abstract

The invention discloses a cross-modal water volume prediction method based on a large model, and the method comprises the steps: obtaining original water volume data, and carrying out the preprocessing of the original water volume data; segmenting the preprocessed original water volume data into data blocks; performing modal alignment on the data blocks, and converting time sequence features in the data blocks into first text features corresponding to original quantity data; setting prompt words; inputting the cue word into the large model to generate a second text feature; inputting the first text feature and the second text feature into a large model for feature extraction; and carrying out linear projection on the output of the large model so as to generate a water volume prediction result. According to the method, the precision of water volume prediction is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of intelligent secondary water supply systems, and in particular relates to a cross-modal high-precision water volume prediction method based on a large model. Background Art

[0002] At present, water supply forecasting methods can be roughly divided into: regression analysis, time series method, machine learning method, and deep learning method. Among them, regression analysis and time series method based on statistical theory belong to traditional forecasting methods, and representative methods include linear regression, exponential smoothing method, and moving average method. Machine learning and deep learning belong to artificial intelligence methods, and representative methods include multilayer perceptron (MLP), back propagation neural network (BP), random forest (RF), support vector regression (SVR), long short-term memory (LSTM) network, etc.

[0003] In the past decade, deep learning technology has made great progress. Large-scale neural network models such as BERT (Bidirectional Encoder Representations from Transformers), GPT (Generative Pre-Trained Transformer) series, Vision Transformer (ViT), etc. have shown excellent performance in single-modality tasks such as Natural Language Processing (NLP) and Computer Vision (CV). Although pre-trained basic models have made great progress in the fields of NLP and CV, their development in the field of time series is limited by the lack of data and how to effectively combine time series data and natural language patterns to utilize these features is also a challenge. Summary of the invention

[0004] In view of this, the present invention hopes to provide a cross-modal water volume prediction method based on a large model to solve or alleviate the problems of low accuracy and insufficient samples in the existing water volume prediction technology.

[0005] The objective of the present invention is achieved through the following technical solutions:

[0006] A cross-modal water volume prediction method based on a large model, characterized by comprising:

[0007] S1: obtaining raw water volume data and preprocessing the raw water volume data, wherein the raw water volume data includes a time series composed of water consumption at different times;

[0008] S2: converting the pre-processed raw water volume data, that is, dividing the pre-processed raw water volume data into data blocks;

[0009] S3: performing modal alignment on the data block, converting the temporal features in the data block into the original quantity data corresponding to the first text feature;

[0010] S4: Set prompt words to guide the large model to complete the water volume series prediction task;

[0011] S5: input the prompt word into the large model to generate the second text feature;

[0012] S6: Inputting the first text feature and the second text feature into the large model for feature extraction;

[0013] S7: Linearly project the output of the large model to generate water volume prediction results.

[0014] Furthermore, preprocessing includes: outlier processing, missing value filling and data set partitioning.

[0015] Furthermore, the granularity of data set division is hourly granularity or minute granularity.

[0016] Furthermore, S2 specifically includes dividing the original water volume data through a sliding window according to the length of the input sequence, the length of each data block and the moving step of the sliding window.

[0017] Furthermore, S3 specifically includes:

[0018] Get the pre-trained vocabulary V;

[0019] Select the time-related words from the vocabulary V to form a subset V';

[0020] Through the multi-head cross attention mechanism, the subset V' is matched with the data block to determine the words in the NLP field corresponding to each data block and generate the first text feature.

[0021] Furthermore, the prompt words include: the source of the water volume data set, the time granularity, the time span, the maximum value, the minimum value, the mean value of the water volume data, the task objectives to be completed, etc.

[0022] Furthermore, S5 specifically includes:

[0023] The prompt word is segmented, and the segmentation result is mapped to a continuous vector representation through a word embedding generator to generate an embedding vector, which is the second text feature.

[0024] Furthermore, S6 specifically includes:

[0025] concatenating the second text feature with the first text feature as a prefix;

[0026] The splicing results are input into the large model for prediction.

[0027] Furthermore, the large model is a LLAMA model.

[0028] The beneficial effects of the present invention are:

[0029] The present invention provides a cross-modal water volume prediction method based on a large model. The method divides the data into data blocks through preprocessing of the data, sets prompt words, and uses the large model to predict the water volume, thereby significantly improving the prediction accuracy.

[0030] Other advantages, objectives and features of the present invention will be described in the following description to some extent, and to some extent, will be obvious to those skilled in the art based on the following examination and study, or can be taught from the practice of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be further described in detail below with reference to the accompanying drawings, in which:

[0032] Figure 1 It is a schematic flow chart of the cross-modal water volume prediction method based on a large model;

[0033] Figure 2 This is the comparison result diagram before and after data preprocessing;

[0034] Figure 3 This is the result graph of the LLAMA model prediction curve and the actual curve. DETAILED DESCRIPTION

[0035] The preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings. It should be understood that the preferred embodiments are only for illustrating the present invention, rather than for limiting the protection scope of the present invention.

[0036] This application proposes a cross-modal water volume prediction method based on a large model, which can provide high-precision water volume prediction results for the intelligent secondary water supply system in smart water services, serving the subsequent leakage detection and pump combination energy saving. Figure 1 , the method specifically comprises the following steps:

[0037] S1: obtaining raw water volume data and preprocessing the raw water volume data, wherein the raw water volume data includes a time series composed of water consumption at different times;

[0038] S2: converting the pre-processed raw water volume data, that is, dividing the pre-processed raw water volume data into data blocks;

[0039] S3: performing modal alignment on the data block, converting the temporal features in the data block into the original quantity data corresponding to the first text feature;

[0040] S4: Set prompt words to guide the large model to complete the water volume series prediction task;

[0041] S5: input the prompt word into the large model to generate the second text feature;

[0042] S6: Inputting the first text feature and the second text feature into the large model for feature extraction;

[0043] S7: Linearly project the output of the large model to generate water volume prediction results.

[0044] In S1, data preprocessing includes: outlier processing, missing value filling and data set division.

[0045] Abnormalities include sudden abnormalities and noise abnormalities. For sudden abnormalities, 3_sigma criteria, box plots, etc. can be used for detection; for noise abnormalities, Kalman filtering, wavelet transform threshold processing, etc. can be selected to filter out noise interference.

[0046] Missing data includes missing data and missing values ​​after outliers are removed. Missing values ​​can be filled, for example, the data of the same day and time of the previous week can be used to fill it. If the data is also missing, it can be postponed to the same time of the week before that. Because water volume data has a periodic change over time, this is reasonable and better maintains the water volume data pattern. Missing values ​​can also be filled by taking the average value.

[0047] The granularity of data set division is hourly or minute-by-minute. For example, taking a certain community as an example:

[0048] (1) Hourly granularity data set: The historical water consumption of a community for two years is used as the training set, the data of the first half of the third year is used as the validation set, and the data of the second half of the year is used as the test set to verify the prediction performance of the model.

[0049] (2) Minute-granularity dataset: The historical water consumption of a community in the first four months is used as the training set, the next two months as the validation set, and the last month as the test set to verify the prediction performance of the model.

[0050] It should be noted that the above division of the data set is only exemplary and is not intended to limit the present invention. The division of the training set, the validation set, and the test set can be performed according to any appropriate time length or ratio.

[0051] Water volume data is a discrete value, which contains relatively little information and is difficult to describe in words. However, each segment of water volume data contains more abundant water volume trend information, such as rise, fall, peak, etc., which can provide more information for the model, form a compact input tag sequence, and reduce the computational burden. Therefore, in S2, the original water volume data can be divided into data blocks one by one. Specifically, the original water volume data can be divided by sliding the window according to the length of the input sequence, the length of each data block, and the moving stride of the sliding window.

[0052] The formula for calculating the total number P of input data blocks is as follows:

[0053]

[0054] L p represents the length of each data block; T represents the total length of the input sequence; S represents the stride of each movement of the horizontal sliding window; in each sliding, the window moves S time steps, and each sliding takes a data block. +1 means to include the last available data block, and the total number of data blocks can be obtained by the above formula.

[0055] Since there is a big difference between time series and text, in order to narrow this gap, the vocabulary pre-trained by the large model can be used. Its rich corpus knowledge can help the model predict more accurately. However, the vocabulary pre-trained by the large model is very large, which will lead to huge occupation and consumption of computing resources. Therefore, S3 can specifically include:

[0056] Get the pre-trained vocabulary V;

[0057] Select the time-related words from the vocabulary V to form a subset V';

[0058] Through the multi-head cross attention mechanism, the subset V' is matched with the data block to determine the words in the NLP field corresponding to each data block and generate the first text feature.

[0059] Specifically, the huge vocabulary V can be projected into a subset V' of it through the projection layer training of S3, and the words in V' are related to the time series. The words in this subset are selected based on their relevance to the water volume time series data, and can retain information that has an important impact on the task. Then, a multi-head cross-attention mechanism is used to match the subset V' related to the time series with the data block, allowing the adaptive selection of relevant information to find the most corresponding words for each data block and the NLP field. This ability is not inherent and requires learning and training. In this way, the data in the time series field is connected with the text field, and modal alignment is achieved, so that the large model understands the characteristics of the time series and activates the time series understanding and reasoning capabilities of the large model.

[0060] The main function of the projection layer is to map the original large vocabulary V to a smaller subset V'. This process is actually a process of feature dimensionality reduction. During the training process, the large model calculates the error loss between the predicted value and the true value based on the given water volume prediction target to reflect the performance of the model in the current prediction. Through back propagation, the error will generate a gradient, which refers to the derivative of the error with respect to the model parameters, and the gradient is used to update the model parameters. The projection layer is a trainable layer that receives gradient updates from the loss function. According to the correlation between the time series features and vocabulary in the training data, the projection layer will gradually adjust its weights to learn which vocabulary is helpful for time series prediction and which vocabulary is irrelevant.

[0061] In S4, prompt words refer to words or sentences that describe the characteristics of water volume data in words, such as the source of the water volume data set, time granularity, time span, maximum value, minimum value, mean value of water volume data, and task objectives to be completed. Putting prompt words before time series data (as prefixes) makes it easier for the big model to understand the task objectives. In this way, the input context can be enriched and the big model can be guided to complete the transformation of time series data.

[0062] Specifically, the prompt word can contain three parts: background information of the data set, task objectives, and statistical information of the input data. Among them, the background of the data set (for example, data source, time span, etc.) is provided to the language model so that the model can understand the basic characteristics of the input water volume time series and the characteristics of the field; the task goal is used as a prompt word to provide task guidance for the large model, so that the large model knows that it is to complete the task in the field of time series prediction, so as to ensure that the model generates accurate predictions in specific tasks; the statistical information of the input data (for example, maximum value, minimum value, mean, etc.) can help the model to perform pattern recognition and reasoning, and combine the input sequence with the key statistical information of the water volume data, so that the model can better understand the characteristics of the time series data.

[0063] In S5, the prompt words are input into the frozen large model, including:

[0064] The prompt word is segmented from a sentence into smaller units of words, and the segmentation result is mapped to a continuous vector representation through a word embedding generator to generate an embedding vector, which is the second text feature. The embedding representation captures the semantic and syntactic features of the word so that the large model can process it in the vector space.

[0065] It should be noted that the frozen large model here refers to a pre-trained large model, whose weights and parameters are all pre-trained and do not need to be retrained during use in this example.

[0066] After the first text feature and the second text feature are determined, the two are input into the large model, i.e., step S6, wherein S6 specifically includes:

[0067] concatenating the second text feature with the first text feature as a prefix;

[0068] The splicing result is input into the large model for prediction. The large model here is also a frozen large model, and the output feature is obtained. The output feature is of the same length as the output of S3 (i.e., the first text feature). This is because the part about the S5 prompt word has been discarded during the prediction process, and the prompt word is only used to guide the large model to better complete the time series prediction task.

[0069] Since the target expected output of the time series prediction task is not text but the predicted numerical result, in S7, the output layer of the large model is replaced with a fully connected mapping layer, and the time series output layer is trained. After training, the text output is mapped to the time series, and the output is flattened for linear projection to obtain the final water volume prediction result S8.

[0070] Since the output of the large model is a sequence in the text field, the sequence is flattened into a vector, that is, the multi-dimensional model output tensor is flattened into one dimension to adapt to the shape of the fully connected layer, so that the final water volume data result can be obtained through linear projection through the fully connected layer. The linear projection operation can extract important features from the data, thereby improving the prediction ability of the model.

[0071] The advantages of the present invention are described below through specific experiments.

[0072] Experimental model and parameter index selection

[0073] Eight models were selected for the experiments, which can be divided into the following three categories:

[0074] (1) Traditional machine learning:

[0075] SVR (Support Vector Regression), an extension of support vector machine, is designed to solve regression problems. It fits data by finding an optimal hyperplane in high-dimensional feature space. It is suitable for processing nonlinear relationships and has good generalization ability.

[0076] RF (Random Forest): An ensemble learning method that improves accuracy by building multiple decision trees and combining their prediction results. It is effective for both classification and regression tasks, has the ability to resist overfitting, and can handle high-dimensional data and missing values.

[0077] (2) Deep Learning:

[0078] LSTM (Long Short-Term Memory): A special recursive neural network that can effectively capture dependencies in long time series data. It controls the flow of information by introducing memory units and gating mechanisms (input gate, forget gate, output gate). It is often used in time series prediction and natural language processing.

[0079] TC_MLP (Two channels Multi-Layer Perceptron): A model that combines horizontal and vertical time series prediction for processing time series data. It extracts temporal features through convolutional layers and then predicts through fully connected layers, which is suitable for capturing local patterns in time series.

[0080] Autoformer: An autoregressive model specifically designed for time series prediction that combines self-attention and convolution to effectively capture long-term dependencies. The model's structural design reduces computational costs.

[0081] (3) Large model:

[0082] LLAMA (Large Language Model MetaAI): A large-scale language model developed by Meta (formerly Facebook) to improve text generation and understanding. It performs well on a variety of natural language processing tasks.

[0083] BERT (Bidirectional Encoder Representations from Transformers): A pre-trained language model proposed by Google, based on the Transformer architecture. It can capture rich relationships between words through bidirectional context understanding.

[0084] GPT-2 (Generative Pre-trained Transformer 2): A generative language model developed by OpenAI based on the Transformer architecture. It is mainly used to generate coherent text and can complete various natural language generation tasks.

[0085] This experiment selected the following four indicators to comprehensively evaluate the performance of the water volume prediction model and help optimize the accuracy of the model.

[0086] (1) MAE (Mean Absolute Error): Mean absolute error, which reflects the average level of prediction error by calculating the average absolute error between the predicted value and the true value. It is expressed as:

[0087]

[0088] Among them, is y i Actual value, is the predicted value and n is the number of data points.

[0089] (2)MAPE (Mean Absolute Percentage Error): Mean absolute percentage error, which measures the average percentage of the prediction error relative to the true value, and is expressed as:

[0090]

[0091] (3) RMSE (Root Mean Squared Error): Root mean square error, which is sensitive to large errors and suitable for situations where large errors are emphasized. It can be expressed as:

[0092]

[0093] (4)R 2 (Coefficient of Determination): The coefficient of determination measures the degree to which the model explains the variability of the data and is expressed as:

[0094]

[0095] in, Represents the mean of the true values.

[0096] Parameter settings and results

[0097] Experiment 1:

[0098] Parameter setting: The hourly granularity HourFlow.csv dataset is used for water volume forecasting, and the forecast sequence length is set to 24 to compare the forecasting effects of different models. Table 1 shows the specific parameter settings:

[0099] Table 1 Parameter settings

[0100]

[0101] According to the parameters set in Table 1, the specific experimental results are shown in Table 2:

[0102] Table 2 Results of Experiment 1

[0103] Model MAE MAPE RMSE <![CDATA[R 2 ]]> 1 SVR 3.54 8.33 5.23 0.87 2 RF 4.98 10.07 6.79 0.78 3 LSTM 4.19 9.74 5.75 0.84 4 TC_MLP 3.08 6.76 4.51 0.90 5 Autoformer 0.32 3.27 0.40 0.84 6 LLAMA 0.24 1.09 0.38 0.87 7 BERT 0.28 1.70 0.42 0.82 8 GPT_2 0.29 0.91 0.45 0.77

[0104] The header of Table 2 is the name of the four evaluation indicators, the left column is the model name, and the values ​​in the table are the indicator values ​​of the corresponding model. Among them, the bold values ​​in each column represent the top three of the four water volume prediction indicators (MAE, MAPE, RMSE, R2) from best to worst. It can be observed that the water volume prediction performance indicators of most deep learning models are better than those of traditional machine learning models, and the effect of large model water volume prediction (i.e., rows 6, 7, and 8 in the table) is better than the prediction effect of deep learning models. This is because deep learning models have more layers of nonlinear transformation capabilities than traditional machine learning models, and can better capture complex high-dimensional data patterns and long-term dependencies. Compared with deep learning models, large models contain richer experience and knowledge, and have stronger expression capabilities, feature extraction capabilities, and the ability to process complex patterns.

[0105] Experiment 2:

[0106] Parameter setting: Select the model with better performance in Experiment 1, keep other parameter settings unchanged, and compare the prediction performance of the model on different data sets HourFlow.csv and MinFlow.csv when the prediction sequence length is 24 and 96 respectively. The experimental results are listed in Tables 3 and 4.

[0107] Table 3 Results of Experiment 2 (results of Autoformer and LLAMA)

[0108]

[0109] Table 4 Results of Experiment 2 (Results of BERT and GPT_2)

[0110]

[0111]

[0112] The table headers in Tables 3 and 4 are the model names, HourFlow and MinFlow represent the names of the hourly granularity dataset and minute granularity dataset respectively, 24 and 96 represent the prediction step sizes respectively, and the values ​​in the tables are the indicator values ​​of the models under the corresponding datasets and prediction step sizes.

[0113] It can be seen from Tables 3 and 4 that when the data set and step size are fixed, the LLAMA model has the best prediction effect among the above common large models (the bold results in the table), and only one is the second best.

[0114] In summary, the water volume prediction model proposed in the present invention can fully capture the complex change pattern of water volume data and effectively utilize the rich corpus knowledge of the large model for prediction. The local deployment of the large model also provides conditions for fine-tuning of subsequent research or modification of the overall model architecture, and has a broader application space.

[0115] The specific implementation steps (taking the LLAMA model for hourly water volume prediction as an example) are described below.

[0116] Step 1: Obtain and process community water volume data

[0117] The water consumption data used in this embodiment is the hourly granularity community water consumption data, the time span is 3 years, the sample size is 26305, the collected original community flow data is pre-processed according to the method described in S1, and the data results before and after the processing are compared. Figure 2 As shown, Figure 2 The figure above (i.e. Figure 2 (a)) shows the result before preprocessing; the following figure (i.e. Figure 2 (b)) represents the result after preprocessing. Figure 2 The horizontal axis represents the data index, and the vertical axis represents the value corresponding to each data. Figure 2 It can be seen that the data after preprocessing has removed obvious abnormal values.

[0118] Step 2: Deploy the large model locally

[0119] Search and select the required large model and click Download to save it to a custom path. You can select any existing open source large language model, such as Qwen, BERT, LLAMA, ChatGPT, etc. Conduct experiments based on water volume data and select a suitable large model. In this example, the LLAMA model with the best effect is selected as the water volume prediction model.

[0120] Step 3: Write the prompt words

[0121] Write English prompt words according to the characteristics of water volume data, which can describe the data source, task objectives, statistical information of the data, etc., to assist the large model in completing the water volume prediction task. The prompt words of this example are as follows, where the statistical information A, B, C, and D are calculated by the code:

[0122] “Large Model for Predicting Community Water Usage: This dataset provides decision-making support for secondary water supply systems by predicting water usage in acommunity. mean value of C,and median of D.”

[0123] The Chinese meaning of the above prompt words is as follows: "The large model is used to predict the water consumption of a community: This dataset provides decision support for the secondary water supply system by predicting the water consumption of a community. The dataset hourFlow.csv represents the cumulative water consumption per hour. The time span of the dataset is from September 1, 2017 to August 31, 2020. The dataset has a total of 26,305 samples, and its maximum mean is A, the minimum is B, the mean is C, and the median is D."

[0124] Step 4: Training

[0125] S2 and S3 are trained to find the words that best describe the characteristics of water volume data through a multi-head self-attention mechanism. The main purpose is to perform input conversion and modal alignment, convert the water volume time series into information in the NLP field to facilitate the understanding of large models, and introduce prompt words to guide, so as to better complete the prediction task.

[0126] This embodiment adopts the loss functions nn.MSELoss and nn.L1Loss commonly used in PyTorch.

[0127] The parameters are set as follows:

[0128] L p =16,S=8,P=64,train_epochs=50,seq_len=512,label_len=48,pred_len=24,T=520,

[0129] batch_size=24, learning_rate=0.01.

[0130] Step 5: Model prediction

[0131] Use the trained model to predict water volume and save the prediction results.

[0132] Step 6: Output prediction results and evaluation indicators

[0133] The evaluation index results of this embodiment are:

[0134] MAE:0.24405711889266968, R2:0.8546303510665894;

[0135] RMSE:0.38106951117515564, MAPE:1.0928150415420532;

[0136] In addition, this embodiment also provides two evaluation indicators: MSPE (Mean Squared Percentage Error) and MSE (Mean-Square Error):

[0137] in,

[0138] In this embodiment, the results of the two evaluation indicators, MSPE and MSE, are:

[0139] MSPE:73.65182495117188, MSE:0.14521397650241852.

[0140] The results of the predicted curve and the true curve in this embodiment are as follows Figure 3 As shown, the predicted curve is represented by red and the true curve is represented by black.

[0141] from Figure 3 As can be seen from the above evaluation index results, the prediction result of the water volume by the method of the present invention is close to the true value, and the prediction accuracy of the method of the present invention is high.

[0142] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solution, which should be included in the scope of the claims of the present invention.

Claims

1. A cross-modal water volume prediction method based on a large model, characterized in that: include: S1: obtaining raw water volume data and preprocessing the raw water volume data, wherein the raw water volume data includes a time series composed of water consumption at different times; S2: converting the pre-processed raw water volume data, that is, dividing the pre-processed raw water volume data into data blocks; S3: performing modal alignment on the data block, converting the temporal features in the data block into the original quantity data corresponding to the first text feature; S4: Set prompt words to guide the large model to complete the water volume series prediction task; S5: input the prompt word into the large model to generate the second text feature; S6: Inputting the first text feature and the second text feature into the large model for feature extraction; S7: Linearly project the output of the large model to generate water volume prediction results.

2. A cross-modal water volume prediction method based on a large model according to claim 1, characterized in that: Preprocessing includes: Outlier handling, missing value filling, and data set partitioning.

3. A cross-modal water volume prediction method based on a large model according to claim 2, characterized in that: The granularity of data set division is hourly or minute-by-minute.

4. A cross-modal water volume prediction method based on a large model according to claim 1, characterized in that: S2 specifically includes dividing the original water volume data through a sliding window according to the length of the input sequence, the length of each data block and the moving step of the sliding window.

5. A cross-modal water volume prediction method based on a large model according to claim 1, characterized in that: S3 specifically includes: Get the pre-trained vocabulary V; Select the time-related words from the vocabulary V to form a subset V'; Through the multi-head cross attention mechanism, the subset V' is matched with the data block to determine the words in the NLP field corresponding to each data block and generate the first text feature.

6. A cross-modal water volume prediction method based on a large model according to claim 1, characterized in that: The prompt words include: the source of the water volume data set, time granularity, time span, maximum value, minimum value, mean value of the water volume data, task objectives to be completed, etc.

7. A cross-modal water volume prediction method based on a large model according to claim 1, characterized in that: S5 specifically includes: The prompt word is segmented, and the segmentation result is mapped to a continuous vector representation through a word embedding generator to generate an embedding vector, which is the second text feature.

8. The cross-modal water volume prediction method based on a large model according to claim 1 is characterized in that: S6 specifically includes: concatenating the second text feature with the first text feature as a prefix; The splicing results are input into the large model for prediction.

9. The cross-modal water volume prediction method based on a large model according to claim 1 is characterized in that: The large model is the LLAMA model.

Images