Airport short-term load prediction method based on large language model migration
Through a method based on large language model migration, short-term airport load data are processed, and normalization, PCA, multi-head self-attention and LoRA technology are used, combined with cross-modal knowledge distillation and dual-branch structure, complex and high-dimensional in the existing technology are solved. The problem of difficult data processing significantly improves prediction accuracy.
Patent Information
- Application Number
- CN202411015413.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-26
- Publication Date
- 2024-11-15
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
The prior art is difficult to effectively process complex and high-dimensional short-term load data at airports, resulting in insufficient prediction accuracy.
Using a method based on large language model migration, time series data is processed through normalization and PCA dimensionality reduction, features are extracted in combination with multi-head self-attention and LoRA technology, and prediction capabilities are enhanced through cross-modal knowledge distillation and dual-branch structure.
It significantly improves the accuracy of short-term load prediction at the airport, overcomes the limitations of processing complex and high-dimensional data, and improves the accuracy and generalization capabilities of prediction.
Smart Images

Figure CN118966422A_ABST
Abstract
Description
Technical Field
[0001] The invention relates to an airport short-term load forecasting method based on large language model migration, and belongs to the field of time series forecasting based on deep learning. Background Art
[0002] In the dynamic system of airports, the demand for accurate short-term load forecasting is increasing. As a complex dynamic system, the operation of an airport involves multiple variables and uncertainties, such as flight takeoffs and landings, passenger flow, weather conditions, etc. Effective load forecasting can not only help management to reasonably allocate resources, such as ground staff, security facilities and boarding gate arrangements, but also improve the flexibility of flight scheduling, reduce passenger waiting time, and ultimately improve the overall passenger experience. However, due to the high volatility of historical load data, historical seasonal changes, and data quality issues such as missing values and noise, coupled with external uncontrollable factors such as weather changes and emergencies. The complexity of these multi-variable interactions together increases the difficulty of prediction.
[0003] The use of deep learning methods for load forecasting has been widely recognized as an effective technology that can provide timely and accurate forecast data for airport loads. In recent years, more and more deep learning technologies have been widely used in load forecasting, such as using various neural network technologies for time series analysis, pattern recognition, and feature extraction to predict load changes; building complex forecasting models based on the temporal and spatial characteristics of historical data to capture the laws in the data and make accurate forecasts; and extracting data features using recursive neural networks and long short-term memory networks by integrating load data with other data sources (such as weather data and socioeconomic data) to improve the accuracy of load forecasting. However, with the continuous increase in data size and complexity, these traditional methods are often difficult to handle the current more complex and higher-dimensional data environment. Summary of the invention
[0004] The technical problem to be solved by the present invention is to provide a method for predicting short-term load of an airport based on large language model migration, thereby improving the accuracy of short-term load prediction of an airport.
[0005] In order to solve the above technical problems, the present invention adopts the following technical solution: a method for predicting airport short-term load based on large language model migration is designed to achieve efficient prediction of airport short-term load data, including the following steps:
[0006] Step A. Normalize the original time series; the normalization method uses the z-score method, first calculates the mean and standard deviation of the original time series data in the training set; then, uses the calculated mean and standard deviation to normalize the original time series data, so that each data point is subtracted from the mean and divided by the standard deviation to obtain the normalized data, and at the same time performs PCA transformation on the huge word embedding matrix in the model to map the high-dimensional word vector to a low-dimensional space, while retaining the important information and structure of the original data as much as possible, identifying and extracting the most critical features for the time series prediction task.
[0007] The normalized time series data is divided into continuous patches, local semantic information is retained by aggregating local information into each patch, and the dimension of each patch is mapped to a high-dimensional vector space using a learnable matrix, and the patch embedding is reprogrammed. A small set of text prototypes is maintained, and a self-attention mechanism is used to achieve adaptive selection of relevant source information. The dot product similarity between each input data point and all text prototypes is calculated, and these similarities are converted into weights; then, the text prototypes are weighted summed to obtain an aggregated representation as supplementary information for the input sequence to align the modalities of time series and natural language, thereby activating the time series understanding and reasoning capabilities of the LLM backbone network.
[0008] The normalization operation is as follows: convert the data into a form with a mean of 0 and a standard deviation of 1. The mathematical expression of the normalization process is shown in formula (1);
[0009]
[0010] Among them, z t represents the normalized time series data point, that is, the standardized value, x t represents the tth data point in the original time series, μ represents the mean of the original time series, and σ represents the standard deviation of the original time series. The principal component analysis (PCA) method is used to reduce the dimension of is the size of the alphabet, M is the word embedding dimension, the purpose is to make semantically similar words form "
[0011] Synonym clusters" are used to reduce the number of entries. The mathematical expression for dimensionality reduction of word embedding dictionary is shown in formula (2):
[0012]
[0013] in, is the word embedding dictionary after dimensionality reduction, d is the predefined dimension, and then proceed to step B;
[0014] In step B, the multi-head self-attention (MHSA) mechanism is used to process the mapped time series data, and multiple attention heads are processed in parallel so that the model can focus on features on different time scales at the same time. In order to further improve the efficiency and expression ability of feature extraction, the low-rank adaptation (LoRA) technology is introduced.
[0015] By performing low-rank decomposition on the model's weight matrix, efficient updating of the model parameters is achieved while keeping the total number of model parameters relatively low.
[0016] Step B. Obtain the normalized time series from step A; embed the time series and reprogram it into the text representation space to align the modalities of the time series and natural language. First, embed the time series; divide the normalized time series into several consecutive overlapping or non-overlapping patches, and then retain the local semantic information by aggregating local information into each patch. The mathematical expression of the time series embedding process is shown in formulas (3)-(4):
[0017]
[0018] Among them, z t represents the normalized time series, L p represents the patch length, S represents the length of patch overlap, represents a continuous sequence of patches, W p represents the learnable embedding matrix, Represents the embedded patch sequence; secondly, reprograms the embedded patch sequence by embedding the reduced-dimensional word into the dictionary As the text prototype set E, the self-attention mechanism is used to achieve adaptive selection of relevant source information. Each input data point calculates the dot product similarity with all text prototypes, and uses the Softmax function to convert these similarities into weights. Then, the text prototypes are weighted summed to obtain an aggregated representation as supplementary information for the input sequence. The mathematical expression of each time step in the reprogramming process is shown in formulas (5)-(8):
[0019]
[0020] in, represents a continuous patch sequence, E represents a text prototype, and Represents the mapping matrix in the attention mechanism, Q k represents the query matrix, K k represents the bond matrix, V k represents the value matrix, d krepresents the matrix dimension, SOFTMAX represents the Softmax function, ΑTTENTION represents the attention operation, Z k Represents the aggregate representation obtained by attention calculation. The multi-head self-attention (MHSA) mechanism is used to process the mapped time series data, and by processing multiple attention heads in parallel, the model can simultaneously focus on features at different time scales. In order to further improve the efficiency and expression ability of feature extraction, the low-rank adaptation (LoRA) technology is introduced. The mathematical expressions are shown in formulas (9)-(11):
[0021]
[0022] Among them, Embedding represents the embedding operation, which is implemented with a single learnable linear layer, Q, K, and V represent query, key, and value matrices respectively, and Attention m represents the multi-head self-attention mechanism, is the learnable projection matrix of the hth attention head, H is the number of attention heads, and d k is the dimension of the key, W is the original weight matrix to be adapted, U and V are learnable matrices obtained by low-rank decomposition, and usually the rank of the product of U and V is much smaller than the size of W, then proceed to step C;
[0023] Step C. Get the aggregated representation in step B and construct a prompt as a prefix; In order to enhance LLM's understanding of the data, the input sequence needs to provide additional contextual information. First, the time dimension is clearly indicated by adding time tags (such as date, time, season, holidays, etc.). Then, the task instructions clearly state the prediction target, time span, and accuracy requirements, etc., to ensure that the model is clear about the tasks to be completed. In addition, background knowledge related to the task is provided as additional information, such as basic information about the airport, recent flight traffic trends, and weather conditions, to further help the model understand the input data, and then proceed to step D;
[0024] In the step D, during the feature extraction process, the features of the time series data are deeply integrated with the text features in the LLM through cross-modal knowledge distillation, ensuring that the information between different modalities can effectively interact and complement each other. Specifically, feature regularization loss and modality consistency loss are used to guide the model to learn the consistency and complementary features between cross-modalities.
[0025] Step D. Through cross-modal knowledge distillation, the features of time series data are deeply integrated with the text features in LLM to ensure that the information between different modalities can effectively interact and complement each other. Specifically, feature regularization loss and modality consistency loss are used to guide the model to learn the consistency and complementary features between cross-modalities. The mathematical expressions are shown in formulas (12)-(13):
[0026]
[0027] L output =sim(Y text ,Y time ) (13)
[0028] Among them, L is the number of model layers, γ is a hyperparameter that controls the loss ratio of different layers, and φ time,l and φ text,l are trainable projection layers for temporal and textual modalities, respectively, and F time,l and F text,l are the temporal and textual modal features of the lth layer, sim is the similarity function, and Y text and Y time They are the outputs of the text modality branch and the time modality branch, respectively, and then enter step E; in step E, an innovative dual-branch structure is adopted, which consists of a text modality branch and a time modality branch to achieve in-depth analysis and prediction of time series data. The text modality branch adopts a task-specific head structure, which allows the model to be optimized for specific prediction tasks. The time modality branch utilizes the inherent temporal association of time series data and captures patterns and regularities in the data through time series analysis technology.
[0029] Step E. A dual-branch structure of text modality branch and time modality branch is used to achieve in-depth analysis and prediction of time series data. The text modality branch uses a task-specific head structure, which allows the model to be optimized for specific prediction tasks. The time modality branch utilizes the intrinsic temporal association of time series data, and its mathematical expression is shown in formulas (14)-(16):
[0030] Y text =Head task (text) (14)
[0031] Y time =φ time,l (F time,L ) (15)
[0032] Loss=L(Y text ,Y time )+R (16)
[0033] Among them, Head task represents the task-specific header structure, F time,L Represents the temporal modal features of the last layer, and Loss is the prediction loss function, which measures Y text and Y timeThe difference between the true value and the true value, R is the regularization term, which is used to control the complexity of the model, and then enter step F;
[0034] Step F. Construct an airport short-term load forecasting model based on large language model migration, and use load data to train the model of the present invention. In the present invention, we use load data sets for experiments, the model is built based on the Pytorch framework, the training process uses the Adam optimizer for gradient update, the weight decay coefficient is 0.0001, the learning rate is 0.001, and the mean square error loss is used as the loss function. After completing the training of the model, the newly input time series data is predicted to generate a prediction result.
[0035] The airport short-term load forecasting method and application based on large language model migration described in the present invention, using the above technical solution compared with the prior art, has the following technical effects:
[0036] The present invention designs a short-term load forecasting model for airports based on a large language model (LLM). The method realizes effective preprocessing of data by normalizing the original time series data and using PCA technology to reduce the dimension. Then, the cross-modal knowledge distillation technology is used to embed the data into the latent representation space of LLM, and the multi-head self-attention and LoRA technology are used to extract features and align the time series with the natural language modality. The model's understanding of the data is further optimized by providing contextual information through constructing prompts. Finally, static and dynamic knowledge are combined, and a text-time dual-branch structure is used to enhance prediction accuracy and generalization ability. The design scheme of the present invention can effectively process airport load data with high uncertainty and complexity. Compared with traditional deep learning methods, this method has obvious advantages in combining local features and global features, overcomes the limitations in processing complex and high-dimensional data, and significantly improves the accuracy of short-term load forecasting. BRIEF DESCRIPTION OF THE DRAWINGS
[0037] Figure 1 is a flow chart of the steps of the present invention;
[0038] Figure 2 The schematic diagram of the framework of the airport short-term load forecasting model based on large language model migration;
[0039] Figure 3 Schematic diagram of the patch-seq method after embedding for reprogramming;
[0040] Figure 4 Schematic diagram of the method used as a prefix for constructing prompts;
[0041] Figure 5 Schematic diagram of the modal consistency loss calculation process;
[0042] Figure 6It is a visualization diagram of the actual value and predicted results of the load series.
[0043] Figure 7 Visualization of the distribution of forecast series errors for load forecasting. DETAILED DESCRIPTION
[0044] The specific implementation modes of the present invention will be further described in detail below in conjunction with the accompanying drawings.
[0045] The present invention designs a method for predicting airport short-term load based on large language model migration, which is used to achieve efficient prediction of airport short-term load data. In practical application, the method specifically includes the following steps.
[0046] Step A. Normalize the original time series; convert the data into a form with a mean of 0 and a standard deviation of 1. The mathematical expression of the normalization process is shown in formula (1);
[0047]
[0048] Among them, z t represents the normalized time series data point, that is, the standardized value, x t represents the tth data point in the original time series, μ represents the mean of the original time series, and σ represents the standard deviation of the original time series. The principal component analysis (PCA) method is used to reduce the dimension of is the size of the alphabet, M is the word embedding dimension, the purpose is to make semantically similar words form "
[0049] Synonym clusters" are used to reduce the number of entries. The mathematical expression for dimensionality reduction of word embedding dictionary is shown in formula (2):
[0050]
[0051] in, is the word embedding dictionary after dimensionality reduction, d is the predefined dimension, and then proceed to step B;
[0052] Step B. Obtain the normalized time series from step A; embed the time series and reprogram it into the text representation space to align the modalities of the time series and natural language. First, embed the time series; divide the normalized time series into several consecutive overlapping or non-overlapping patches, and then retain the local semantic information by aggregating local information into each patch. The mathematical expression of the time series embedding process is shown in formulas (3)-(4):
[0053]
[0054] Among them, z t represents the normalized time series, L p represents the patch length, S represents the length of patch overlap, represents a continuous sequence of patches, W p represents the learnable embedding matrix, Represents the embedded patch sequence; secondly, reprograms the embedded patch sequence. By embedding the reduced word into the dictionary As the text prototype set E, the self-attention mechanism is used to achieve adaptive selection of relevant source information. Each input data point calculates the dot product similarity with all text prototypes, and uses the Softmax function to convert these similarities into weights. Then, the text prototypes are weighted summed to obtain an aggregated representation as supplementary information for the input sequence. The mathematical expression of each time step in the reprogramming process is shown in formulas (5)-(8):
[0055]
[0056] in, represents a continuous patch sequence, E represents a text prototype, and Represents the mapping matrix in the attention mechanism, Q k represents the query matrix, K k represents the bond matrix, V k represents the value matrix, d k represents the matrix dimension, SOFTMAX represents the Softmax function, ΑTTENTION represents the attention operation, Z k Represents the aggregate representation obtained by attention calculation. The multi-head self-attention (MHSA) mechanism is used to process the mapped time series data, and by processing multiple attention heads in parallel, the model can simultaneously focus on features at different time scales. In order to further improve the efficiency and expression ability of feature extraction, the low-rank adaptation (LoRA) technology is introduced. The mathematical expressions are shown in formulas (9)-(11):
[0057]
[0058] Among them, Embedding represents the embedding operation, which is implemented with a single learnable linear layer, Q, K, and V represent query, key, and value matrices respectively, and Attention m represents the multi-head self-attention mechanism, is the learnable projection matrix of the hth attention head, H is the number of attention heads, and d k is the dimension of the key, W is the original weight matrix to be adapted, U and V are learnable matrices obtained by low-rank decomposition, and usually the rank of the product of U and V is much smaller than the size of W, then proceed to step C;
[0059] Step C. Get the aggregated representation in step B and construct a prompt as a prefix; In order to enhance LLM's understanding of the data, the input sequence needs to provide additional contextual information. First, the time dimension is clearly indicated by adding time tags (such as date, time, season, holidays, etc.). Then, the task instructions clearly state the prediction target, time span, and accuracy requirements, etc., to ensure that the model is clear about the tasks to be completed. In addition, background knowledge related to the task is provided as additional information, such as basic information about the airport, recent flight traffic trends, and weather conditions, to further help the model understand the input data, and then proceed to step D;
[0060] Step D. Through cross-modal knowledge distillation, the features of time series data are deeply integrated with the text features in LLM to ensure that the information between different modalities can effectively interact and complement each other. Specifically, feature regularization loss and modality consistency loss are used to guide the model to learn the consistency and complementary features between cross-modalities. The mathematical expressions are shown in formulas (12)-(13):
[0061]
[0062] L output =sim(Y text ,Y time ) (13)
[0063] Among them, L is the number of model layers, γ is a hyperparameter that controls the loss ratio of different layers, and φ time,l and φ text,l are trainable projection layers for temporal and textual modalities, respectively, and F time,l and F text,l are the temporal and textual modal features of the lth layer, sim is the similarity function, and Y text and Y time They are the outputs of the text modality branch and the time modality branch respectively, and then enter step E;
[0064] Step E. A dual-branch structure of text modality branch and time modality branch is used to achieve in-depth analysis and prediction of time series data. The text modality branch uses a task-specific head structure, which allows the model to be optimized for specific prediction tasks. The time modality branch utilizes the intrinsic temporal association of time series data, and its mathematical expression is shown in formulas (14)-(16):
[0065] Y text =Head task (text) (14)
[0066] Y time =φ time,l (Ftime,L ) (15)
[0067] Loss=L(Y text ,Y time )+R (16)
[0068] Among them, Head task represents the task-specific header structure, F time,L Represents the temporal modal features of the last layer, and Loss is the prediction loss function, which measures Y text and Y time The difference between the true value and the true value, R is the regularization term, which is used to control the complexity of the model, and then enter step F;
[0069] Step F. Construct an airport short-term load forecasting model based on large language model migration, and use load data to train the model of the present invention. In the present invention, we use load data sets for experiments, the model is built based on the Pytorch framework, the training process uses the Adam optimizer for gradient update, the weight decay coefficient is 0.0001, the learning rate is 0.001, and the mean square error loss is used as the loss function. After completing the training of the model, the newly input time series data is predicted to generate a prediction result.
[0070] The above-mentioned airport short-term load forecasting method based on large language model migration is used in the actual application process. The Electricity and UMass load data sets are taken as examples to verify the effect of the design and application of the present invention on short-term load data forecasting.
[0071] Experimental conditions: A computer is selected for the experiment. The computer is configured with NVIDIA GeForce GTXA6000 GPU, 64-bit operating system, Python (version 3.9) as the programming language, and Pytorch as the deep learning framework.
[0072] The experimental data are time series data of the Electricity and UMass load datasets. The Electricity load dataset is an important dataset widely used in power load forecasting research. It contains detailed electricity consumption information of 321 users and provides a rich and diverse sample library. Due to its wide coverage and covering the electricity consumption behaviors of different types of users, the Electricity dataset can reflect various electricity consumption patterns and demand changes. The UMass load dataset includes high-resolution electricity consumption data collected from various buildings and facilities on the University of Massachusetts campus. It provides detailed electricity consumption pattern information, which can be analyzed and predicted in detail. It is widely used to evaluate power load forecasting models in actual environments. After completing the training of the model according to the above design method, the newly input time series data is predicted to obtain the final prediction result. The complete implementation process is as follows. Figure 1 As shown, Figure 6 The figure shows the prediction result of the future series after executing the whole process. The line marked with GroundTruth on the figure represents the true value of the future load series, and the line marked with Prediction represents the predicted value of the future load series using the above design method.
[0073] In order to verify the prediction accuracy of the present invention for future load sequences, the mean square error (MSE) is used as an evaluation index to evaluate the model performance:
[0074]
[0075] Among them, Y i and The real time series in the future and the time series predicted by the model are respectively, t represents the number of time steps, and the smaller the MSE is, the better the model prediction effect is. On the Electricity and UMass data sets, the MSE indicators of the method of the present invention and other prediction methods are as shown in Table 1 below. Among them, the Electricity data set contains the electricity consumption series of 321 users, and the time sampling interval is 15min / point; the UMass data set includes the electricity load series of 114 households, and the time sampling interval is 15min / point.
[0076] The Electricity dataset comes from the UC Irvine Machine Learning Repository, which is a dataset repository serving the machine learning community. For details, see https: / / archive.ics.uci.edu / dataset / 321 / electricityloaddiagrams20112014.
[0077] The UMass dataset contains electricity load sequences of 114 households. These sequences come from minute-level electricity consumption data of multiple anonymous households. The UMass dataset comes from the UMass Trace Repository, a project aimed at optimizing household energy consumption. For details, see https: / / traces.cs.umass.edu / index.php / Smart / Smart.
[0078] MSE is one of the important indicators for evaluating the performance of a prediction model and is often used in regression problems. It measures the prediction accuracy of a model by calculating the difference between the predicted value and the actual value.
[0079]
[0080]
[0081] Based on what is shown in Table 1, it is proved that the design method of the present invention can achieve relatively accurate prediction for future short-term load sequences. On the Electricity dataset, the present invention achieved an MSE value of 0.148, which is the lowest value among all the models involved in the comparison, and is significantly better than other models. It is worth noting that compared with the worst comparison method Informer, the prediction accuracy of the present invention is improved by 42.8%; compared with the suboptimal comparison model Autoformer, the prediction accuracy of the present invention is improved by 29.5%. On the UMass dataset, the method of the present invention also showed excellent performance, with an MSE value of 0.143, proving its generalization ability and accuracy on different datasets. Combining the experimental results of the two datasets, it is verified that the method of the present invention has high accuracy and reliability in predicting future load sequences, especially in improving prediction accuracy and reducing model errors. In addition, the cross-modal knowledge distillation technology and dual-branch structure design of the method of the present invention provide a new research direction and solution for the field of load sequence prediction.
[0082] The airport short-term load forecasting model based on the large language model (LLM) designed by the above technical solution. This method realizes effective data preprocessing by normalizing the original time series data and using PCA technology to reduce the dimension. Then, the cross-modal knowledge distillation technology is used to embed the data into the latent representation space of LLM, and the multi-head self-attention and LoRA technology are used to extract features and align the time series with the natural language modality. Contextual information is provided by constructing prompts to further optimize the model's understanding of the data. Finally, static and dynamic knowledge are combined, and the text-time dual branch structure is used to enhance the prediction accuracy and generalization ability. The design scheme of the present invention can effectively process airport load data with high uncertainty and complexity. Compared with traditional deep learning methods, this method has obvious advantages in combining local features and global features, overcomes the limitations in processing complex and high-dimensional data, and significantly improves the accuracy of short-term load forecasting.
[0083] The embodiments of the present invention are described in detail above with reference to the accompanying drawings, but the present invention is not limited to the above embodiments, and various changes can be made within the knowledge scope of ordinary technicians in this field without departing from the purpose of the present invention.
Claims
1. A method for predicting airport short-term load based on large language model migration, used to achieve efficient prediction of airport short-term load data, characterized in that: The steps include: Step A: normalize the original time series data or the airport historical load series data, convert the original time series data or load series data into normalized data with a mean of 0 and a standard deviation of 1, and embed the word dictionary through principal component analysis (PCA) to extract key features, so that words with similar semantics will form "synonym clusters", and then enter step B; Step B: Using cross-modal knowledge distillation technology, the normalized time series data is embedded into the same latent representation space as the pre-trained large language model through mapping and reprogramming, and multi-head self-attention and low-rank adaptation (LoRA) technology are used to extract features from the time series, and the modalities of the time series and natural language are aligned, and then step C is entered; Step C: Use the dataset-related description and task description as supplementary information, construct prompts as prefixes, and provide additional contextual information for the embedded time series data to enhance the large language model's understanding of the data, and then proceed to step D; Step D: Construct static and dynamic knowledge related to the target data source, and jointly optimize the time series data and text representation through feature regularization loss and modality consistency loss to enhance the model's understanding of the target source data pattern, and then proceed to step E; Step E: Using the dual-branch structure of the text modality branch and the time modality branch, the prediction output is generated through the task-specific head structure, and the static knowledge and dynamic knowledge in the pre-trained LLMs are used to enhance the accuracy and generalization ability of time series prediction, and then enter step F; Step F: Apply the trained model to perform end-to-end prediction on the newly input time series data or airport load series data, generate prediction results, and verify the prediction performance through model evaluation indicators.
2. The airport short-term load forecasting method based on large language model migration according to claim 1 is characterized by: The normalization processing method uses the z-score method, firstly calculating the mean and standard deviation of the original time series data in the training set; then, using the calculated mean and standard deviation, the original time series data is normalized, so that each data point is subtracted from the mean and divided by the standard deviation to obtain the normalized data, and at the same time, the huge word embedding matrix in the model is PCA transformed to map the high-dimensional word vector to a low-dimensional space, while retaining the important information and structure of the original data as much as possible, and identifying and extracting the most critical features for the time series prediction task.
3. The airport short-term load forecasting method based on large language model migration according to claim 1 is characterized by: The normalized time series data is divided into continuous patches, local semantic information is retained by aggregating local information into each patch, and the dimension of each patch is mapped to a high-dimensional vector space using a learnable matrix, and the patch embedding is reprogrammed. A small set of text prototypes is maintained, and a self-attention mechanism is used to achieve adaptive selection of relevant source information. The dot product similarity of each input data point and all text prototypes is calculated, and these similarities are converted into weights; Then, the text prototypes are weighted summed to obtain an aggregated representation as complementary information to the input sequence to align the modalities of time series and natural language, thereby activating the time series understanding and reasoning capabilities of the LLM backbone network.
4. The airport short-term load forecasting method based on large language model migration according to claim 1 is characterized by: In step C, first, the time dimension is clearly indicated by adding time tags, including date, time, season, and holidays. Then, the task instructions clearly state the prediction target, time span, and accuracy requirements, etc., to ensure that the model is clear about the tasks to be completed. In addition, background knowledge related to the task is provided as additional information, including basic information of the data set, time series trend, and seasonal component changes, to further help the model understand the input data.
5. The airport short-term load forecasting method based on large language model migration according to claim 1 is characterized by: In the step D, during the feature extraction process, the features of the time series data are deeply integrated with the text features in the LLM through cross-modal knowledge distillation, ensuring that the information between different modalities can effectively interact and complement each other. Specifically, feature regularization loss and modality consistency loss are used to guide the model to learn the consistency and complementary features between cross-modalities.
6. The airport short-term load forecasting method based on large language model migration according to claim 1 is characterized by: In step F, after completing the training and tuning of the model, the model is deployed to the actual prediction task, and when the trained model is used for prediction, an end-to-end method is adopted to generate the prediction results, and the mean square error (MSE) is used as an evaluation indicator to measure the difference between the predicted value and the actual observed value.
7. The airport short-term load forecasting method based on large language model migration according to claim 1 is characterized by: Step A. Normalize the original time series; convert the data into a form with a mean of 0 and a standard deviation of 1. The mathematical expression of the normalization process is shown in formula (1); Among them, z t represents the normalized time series data point, that is, the standardized value, x t represents the tth data point in the original time series, μ represents the mean of the original time series, σ represents the standard deviation of the original time series, and at the same time, the huge word embedding dictionary in the model The principal component analysis (PCA) method is used to reduce the dimension of is the size of the alphabet, M is the word embedding dimension, the purpose is to make semantically similar words form " Synonym clusters" are used to reduce the number of entries. The mathematical expression for dimensionality reduction of word embedding dictionary is shown in formula (2): in, is the word embedding dictionary after dimensionality reduction, d is the predefined dimension, and then proceed to step B.
8. The method for airport short-term load forecasting based on large language model migration according to claim 1 is characterized by: Step B. Obtain the normalized time series from step A; embed the time series and reprogram it into the text representation space to align the modalities of the time series and natural language. First, embed the time series; divide the normalized time series into several consecutive overlapping or non-overlapping patches, and then retain the local semantic information by aggregating local information into each patch. The mathematical expression of the time series embedding process is shown in formulas (3)-(4): Among them, z t represents the normalized time series, L p represents the patch length, S represents the length of patch overlap, represents a continuous sequence of patches, Wp represents a learnable embedding matrix, Represents the embedded patch sequence; secondly, reprograms the embedded patch sequence by embedding the reduced-dimensional word into the dictionary As the text prototype set E, the self-attention mechanism is used to realize the adaptive selection of relevant source information. The dot product similarity between each input data point and all text prototypes is calculated, and the Softmax function is used to convert these similarities into weights. Then, the text prototypes are weighted summed to obtain an aggregated representation as the supplementary information of the input sequence. The mathematical expression of each time step in the reprogramming process is shown in formulas (5)-(8): in, represents a continuous patch sequence, E represents a text prototype, and Represents the mapping matrix in the attention mechanism, Q k represents the query matrix, K k represents the bond matrix, V k represents the value matrix, d k represents the matrix dimension, SOFTMAX represents the Softmax function, ATTENTION represents the attention operation, and Z k represents the aggregate representation obtained by attention calculation. The multi-head self-attention (MHSA) mechanism is used to process the mapped time series data. By processing multiple attention heads in parallel, the model can simultaneously focus on features at different time scales. In order to further improve the efficiency and expression ability of feature extraction, the low-rank adaptation (LoRA) technology is introduced. The mathematical expressions are shown in formulas (9)-(11): LoRA(W)=W+UV T (11) Among them, Embedding represents the embedding operation, which is implemented with a single learnable linear layer, Q, K, and V represent query, key, and value matrices respectively, and Attention m represents the multi-head self-attention mechanism, is the learnable projection matrix of the hth attention head, H is the number of attention heads, and d k is the dimension of the key, W is the original weight matrix to be adapted, U and V are the learnable matrices obtained by low-rank decomposition, and usually the rank of the product of U and V is much smaller than the size of W, then proceed to step C.
9. The method for airport short-term load forecasting based on large language model migration according to claim 1, characterized in that: Step D. Through cross-modal knowledge distillation, the features of time series data are deeply integrated with the text features in LLM to ensure that the information between different modalities can effectively interact and complement each other. Specifically, feature regularization loss and modality consistency loss are used to guide the model to learn the consistency and complementary features between cross-modalities. The mathematical expressions are shown in formulas (12)-(13): L output =sim(Y text ,Y time ) (13) Among them, L is the number of model layers, γ is a hyperparameter that controls the loss ratio of different layers, and φ time,l and φ text,l are trainable projection layers for temporal and textual modalities, respectively, and F time,l and F text,l are the temporal and textual modal features of the lth layer, sim is the similarity function, and Y text and Y time They are the outputs of the text modality branch and the time modality branch respectively, and then enter step E.
10. The airport short-term load forecasting method based on large language model migration according to claim 1 is characterized by: Step E. A dual-branch structure of text modality branch and time modality branch is used to achieve in-depth analysis and prediction of time series data. The text modality branch adopts a task-specific head structure, which allows the model to be optimized for specific prediction tasks. The time modality branch utilizes the intrinsic temporal association of time series data. Its mathematical expressions are shown in formulas (14)-(16): Y text =Head task (text) (14) Y time =φ time,l (F time,L ) (15) Loss=L(Y text ,Y time )+R (16) Among them, Head task represents the task-specific header structure, F time,L Represents the temporal modal features of the last layer, and Loss is the prediction loss function, which measures Y text and Y time The difference between the true value and the true value, R is the regularization term used to control the complexity of the model, and then enter step F.
Citation Information
Patent Citations
Power consumption prediction method and device, and terminal
CN117094451A
Visual big language model construction method for multi-source ship remote sensing image interpretation
CN117994677A
Video text retrieval method based on BEiT-3 multi-mode large model
CN118377930A
Cited By
Power load prediction method based on large language model and social population statistical information
CN119443751A
Power load forecasting method based on large language model and socio-demographic information
CN119443751B
Cross-modal water volume prediction method based on large model
CN119623704A
Co-traveling vehicle analysis method based on shared prototype space
CN119760323A
A peer vehicle analysis method based on shared prototype space
CN119760323B
