A Text Augmentation Temporal Prediction Method and System Based on Large Model Word Embedding Space
By employing a text-enhanced temporal prediction method based on a large model word embedding space, this method utilizes the frozen word embedding matrix of a large language model and a multi-head cross-attention mechanism to reprogram the temporal segment embedding vectors into semantic vectors of the same dimension as the text. Combined with a gated cross-attention fusion module, this method solves the problem of insufficient fusion between temporal and textual modalities, achieving high-precision and interpretable predictions.
Patent Information
- Application Number
- CN202610559700.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-04-27
- Publication Date
- 2026-05-26
- Estimated Expiration
- 2046-04-27
AI Technical Summary
In existing technologies, the fusion of temporal and textual modalities is not deep enough, continuous numerical data is difficult to adapt to the discrete lexical architecture of large models, and the quantitative analysis of textual influence lacks interpretability, resulting in decreased prediction accuracy and difficulty in tracing the source of results.
By constructing a text-enhanced temporal prediction method based on a large model word embedding space, and utilizing the frozen word embedding matrix of the large language model and the multi-head cross-attention mechanism, the temporal slice embedding vectors are reprogrammed into semantic vectors of the same dimension as the text. Combined with the gated cross-attention fusion module, the deep fusion of temporal and textual data in the same semantic space is achieved.
It achieves deep integration of time series and text data, improves prediction accuracy and meets the interpretability of decision-making, solves the problem of quantitative modeling of text influence, and improves the prediction accuracy and interpretability of energy systems.
Smart Images

Figure CN122087743A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of large language model and integrated energy system technology, and particularly relates to a text enhancement time-series prediction method and system based on large model word embedding space. Background Technology
[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.
[0003] With the rapid development of integrated energy systems, energy load time-series forecasting has become a core technical support for system planning, optimized scheduling, and safe operation. Traditional forecasting methods are mainly numerically driven, relying on deep learning models such as LSTM and Transformer. They can only extrapolate and model structured time-series data such as historical loads and meteorological data, and are unable to capture the impact of unstructured textual information such as meteorological disasters, equipment maintenance, and social events on load fluctuations. Under sudden scenarios, the prediction accuracy of such models drops significantly, failing to meet the refined operation requirements of modern energy systems.
[0004] The rise of large language models has provided a new direction for the fusion and prediction of unstructured text and time-series data. Their powerful semantic understanding and context encoding capabilities can transform textual information into high-dimensional features, achieving multimodal information complementarity. Current research mainly employs two technical approaches: one is to directly concatenate text and time-series data into the model using fine-tuning methods such as LoRA, freezing the main body of the large model and training a small number of side-channel parameters; the other is to construct a large model intelligent agent framework to simulate the qualitative analysis of the impact of expert logic on the text.
[0005] However, existing methods have obvious drawbacks: First, text and time-series data are only superficially concatenated and not deeply integrated in a unified space; second, the word segmentation mechanism of large models is designed for discrete word units, and directly processing continuous numerical time-series data will cause serious information loss; third, the text impact analysis based on agents is mainly qualitative and lacks interpretable quantitative modeling basis, making it difficult to trace the source of prediction results. Summary of the Invention
[0006] To overcome the shortcomings of the prior art, this invention provides a text-enhanced temporal prediction method and system based on a large model word embedding space. It aims to solve the technical problems in the prior art, such as the shallow fusion of temporal and text modalities, the difficulty in adapting continuous numerical data to the discrete word matrix architecture of large models, and the lack of interpretability in the quantitative analysis of text influence.
[0007] To achieve the above objectives, one or more embodiments of the present invention provide the following technical solutions: The first aspect of this invention provides a text-enhanced temporal prediction method based on a large model word embedding space; Text augmentation time-series prediction methods based on large model word embedding spaces include: Acquire historical load time-series data and perform normalization processing; Normalized historical load time-series data is input into the time-series piece reprogramming module. The time-series piece embedding vector is obtained through linear projection and layer normalization. The time-series piece embedding vector is reprogrammed into a time-series semantic vector with the same dimension as the text by using the frozen word embedding matrix of the large language model and multi-head cross attention. Unstructured text data corresponding to historical load time series data is obtained and input into the text encoding module. The frozen large language model is used for encoding to obtain word-level text semantic vectors. Using temporal semantic vectors as queries and textual semantic vectors as keys and values, a gated cross-attention fusion module is used to perform gated cross-attention calculation to obtain deep fusion features. The deep fusion features are input into the prediction head module, and the final prediction result is output.
[0008] As a further technical solution, the process of obtaining time-series piecewise embedding vectors from normalized historical load time-series data through linear projection and layer normalization includes: The normalized historical load time-series data is divided into multiple time-series fragments using a sliding window. Each temporal slice is mapped to a hidden dimension through a learnable linear projection layer, and then normalized to become a slice embedding vector.
[0009] As a further technical solution, the word embedding matrix of the frozen large language model is utilized, and the temporal slice embedding vectors are reprogrammed into temporal semantic vectors of the same dimension as the text through multi-head cross-attention, including: A candidate word embedding set is constructed by selecting multiple representative word vectors from the word embedding matrix; Using the segmented embedding vector as the query and the candidate word embedding set as the key and value, cross-attention calculation is performed to obtain a continuous semantic vector aligned with the word embedding space of the large language model.
[0010] As a further technical solution, unstructured text data is encoded using a frozen large language model to obtain word-level text semantic vectors, including: The unstructured text data is segmented into word sequences using a large language model's word segmenter, and then padded and truncated on the right. The processed word sequence is fed into the Transformer module of the model for layer-by-layer processing, and the hidden state output of the last layer is extracted. After filtering the output hidden state through attention masking, dimensionality reduction through linear projection layer, and layer normalization to stabilize the distribution, a word-level semantic representation of the text is obtained.
[0011] As a further technical solution, the process of using a gated cross-attention fusion module to perform gated cross-attention calculation to obtain deep fusion features includes: Using the temporal semantic vector as the query matrix and the word-level text semantic vector as both the key and value matrix, multi-head cross-attention calculation is performed to obtain the initial text enhancement vector. The initial text enhancement vector is input into the gated sublayer, and the gated network independently calculates a scalar gate coefficient for each temporal slice vector. The attention output is weighted and fused based on the acquired gating coefficients to obtain deep fusion features.
[0012] As a further technical solution, the process of outputting the final prediction result using the prediction head module includes: The deep fusion features are flattened into a one-dimensional vector along the piecewise dimension, and then the dimension is gradually compressed through two fully connected networks and mapped to the output of the prediction length dimension to obtain the predicted value. The final prediction result is obtained after inverse normalization of the predicted values.
[0013] A second aspect of the present invention provides a text-enhanced temporal prediction system based on a large model word embedding space.
[0014] A text-enhanced temporal prediction system based on a large model word embedding space includes: The data acquisition and normalization module is configured to: acquire historical load time-series data and perform normalization processing; The temporal segmentation reprogramming module is configured to: accept normalized historical load temporal data as input, obtain temporal segmentation embedding vectors through linear projection and layer normalization; and reprogram the temporal segmentation embedding vectors into temporal semantic vectors of the same dimension as the text by using the word embedding matrix of the frozen large language model and multi-head cross attention. The text encoding module is configured to: acquire unstructured text data corresponding to historical load time series data, encode the unstructured text data using a frozen large language model, and obtain word-level text semantic vectors; The gated cross-attention fusion module is configured to use temporal semantic vectors as queries and textual semantic vectors as keys and values to calculate deep fusion features through gated cross-attention. The prediction head module is configured to output the final prediction result based on the deep fusion features.
[0015] A third aspect of the present invention provides a computer-readable storage medium having a program stored thereon that, when executed by a processor, implements the steps of a text-enhanced temporal prediction method based on a large model word embedding space as described in the first aspect of the present invention.
[0016] A fourth aspect of the present invention provides an electronic device, including a memory, a processor, and a program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of a text-enhanced temporal prediction method based on a large model word embedding space as described in the first aspect of the present invention.
[0017] The fifth aspect of the present invention provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps in the text enhancement temporal prediction method based on a large model word embedding space described in the first aspect of the present invention.
[0018] The above one or more technical solutions have the following beneficial effects: (1) This invention transforms unstructured multi-source information into text information, and maps numerical time-series data and text data to the same semantic vector space through the word embedding matrix of a large language model, realizing natural and deep fusion of the two modalities at the feature level. Compared with the shallow fusion method of simple splicing in the prior art, this invention enables time-series features to actively query and selectively absorb related text semantic information through a gated cross-attention mechanism, effectively solving the problem of quantitative modeling of the impact of external events on the load.
[0019] (2) The temporal segmentation reprogramming mechanism proposed in this invention reprograms continuous numerical temporal data into continuous semantic vectors instead of discrete words by embedding the data into segments and then reprogramming it using cross-attention and the word embedding matrix of a large language model. This design avoids the information fragmentation and loss caused by traditional methods that directly segment numerical temporal data into discrete words through a word segmenter, enabling temporal data and text data to achieve natural alignment in the same vector space. The continuous semantic vectors retain the numerical pattern features of the original temporal sequence and also gain the ability to align with the semantic space of the large model.
[0020] (3) This invention enables the model to automatically query and select the text information most relevant to the current time series pattern through a learnable cross-attention mechanism and a gating network. The gating network independently calculates a scalar gating coefficient for each time series segment, which intuitively reflects the degree of dependence of each time series segment on text information, thus meeting the actual needs of energy systems for decision interpretability.
[0021] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description
[0022] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.
[0023] Figure 1 This is a flowchart of the method in the first embodiment.
[0024] Figure 2 This is a schematic diagram of the gated cross-attention fusion module in the second embodiment.
[0025] Figure 3 This is a system structure diagram of the second embodiment. Detailed Implementation
[0026] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.
[0027] It should be noted that the terminology used herein is for the purpose of describing particular implementations only and is not intended to limit the exemplary implementations of the present invention.
[0028] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.
[0029] Example 1 This embodiment discloses a text-enhanced temporal prediction method based on a large model word embedding space. By freezing the large language model to provide word embedding matrix and text encoding capabilities, a temporal segmentation reprogramming module, a text encoding module, a gated cross-attention fusion module, and a prediction head module are constructed to achieve deep fusion of temporal data and text data in the same semantic space. This achieves deep spatial fusion of temporal numerical values and text semantics, improving prediction accuracy and interpretability.
[0030] The proposed text-enhanced temporal prediction framework based on a large language model word embedding matrix space comprises four core modules: a temporal segmentation reprogramming module, a text encoding module, a gated cross-attention fusion module, and a prediction head module. The large language model (in this embodiment, Qwen3-8B-Base is used, with a hidden layer dimension of...) is... All parameters of (=4096) are frozen, providing only word embedding matrix and text encoding capabilities. This framework defines a unified hidden dimension. (In this embodiment) =256) serves as the common representation space dimension for all learnable modules. Temporal slices and text features are projected onto this dimension for interactive computation. All learnable parameters in the framework reside only in the above four modules, thus significantly saving training time and computational resources while fully preserving the original semantic understanding capabilities of the large model.
[0031] like Figure 1As shown, the text-enhanced temporal prediction method based on a large model word embedding space includes: Step S1: Obtain historical load time series data and perform normalization processing.
[0032] Hourly records are collected from the data monitoring platform or dispatch center of the target area to obtain historical time-series data. The historical time-series data is a one-dimensional continuous numerical sequence arranged chronologically. In this embodiment, hourly power load records are collected, with each data point corresponding to the total regional power grid load value (unit: MW) for one hour. The historical load time-series data is then processed... Normalization: using the mean of all load values in the training set. and standard deviation As a normalization parameter, the following transformation is performed for each load value:
[0033] Among them Original load value, To prevent extremely small constants from being divided by zero, which are calculated only from the training set, the validation and test sets reuse the same normalization parameters to avoid data leakage.
[0034] Set the normalized historical load time series data as The length of the input sequence is (In this embodiment) =96, which means 96 hours.
[0035] Step S2: Input the normalized historical load time series data into the time series segmentation reprogramming module, and obtain the time series segmentation embedding vector through linear projection and layer normalization; use the frozen word embedding matrix of the large language model, and reprogram the time series segmentation embedding vector into a time series semantic vector with the same dimension as the text through multi-head cross attention.
[0036] Step S21, for the normalized historical load time series data First, it is divided using a sliding window method, then the number of fragments is... As shown below:
[0037] in, For the number of fragments, This is the fragment length. This is the sliding step size.
[0038] Specifically, a one-dimensional expansion operation is used to extract the normalized load sequence by step size. Cut the length in sequence as follows The continuous subsequences are used to obtain the piecewise matrix, whose dimensions and shape are... Each shard Hidden dimensions are mapped through a learnable linear projection layer. After further normalization to stabilize the distribution, the temporal segment embedding vector is obtained, as shown below:
[0039] in, For time-series slice embedding vectors, The learnable piecewise linear projection weight matrix; This is the bias vector.
[0040] In this embodiment, continuous time series data is transformed into a set of discrete semantic slices through segmentation. Each slice carries local time series pattern information (such as the load increase trend within a 4-hour period, intraday periodic peaks and troughs, etc.). Adjacent slices are separated by a step size. Less than the fragment length And exist The overlap of time steps ensures the temporal continuity between segments, laying the foundation for subsequent alignment with the word embedding space.
[0041] Step S22: Using the word embedding matrix of the frozen large language model, the temporal slice embedding vector is reprogrammed into a temporal semantic vector with the same dimension as the text through multi-head cross attention.
[0042] The core idea of the reprogramming module is to use the word embedding matrix of a large language model to "translate" temporal segments from the numerical space to the semantic space of the large model, so that they can be fused with text features in the same space in subsequent steps.
[0043] First, the word embedding matrix of the frozen large model is: ,in For vocabulary size, This represents the hidden layer dimension of a large model. Due to the word embedding space... Vectors with larger norms have more significant activation values across all dimensions, meaning they occupy a more prominent position in the embedding space, possess the strongest spatial discriminative power, and can provide the richest semantic anchors for temporal segmentation. Therefore, in this embodiment, based on each word vector... Sort the word vectors by norm from largest to smallest, and select the most representative ones to form a candidate word embedding set. .
[0044] Then, the time-series slice embedding vector is used. spliced into a matrix And mapped to the hidden layer dimension of the large model via a learnable query projection layer. , as a query, with candidate word embedding set Simultaneously serving as both key and value, multi-head cross-attention computation is performed. Specifically, the query, key, and value are each decomposed into multiple attention heads through their respective linear projections, with each attention head formula as shown below:
[0045]
[0046]
[0047]
[0048] in, , , These are the query vector, key vector, and value vector for the reprogramming layer, respectively. The time series matrix is formed by concatenating the embedding vectors of the time series segments; To map to the hidden layer dimension of a large model Matrix; Matrix makes Mapped to ; The learnable query projection weight matrix in the reprogramming module; The learnable key projection weight matrix in the reprogramming module; Project a weight matrix onto the learnable values in the reprogramming module; Dimensions of each attention head in multi-head attention.
[0049] The above formula shows that each head independently calculates the scaled dot product attention, as follows:
[0050] in, This represents the dimension of each attention head. The outputs of each head are concatenated and then linearly projected to form a unified representation.
[0051] Finally, the output of the cross-attention is projected from the hidden layer dimension of the large model via a learnable linear projection layer. Map back to hidden dimensions The residual connections and layer normalization performed with the original piecewise embeddings are shown below:
[0052] in, This is a temporal semantic matrix; To reprogram the learnable output projection weight matrix in the module, the multi-head concatenation result is transferred from... Dimension Mapping Back dimensional, while the original temporal piecewise embedding matrix yes Dimensional, no transformation required.
[0053] Residual connections ensure that the reprogrammed vectors retain their original temporal numerical pattern features while achieving semantic space alignment. After reprogramming, each temporal slice vector is no longer a simple numerical feature, but a continuous semantic vector aligned with the word embedding space of the large language model, establishing a unified vector space foundation for subsequent fusion with text features.
[0054] Step S3: Obtain unstructured text data corresponding to historical load time series data and input it into the text encoding module. Encode the data using the frozen large language model to obtain word-level text semantic vectors.
[0055] The text encoding module is responsible for transforming unstructured text into semantic vector representations aligned with temporal slices, forming the foundation for multimodal fusion. This module fully leverages the powerful semantic understanding capabilities of the frozen large language model, mapping the original text into high-dimensional feature vectors containing rich contextual information. These vectors are then reduced to the same dimensionality as the temporal slices using a learnable projection layer. Hidden space.
[0056] Step S31: For the text description corresponding to the last moment in the input window (i.e., the latest context information available at the prediction moment), firstly, the large model's built-in word segmenter is used to segment it into a word sequence, and then right-side padding and truncation are performed, with a maximum length limit of 512 words. If the word segmenter does not preset padding words, the ending word is used instead.
[0057] In step S32, the word sequence is fed into the frozen large language model and processed layer by layer by the multi-layer Transformer module within the model. In each layer, the multi-head self-attention operator enables each word vector to interact and aggregate information with all other words in the sequence, thereby continuously enriching the context awareness of each word. Specifically, let the last hidden state be... , No. The hidden state of the layer is Initial layer input From lexical IDs via word embedding matrix The mapping is obtained. The multi-head self-attention query, key, and value matrices of the layer are obtained by projecting the frozen parameters, as shown in the formula:
[0058] in, , , These are the query vector, key vector, and value vector in the self-attention operation, respectively. This is the learnable query projection weight matrix in the fusion module; This is the learnable key projection weight matrix in the fusion module; Project the weight matrix onto the learnable values in the fusion module; Dimensions of each attention head in multi-head attention.
[0059] Perform multi-head self-attention calculation based on the above results, as shown below:
[0060] The feedforward neural network sublayer performs nonlinear transformation and feature extraction on the attention output.
[0061]
[0062] in, , This represents the weight matrix of the two fully connected layers in the feedforward layer. b 1. b 2 represents the corresponding bias vector; ReLU is the nonlinear activation function.
[0063] After being refined layer by layer, the hidden state output of the last layer is extracted, where M is the actual length of the word sequence.
[0064] The reason for extracting the last hidden state instead of the initial word embedding matrix mapping is that the same word has different semantic meanings in different sentences. However, since they share the same word ID, they correspond to the same high-dimensional initial vector in the word embedding matrix, so they do not possess contextual information. For example, the word "high" in "high temperature" and "high wind" points to the exact same initial vector in the word embedding matrix, but their semantics are clearly different in the context of energy load prediction—the former predicts an increase in cooling load, while the latter may affect wind power output. Only after the contextual attention calculation of multiple Transformer modules will "high" obtain differentiated semantic representations in the two sentences.
[0065] In step S33, the hidden state is first filtered by an attention mask, and the vector at the filling position is set to zero to avoid invalid words interfering with subsequent fusion.
[0066] Let the final hidden state output be The mask vector is ,in:
[0067] The masking filtering operation involves element-wise multiplication and broadcasting the multiplication along the feature dimension, i.e. This avoids interfering with subsequent semantic fusion.
[0068] Then, the hidden layer dimension D of the large model is reduced to the hidden dimension through a learnable linear projection layer. After being normalized and stabilized, the final text semantic representation at the word level is obtained, as shown below:
[0069] in, For word-level text semantic representation, For learnable dimension-reduced linear projection matrix, This is the bias vector.
[0070] The reason for retaining the complete word-level representation here instead of aggregating it into a single vector is to enable each temporal slice to selectively focus on different text words in a fine-grained manner in the subsequent gated cross-attention fusion module, thereby achieving more accurate semantic matching.
[0071] Step S4: Using the temporal semantic matrix as the query and the text semantic vector as the key and value, the gated cross-attention fusion module is used to perform gated cross-attention calculation to obtain deep fusion features.
[0072] like Figure 2 As shown, the gated cross-attention fusion module treats each temporal slice as a queryer, actively searching for semantic information most relevant to its own temporal pattern within the text lexical sequence. For example, the temporal slice representing a sudden drop in temperature will assign higher weights to text lexical terms such as "sudden temperature rise" and "blizzard" in the attention weight distribution, thus selectively absorbing the semantic information of the text vector corresponding to extreme weather; while the slice representing a normal daytime load increase pattern will have lower attention weights on such warning lexical terms, naturally ignoring text content irrelevant to itself.
[0073] In this embodiment, the gated cross-attention fusion module selectively absorbs and dynamically adjusts temporal features to semantic information in the text, making it a key module for achieving multimodal fusion. The module consists of three parts: a cross-attention sublayer, a gated sublayer, and a feedforward neural network sublayer.
[0074] Among them, the cross-attention sublayer uses the reprogrammed temporal semantic matrix As a query, it is represented by text lexical semantics. As both a key and a value, each attention head in multi-head attention is formulated as follows:
[0075]
[0076]
[0077] in, , , These are the query vector, key vector, and value vector of the fusion layer, respectively. This is the learnable query projection weight matrix in the fusion module; This is the learnable key projection weight matrix in the fusion module; Project the weight matrix onto the learnable values in the fusion module; Dimensions of each attention head in multi-head attention.
[0078] Perform multi-head cross-attention calculation based on the above results, as shown below:
[0079] in, The cross-attention output represents a temporal semantic representation that incorporates relevant textual information.
[0080] The cross-attention output is processed by layer normalization, as shown below:
[0081] in, The cross-attention sublayer provides each slice with a corresponding text enhancement vector, initially filtering out texts that can be effectively applied to the temporal sequence of that slice. However, the influence of each text on the current slice is different, and the force of the current text on each slice also varies.
[0082] Therefore, to precisely control the injection strength of text information, a learnable gating network is introduced in the gating sublayer, independently calculating a scalar gating coefficient for each temporal slice. First, the semantic representation of the text at the word level is analyzed. Perform mask mean pooling along the word dimension to obtain the global text summary vector. As shown below,
[0083] Where is the attention mask value of the j-th word (non-padding bits are 1, padding bits are 0). for The vector of the j-th word.
[0084] The mask mean pooling only averages the token vectors at non-padded positions, avoiding the dilution of the global text representation by padded tokens. Then... After expanding to a shape consistent with the number of slices, it is concatenated with each temporal slice vector along the feature dimension.
[0085] splicing vectors Processed sequentially through two layers of fully connected networks: Layer 1 Dimension mapping to Applying ReLU activation to the second layer introduces nonlinearity; the second layer will... The dimensionality is mapped to a scalar and a sigmoid activation function is applied to compress the output to the (0,1) interval, resulting in scalar gating coefficients, as shown below:
[0086] in, These are scalar gating coefficients; , These are the learnable weight matrices for the first and second layers of the gated network, respectively. , These are the learnable bias vectors of the first and second layers of the gated network, respectively.
[0087] Since the influence intensity produced by matching different semantic information with different time slices is different, the gating coefficient... It jointly considers the pattern features of the current time series segment and the global semantic overview of the text, adaptively judges the influence factor of the semantic information on the current time series segment, and accurately produces the correct influence on time series prediction through training.
[0088] In gated fusion and feedforward sublayers, when the gate value When =0, The model completely degenerates into a pure time-series prediction model, ensuring that the text module does not introduce noise interference when there is no effective text information; when When the value is greater than 0, text enhancement information is proportionally superimposed on the temporal representation, achieving progressive semantic injection. Therefore, the final fusion output is shown below:
[0089] in, Enhance the temporal semantic vectors of the text in each segment time period; For cross-attention output The i-th vector.
[0090] Then, the fusion vectors of all the fragments are stacked row by row to obtain the whole, as shown below:
[0091] in, This involves concatenating the temporal semantic vectors of the text segmented by time period.
[0092] The fused vector is then further processed by a feedforward neural network (FFN) to extract and integrate features. The FFN consists of two fully connected layers: the first layer... Dimensions extended to Applying the GELU activation function to introduce smooth nonlinearity, and adding regularization in the middle to prevent overfitting; the second layer then... Dimensional compression back The dimension is also regularized. After residual concatenation of the FFN output and input, layer normalization is performed to obtain the final fused representation, as shown below:
[0093] in, For the final deep fusion features, Step S5: Input the deep fusion features into the prediction head module and output the final prediction result.
[0094] The forecast head module transforms the fused representation into the final load forecast. First, it... Flattened into a one-dimensional vector along the partition dimension Then, the dimensions are progressively compressed and mapped to the predicted length dimension through two fully connected layers: the first layer maps the dimension to... The dimensionality is applied with the GELU activation function, and Dropout regularization is added in between; the second layer maps the dimensionality to the prediction length. Dimensioning the dimensions yields the final output, as shown below:
[0095] in, For predicting output, i.e., the future Load forecast values for each time step; The learnable weight matrix for the first layer of the prediction head; This is the bias vector for the first layer; The learnable weight matrix for the second layer of the prediction head; This is the bias vector for the second layer. The output predicted value is the load sequence in the normalized domain, which is obtained by inverse normalization using the mean and standard deviation calculated from the training set.
[0096] To verify the effectiveness of the proposed framework, a systematic experimental evaluation was conducted on a city's electricity load dataset. Mean squared error (MSE), mean absolute error (MAE), root mean square error (RMSE), and mean absolute percentage error (MAPE) were used to comprehensively evaluate the prediction performance. To verify the effectiveness of text augmentation, a pure time-series baseline model was designed as a control. The baseline model removed the large language model, reprogramming layer, text encoder, and gated fusion module, retaining only the piecewise embedding layer, standard Transformer encoder layer, and prediction head. Its structural parameters remained consistent with the corresponding modules in the proposed framework to ensure fairness in the comparison. The overall test results of the baseline model are compared with those of the proposed framework in Table 1.
[0097] Table 1 Overall Test Performance Comparison
[0098] As shown in Table 1, the framework proposed in this invention significantly outperforms the pure time-series baseline model across all four evaluation metrics. Specifically, MSE is reduced by 14.09%, MAE by 8.96%, RMSE by 7.31%, and MAPE by 3.774% (a reduction of 10.74%), down from 4.228%. This indicates that deeply fusing unstructured text information with time-series data through a word embedding matrix space can effectively improve load forecasting accuracy.
[0099] Example 2 This embodiment discloses a text-enhanced temporal prediction system based on a large model word embedding space; like Figure 3 As shown, a text-enhanced temporal prediction system based on a large model word embedding space includes: The data acquisition and normalization module is configured to: acquire historical load time-series data and perform normalization processing; The temporal segmentation reprogramming module is configured to: accept normalized historical load temporal data as input, obtain temporal segmentation embedding vectors through linear projection and layer normalization; and reprogram the temporal segmentation embedding vectors into temporal semantic vectors of the same dimension as the text by using the word embedding matrix of the frozen large language model and multi-head cross attention. The text encoding module is configured to: acquire unstructured text data corresponding to historical load time series data, encode the unstructured text data using a frozen large language model, and obtain word-level text semantic vectors; The gated cross-attention fusion module is configured to use temporal semantic vectors as queries and textual semantic vectors as keys and values to calculate deep fusion features through gated cross-attention. The prediction head module is configured to output the final prediction result based on the deep fusion features.
[0100] Example 3 The purpose of this embodiment is to provide a computer-readable storage medium.
[0101] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in a text-enhanced temporal prediction method based on a large model word embedding space as described in Example 1.
[0102] Example 4 The purpose of this embodiment is to provide an electronic device.
[0103] An electronic device includes a memory, a processor, and a program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps in a text-enhanced temporal prediction method based on a large model word embedding space as described in Embodiment 1.
[0104] Example 5 Embodiment 5 of the present invention provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps in the text enhancement temporal prediction method based on a large model word embedding space as described in Embodiment 1.
[0105] The steps and methods involved in the apparatuses of Embodiments 2, 3, 4, and 5 above correspond to those in Embodiment 1. For specific implementation details, please refer to the relevant description section of Embodiment 1. The term "computer-readable storage medium" should be understood as a single medium or multiple media including one or more instruction sets; it should also be understood as including any medium capable of storing, encoding, or carrying an instruction set for execution by a processor and enabling the processor to perform any of the methods in this invention.
[0106] Those skilled in the art will understand that the modules or steps of the present invention described above can be implemented using general-purpose computer devices. Optionally, they can be implemented using computer-executable program code, thereby allowing them to be stored in a storage device for execution by a computer device, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. The present invention is not limited to any particular combination of hardware and software.
[0107] While the specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.
Claims
1. A text-enhanced temporal prediction method based on a large model word embedding space, characterized in that, include: Acquire historical load time-series data and perform normalization processing; Normalized historical load time series data is input into the time series sharding reprogramming module, and the time series sharding embedding vector is obtained through linear projection and layer normalization. By utilizing the word embedding matrix of a frozen large language model, the temporal slice embedding vectors are reprogrammed into temporal semantic vectors of the same dimension as the text through multi-head cross attention; Unstructured text data corresponding to historical load time series data is obtained and input into the text encoding module. The frozen large language model is used for encoding to obtain word-level text semantic vectors. Using temporal semantic vectors as queries and textual semantic vectors as keys and values, a gated cross-attention fusion module is used to perform gated cross-attention calculation to obtain deep fusion features. The deep fusion features are input into the prediction head module, and the final prediction result is output.
2. The text-enhanced temporal prediction method based on a large model word embedding space as described in claim 1, characterized in that, The process of obtaining time-series piecewise embedding vectors from normalized historical load time-series data through linear projection and layer normalization includes: The normalized historical load time-series data is divided into multiple time-series fragments using a sliding window. Each temporal slice is mapped to a hidden dimension through a learnable linear projection layer, and then normalized to become a slice embedding vector.
3. The text-enhanced temporal prediction method based on a large model word embedding space as described in claim 1, characterized in that, By utilizing the word embedding matrix of a frozen large language model, the temporal slice embedding vectors are reprogrammed into temporal semantic vectors of the same dimension as the text through multi-head cross-attention, including: A candidate word embedding set is constructed by selecting multiple representative word vectors from the word embedding matrix; Using the segmented embedding vector as the query and the candidate word embedding set as the key and value, cross-attention calculation is performed to obtain a continuous semantic vector aligned with the word embedding space of the large language model.
4. The text-enhanced temporal prediction method based on a large model word embedding space as described in claim 1, characterized in that, Unstructured text data is encoded using a frozen large language model to obtain word-level text semantic vectors, including: The unstructured text data is segmented into word sequences using a large language model's word segmenter, and then padded and truncated on the right. The processed word sequence is fed into the Transformer module of the model for layer-by-layer processing, and the hidden state output of the last layer is extracted. After filtering the output hidden state through attention masking, dimensionality reduction through linear projection layer, and layer normalization to stabilize the distribution, a word-level semantic representation of the text is obtained.
5. The text-enhanced temporal prediction method based on a large model word embedding space as described in claim 1, characterized in that, The process of calculating gated cross-attention using a gated cross-attention fusion module to obtain deep fusion features includes: Using the temporal semantic vector as the query matrix and the word-level text semantic vector as both the key and value matrix, multi-head cross-attention calculation is performed to obtain the initial text enhancement vector. The initial text enhancement vector is input into the gated sublayer, and the gated network independently calculates a scalar gate coefficient for each temporal slice vector. The attention output is weighted and fused based on the acquired gating coefficients to obtain deep fusion features.
6. The text-enhanced temporal prediction method based on a large model word embedding space as described in claim 1, characterized in that, The process of outputting the final prediction result using the prediction head module includes: The deep fusion features are flattened into a one-dimensional vector along the piecewise dimension, and then the dimension is gradually compressed through two fully connected networks and mapped to the output of the prediction length dimension to obtain the predicted value. The final prediction result is obtained after inverse normalization of the predicted values.
7. A text-enhanced temporal prediction system based on a large model word embedding space, characterized in that, include: The data acquisition and normalization module is configured to: acquire historical load time-series data and perform normalization processing; The time-series sharding reprogramming module is configured to: accept normalized historical load time-series data as input, and obtain time-series sharding embedding vectors through linear projection and layer normalization; By utilizing the word embedding matrix of a frozen large language model, the temporal slice embedding vectors are reprogrammed into temporal semantic vectors of the same dimension as the text through multi-head cross attention; The text encoding module is configured to: acquire unstructured text data corresponding to historical load time series data, encode the unstructured text data using a frozen large language model, and obtain word-level text semantic vectors; The gated cross-attention fusion module is configured to use temporal semantic vectors as queries and textual semantic vectors as keys and values to calculate deep fusion features through gated cross-attention. The prediction head module is configured to output the final prediction result based on the deep fusion features.
8. A computer-readable storage medium having a program stored thereon, characterized in that, When executed by a processor, the program implements the steps in the text-enhanced temporal prediction method based on a large model word embedding space as described in any one of claims 1-6.
9. An electronic device comprising a memory, a processor, and a program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps in the text-enhanced temporal prediction method based on a large model word embedding space as described in any one of claims 1-6.
10. A computer program product comprising a computer program / instructions, characterized in that, When executed by a processor, the computer program / instruction implements the steps in the text-enhanced temporal prediction method based on a large model word embedding space as described in any one of claims 1-6.
Citation Information
Patent Citations
Word embedding vector extraction method and system based on large language model
CN120218061A
Remote sensing visual question and answer method based on large language model and multi-level attention mechanism
CN121542456A
Multi-turn human-machine conversation method and apparatus based on time-sequence feature screening encoding module
US12014149B1
