Energy consumption prediction method and system based on multi-modal large model and context awareness
By using a multimodal large model and context-aware methods, the problems of external data fusion and modal alignment in energy consumption prediction models are solved, achieving efficient energy consumption prediction and improving prediction accuracy and adaptability.
Patent Information
- Application Number
- CN202511048582.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-07-29
- Publication Date
- 2025-11-14
AI Technical Summary
Existing energy consumption prediction models struggle to effectively integrate external environmental data with textual knowledge, suffer from modal alignment difficulties, have limited prediction accuracy, and lack context awareness.
Employing a multimodal large model and context-aware approach, this method collects multimodal time-series data, constructs structured natural language prompts, generates word vectors using a large language model, and combines learnable routing prototypes and mutual attention mechanisms to select key words relevant to the energy consumption prediction task. It then merges environmental sequences and historical energy consumption sequences, inputs them into the prediction branches of the context-independent prediction model and the large language model, and generates the final energy consumption prediction result.
It achieves effective fusion of multimodal semantic information, improves prediction accuracy and domain adaptability, solves the problem of information entanglement between modes, and enhances the understanding and prediction efficiency of energy consumption influencing factors.
Smart Images
Figure CN120951238A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of energy consumption prediction and artificial intelligence technology, and in particular to an energy consumption prediction method and system based on multimodal large models and context-awareness. Background Technology
[0002] Traditional energy consumption forecasting tasks typically rely on historical energy consumption time series data from a single mode for modeling and prediction. While these methods have made some progress in modeling stable load patterns and short-term trends, they exhibit significant limitations in complex real-world environments. Real-world energy consumption changes are determined not only by their own historical behavior but also by the combined influence of various heterogeneous information, including external environmental factors (such as temperature, humidity, wind speed, and solar radiation) as well as internal structural configurations, operating strategies, and user behavior habits. However, traditional models often neglect these non-energy-consuming modal intervention variables in their design, resulting in insufficient modeling of key driving factors and an inability to comprehensively characterize the dynamic evolution of the energy consumption system.
[0003] Most existing energy consumption prediction models still rely on statistical modeling methods (such as ARIMA and SARIMA) or shallow machine learning methods (such as support vector regression and random forests). These methods have advantages in handling linear patterns or short-term trends, but their performance is limited when modeling complex nonlinear relationships, long-term dependency patterns, and multivariate interaction effects. In recent years, the introduction of deep learning technologies, especially recurrent neural networks (RNNs), long short-term memory networks (LSTMs), convolutional neural networks (CNNs), and the Transformer series of models, has demonstrated superior modeling capabilities in time series prediction tasks. They can effectively uncover potential temporal dependency structures and maintain modeling accuracy in high-dimensional, multivariate scenarios. However, most of these models only model the energy consumption sequence itself and struggle to integrate external modal data such as environmental perception, structural information, or expert knowledge, thus remaining insufficient in generalization and robustness in complex scenarios. Especially when deployed in real-world environments, these models are prone to overfitting or prediction distortion.
[0004] More complexly, the dynamic changes in energy consumption systems are often jointly influenced by a large number of heterogeneous data sources. This information includes not only structural parameters (such as floor layout and equipment deployment) and operational settings (such as control strategies and air conditioning load allocation), but also user behavior habits (such as usage frequency and control preferences) and seasonal climate background. This information mostly exists in unstructured forms, such as operating instructions, user documents, and expert annotations, which traditional numerical modeling methods struggle to directly process. In recent years, Large Language Models (LLMs), represented by BERT, GPT, and LLaMA, have achieved significant breakthroughs in natural language processing tasks. Pre-trained on large-scale text corpora, they possess powerful semantic understanding, text embedding, and reasoning capabilities. This provides unprecedented possibilities for introducing unstructured text information into energy consumption modeling, laying the foundation for multimodal perception and reasoning mechanisms.
[0005] Although some studies have begun to incorporate multimodal information into energy consumption prediction models, such as jointly modeling meteorological time series and energy consumption sequences, or introducing textual knowledge to enhance model understanding, these methods still face several challenges. Firstly, modal fusion processes are often simplistically designed, frequently employing crude temporal and textual embedding concatenation methods. They lack deep alignment mechanisms for the semantic structures of different modalities, easily introducing information redundancy or semantic conflicts, leading to information entanglement between modalities and affecting downstream prediction performance. Secondly, textual information itself is characterized by high noise, significant redundancy, and weak domain adaptability, lacking mechanisms to identify and filter semantic units truly relevant to the prediction target. Furthermore, the dense word space constructed by general-purpose large language models is not entirely suitable for energy consumption scenarios. Models are often forced to perform matching in large-scale non-domain-related semantic spaces, resulting in decreased prediction accuracy, increased computational overhead, and a lack of efficient and interpretable semantic constraint mechanisms.
[0006] More importantly, current energy consumption prediction models are context-independent, often simply stacking external data as additional features without a mechanism to achieve true context awareness and multimodal semantic fusion. This design limits the model's ability to model the relationship between external semantic variables and energy consumption changes, making it difficult to capture energy consumption drivers in complex environments. Summary of the Invention
[0007] Purpose of the invention: The purpose of this invention is to provide an energy consumption prediction method and system based on a multimodal large model and context-awareness, which solves the problems of difficulty in integrating external environmental data and textual knowledge, difficulty in modal alignment, and limited prediction accuracy.
[0008] Technical solution: The energy consumption prediction method based on multimodal large model and context awareness described in this invention includes the following steps:
[0009] Step 1: Collect multimodal time-series data of the energy system over a period of time, including environmental sensor sequences, historical energy consumption sequences, structural features, and expert knowledge text;
[0010] Step 2: Construct structured natural language prompts based on multimodal time series data, and input the structured natural language prompts into a large language model to generate corresponding word vectors;
[0011] Step 3: Patch the environmental sensor sequence, use the patches as queries and word vectors as keys, and calculate the relevance to obtain the fused environmental sequence;
[0012] Step 4: Using learnable route prototypes, environmental sensor sequences, and historical energy consumption sequences, filter and retain a subset of key words related to the energy consumption prediction task from all word vectors in the large language model.
[0013] Step 5: Calculate the correlation between the fused environmental sequence and the historical energy consumption sequence to obtain the fused input sequence. Input the fused input sequence into the context-independent prediction model and the prediction branch of the large language model. Combine the prediction results of the two to obtain the final energy consumption prediction result.
[0014] Furthermore, step 4 specifically includes the following steps:
[0015] By using learnable routing prototypes and all word vectors of a large language model to calculate the relevance, key words related to the energy consumption prediction task are filtered and retained.
[0016] For the key words, the correlation between the key words and environmental sensor sequences and historical energy consumption sequences is calculated, and the key words related to the energy consumption prediction task are filtered and retained again to obtain a subset of key words.
[0017] Furthermore, the correlation is calculated using a mutual attention mechanism.
[0018] Furthermore, the construction of structured natural language prompts based on multimodal time-series data in step 2 includes:
[0019] Statistical features are extracted from environmental sensor sequences, and structured natural language prompts are constructed using predefined natural language prompt templates, combined with system structure information and expert knowledge. The statistical features include load mean, fluctuation amplitude, and slope of change trend.
[0020] Furthermore, in step 1, the environmental sensor sequence is the monitoring data of indoor and outdoor environmental sensors over a period of time; the historical energy consumption sequence is the energy consumption data of each sub-energy system over a period of time; and the structural features and expert knowledge text includes text data of building floor layout, equipment deployment plan, operation strategy description and energy efficiency assessment report.
[0021] Furthermore, step 3 includes normalizing the environmental sensor sequence before patching the environmental sensor sequence.
[0022] Furthermore, in step 5, the context-independent prediction model is the SPAformer model; in steps 2 and 5, the large language model is the LLaMA model or the BERT model.
[0023] The energy consumption prediction system based on a multimodal large model and context awareness described in this invention includes:
[0024] The multimodal time series data acquisition unit is used to collect multimodal time series data of the energy system over a period of time, including environmental sensor sequences, historical energy consumption sequences, structural features, and expert knowledge text.
[0025] The word vector generation unit is used to construct structured natural language prompts based on multimodal time series data, and input the structured natural language prompts into a large language model to generate corresponding word vectors.
[0026] The environmental sequence fusion unit is used to divide the environmental sensor sequence into patches, use the patches as queries and the word vectors as keys, and calculate the relevance to obtain the fused environmental sequence.
[0027] The key vocabulary filtering unit is used to filter and retain a subset of key vocabulary related to the energy consumption prediction task from all word vectors of the large language model by utilizing learnable routing prototypes, environmental sensor sequences and historical energy consumption sequences.
[0028] The energy consumption prediction unit is used to calculate the correlation between the fused environmental sequence and the historical energy consumption sequence using a mutual attention mechanism to obtain a fused input sequence. The fused input sequence is then input into the context-independent prediction model and the prediction branch of the large language model. The prediction results of the two are then fused to obtain the final energy consumption prediction result.
[0029] The electronic device of the present invention includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is loaded onto the processor, it implements the energy consumption prediction method based on a multimodal large model and context awareness.
[0030] The computer-readable storage medium of the present invention stores a computer program, which, when executed by a processor, implements the energy consumption prediction method based on a multimodal large model and context awareness.
[0031] Beneficial effects: Compared with the prior art, the advantages of the present invention are: (1) The present invention introduces multimodal semantic information and introduces a large language model to process unstructured text information, which enhances the model’s comprehensive understanding of energy consumption influencing factors; (2) The present invention designs a modal alignment mechanism, which effectively solves the entanglement problem when time sequence and text information are fused, realizes the effective fusion and interpretation of multimodal context information, and improves prediction efficiency and domain adaptability. Attached Figure Description
[0032] Figure 1 This is a flowchart of the method of the present invention. Detailed Implementation
[0033] The technical solution of the present invention will be further described below with reference to the accompanying drawings.
[0034] like Figure 1 As shown, the energy consumption prediction method based on multimodal large model and context-awareness includes the following steps.
[0035] Step 1, Multimodal data acquisition.
[0036] Collect historical energy consumption sequences, environmental sensor time-series data, and text descriptions such as structural features and expert knowledge to form multimodal metadata required for energy consumption prediction.
[0037] Step 2: Text prompt generation.
[0038] Based on each environmental sensor sequence and its statistical characteristics, combined with system structure information and expert knowledge, a structured natural language prompt is constructed. This prompt is then input into a large language model to generate a corresponding text prompt semantic embedding (word vector representation), which is used for subsequent multimodal alignment and fusion.
[0039] Step 3: Time series data partitioning, patching, and modal alignment.
[0040] The environmental sequence is normalized and divided into patches. The patches are used as queries, and the embedded text prompts are used as key / value pairs. The relevance is calculated through a mutual attention mechanism to obtain the fused multimodal environmental representation.
[0041] Step 4: Sparsification of word domains in large models.
[0042] First, learnable routing prototypes are used to select key words related to energy consumption prediction from large language models. Then, combined with the input multimodal data, the model is further simplified, retaining only the words most relevant to the prediction, reducing irrelevant information and computation.
[0043] Step 5: Context fusion and energy consumption prediction.
[0044] The modally aligned multimodal environment representation and historical energy consumption sequences are input together. Their context dependencies are modeled through a mutual attention mechanism and fused into a unified multimodal representation. This representation is then input into a context-independent prediction model (such as SPAformer) and a large language model branch to generate prediction results. Finally, the two outputs are integrated through residual connections and a weighted fusion mechanism to form the predicted energy consumption value.
[0045] The method described in this invention will be verified through two specific experiments below.
[0046] Experiment 1: Using a super high-rise office building with a total construction area of approximately 120,000 square meters as the subject, the energy consumption prediction method based on multimodal large model and context awareness of the present invention was deployed and implemented.
[0047] (1.1) Historical energy consumption sequences were collected through the Building Energy Management Platform (BEMS), including energy consumption data from major subsystems such as the central air conditioning system, elevator group control system, lighting system, and fresh air system. The data sampling frequency was 15 minutes, and the historical observation window length was set to 30 days. Simultaneously, indoor and outdoor environmental parameters, including temperature, humidity, wind speed, wind direction, CO2 concentration, PM2.5 index, and solar radiation intensity, were collected in real time. Combining structural features in text form, such as building floor layout, equipment deployment plans, operation strategy descriptions, and energy efficiency assessment reports, with expert knowledge documents, a multimodal metadata system was formed, encompassing historical energy consumption sequences, environmental sensor time-series data, and textual descriptions of structural features and expert knowledge.
[0048] (1.2) Automatically generate text prompts based on predefined natural language prompt templates. Specifically, this includes: firstly, extracting statistical features from environmental sensor time-series data, including indicators such as the average load, fluctuation range, and slope of change trends for each subsystem; secondly, combining system structure information and expert knowledge to generate structured natural language prompts such as "The load fluctuation of office areas on floors 10 to 30 has exceeded 30% in the past 24 hours" or "The outside temperature dropped sharply by more than 5°C at night, which may cause a sudden change in the load of the HVAC system." The prompt is then input into a large language model, where semantic encoding is performed using the LLaMA-7B model to generate corresponding text prompt semantic embeddings (word vector representations) for subsequent multimodal alignment and fusion.
[0049] (1.3) Temporal Data Partitioning and Modal Alignment. The normalized environmental sensor temporal data is partitioned into patches. Specifically, each environmental variable sequence is divided into fixed-length patches using a sliding window operation with a set window length and sliding step size to preserve its local temporal dynamics. Each patch is embedded as a fixed-dimensional vector through linear projection or convolutional coding, serving as a query vector input to the subsequent modal alignment module. Simultaneously, the text prompt word vectors generated by the large language model in step (1.2) are used as keys and values, and jointly input into the mutual attention mechanism module. By calculating the correlation between the patch vectors and the prompt word vectors, a weighted representation of the environmental sensor patches in the semantic space is obtained, thereby achieving the alignment of the environmental modality and the text modality, resulting in a fused multimodal environmental sequence representation.
[0050] (1.4) Initialize the learnable routing prototype and perform the first-stage relevance calculation with the word vector space of the large language model to select an initial vocabulary subset related to building energy consumption prediction, such as "peak load", "night cooling", and "occupancy rate". Based on this, perform the second-stage relevance calculation. Specifically, the fused environmental patch sequence and historical energy consumption sequence in step (1.3) are divided into patches and uniformly embedded into fixed-length vector representations. Using these patches as queries and the key word vectors selected in the first stage as keys, the relevance is calculated using a mutual attention mechanism to further identify semantic units strongly related to the current input, and only the vocabulary subset most relevant to the current prediction task is retained.
[0051] (1.5) The modally aligned multimodal environment representation and historical component energy consumption sequences are fused using a mutual attention mechanism. The fused representation is then input into a hybrid architecture jointly constructed by the context-free prediction model SPAformer and the large language model prediction branch (LLM). The SPAformer backbone utilizes the spectral-patch attention mechanism (SPA) to model long-term temporal dependencies, while the large language model prediction branch outputs energy consumption prediction results based on text prompts. Finally, the outputs of SPAformer and LLM are fused using a weighted average and residual connection mechanism to obtain the final energy consumption prediction result.
[0052] Experiment 2: Taking a regional energy system as the object, the platform needs to perform unified scheduling and multi-step rolling forecasting of the comprehensive energy demand of multiple building groups. The regional energy system integrates combined cooling, heating and power equipment, ground source heat pumps, photovoltaic systems, energy storage units, load aggregation controllers and energy efficiency management centers, and connects to the regional dispatch center's operation instructions, weather forecast data, energy consumption data of each building, equipment operation logs and user behavior feedback information.
[0053] (2.1) Collect multimodal data of the regional energy system.
[0054] (2.2) Based on the statistical characteristics of the load sequence, and combined with the scheduling strategy logs and equipment operation descriptions, standardized text prompts are automatically generated. For example, prompts such as "The current system is in peak shaving mode, and the load is expected to be transferred to the energy storage unit" and "Some office buildings are in energy-saving operation mode, and the air conditioning load has decreased by 10%" are combined with meteorological descriptions such as "There will be a sharp drop in temperature tonight" to construct composite natural language prompts. The prompts are input into a large language model (such as LLaMA or BERT) to generate corresponding text prompt semantic embeddings (word vector representations) for subsequent multimodal alignment and fusion.
[0055] (2.3) Historical load sequences and meteorological data are divided into time blocks (Patches) according to hourly frequency, and frequency domain smoothing is performed by Fourier transform to retain the main periodic components and reduce high-frequency noise interference. Each patch is embedded and location-encoded and then input into a Transformer-based encoder. The system uses the load patch as a query and the text prompt semantic embedding as a key / value pair. Through a mutual attention mechanism, modal alignment between policy semantics and load changes is achieved to obtain the fused multimodal environment representation.
[0056] (2.4) The first stage of screening is performed using a learnable routing prototype and the word vector space of a large language model to adaptively extract keywords closely related to "power strategy regulation", such as "demand response", "valleyfilling", "pre-cooling", and "peak shaving". Then, combined with the current input load sequence and meteorological data, a second screening is performed to retain only the subset of words most relevant to the prediction, thereby reducing irrelevant information and computational load.
[0057] (2.5) The modally aligned multimodal environmental characterization and historical load sequences are input into the prediction module. The SPAformer backbone structure and the Large Language Model (LLM) prediction branch are jointly modeled, and their outputs are fused as the final energy load prediction result. At the same time, a historical error estimation submodule is introduced through a residual connection mechanism to dynamically correct potential cumulative biases.
[0058] The energy consumption prediction system based on a multimodal large model and context awareness described in this invention includes:
[0059] The multimodal time series data acquisition unit is used to collect multimodal time series data of the energy system over a period of time, including environmental sensor sequences, historical energy consumption sequences, structural features, and expert knowledge text.
[0060] The word vector generation unit is used to construct structured natural language prompts based on multimodal time series data, and input the structured natural language prompts into a large language model to generate corresponding word vectors.
[0061] The environmental sequence fusion unit is used to divide the environmental sensor sequence into patches, use the patches as queries and the word vectors as keys, and calculate the relevance to obtain the fused environmental sequence.
[0062] The key vocabulary filtering unit is used to filter and retain a subset of key vocabulary related to the energy consumption prediction task from all word vectors of the large language model by utilizing learnable routing prototypes, environmental sensor sequences and historical energy consumption sequences.
[0063] The energy consumption prediction unit is used to calculate the correlation between the fused environmental sequence and the historical energy consumption sequence using a mutual attention mechanism to obtain a fused input sequence. The fused input sequence is then input into the context-independent prediction model and the prediction branch of the large language model. The prediction results of the two are then fused to obtain the final energy consumption prediction result.
[0064] The electronic device of the present invention includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is loaded onto the processor, it implements the energy consumption prediction method based on a multimodal large model and context awareness.
[0065] The computer-readable storage medium of the present invention stores a computer program, which, when executed by a processor, implements the energy consumption prediction method based on a multimodal large model and context awareness.
[0066] The computer-readable storage medium may include RAM, ROM, EEPROM, CD-ROM or other optical disc storage devices, magnetic disk storage devices or other magnetic storage devices, flash memory or any other medium that can be used to store program code in the form of instructions or data structures and is accessible by a computer.
[0067] The processor is used to execute a computer program stored in memory to implement the various steps in the methods described in the above embodiments.
Claims
1. An energy consumption prediction method based on a multimodal large model and context-awareness, characterized in that, Includes the following steps: Step 1: Collect multimodal time-series data of the energy system over a period of time, including environmental sensor sequences, historical energy consumption sequences, structural features, and expert knowledge text; Step 2: Construct structured natural language prompts based on multimodal time series data, and input the structured natural language prompts into a large language model to generate corresponding word vectors; Step 3: Patch the environmental sensor sequence, use the patches as queries and word vectors as keys, and calculate the relevance to obtain the fused environmental sequence; Step 4: Using learnable route prototypes, environmental sensor sequences, and historical energy consumption sequences, filter and retain a subset of key words related to the energy consumption prediction task from all word vectors in the large language model. Step 5: Calculate the correlation between the fused environmental sequence and the historical energy consumption sequence to obtain the fused input sequence. Input the fused input sequence into the context-independent prediction model and the prediction branch of the large language model. Combine the prediction results of the two to obtain the final energy consumption prediction result.
2. The energy consumption prediction method based on a multimodal large model and context-awareness as described in claim 1, characterized in that, Step 4 specifically includes the following steps: By using learnable routing prototypes and all word vectors of a large language model to calculate the relevance, key words related to the energy consumption prediction task are filtered and retained. For the key words, the correlation between the key words and environmental sensor sequences and historical energy consumption sequences is calculated, and the key words related to the energy consumption prediction task are filtered and retained again to obtain a subset of key words.
3. The energy consumption prediction method based on a multimodal large model and context-awareness according to claim 1 or 2, characterized in that, The correlation is calculated using the mutual attention mechanism.
4. The energy consumption prediction method based on a multimodal large model and context-awareness as described in claim 1, characterized in that, Step 2, which involves constructing structured natural language prompts based on multimodal time-series data, includes: Statistical features are extracted from environmental sensor sequences, and structured natural language prompts are constructed using predefined natural language prompt templates, combined with system structure information and expert knowledge. The statistical features include load mean, fluctuation amplitude, and slope of change trend.
5. The energy consumption prediction method based on a multimodal large model and context-awareness as described in claim 1, characterized in that, In step 1, the environmental sensor sequence is the monitoring data of indoor and outdoor environmental sensors over a period of time; the historical energy consumption sequence is the energy consumption data of each sub-energy system over a period of time; the structural features and expert knowledge text include text data of building floor layout, equipment deployment plan, operation strategy description and energy efficiency assessment report.
6. The energy consumption prediction method based on a multimodal large model and context-awareness according to claim 1, characterized in that, Step 3, before patching the environmental sensor sequence, also includes normalizing the environmental sensor sequence.
7. The energy consumption prediction method based on a multimodal large model and context-awareness according to claim 1, characterized in that, In step 5, the context-independent prediction model is the SPAformer model; in steps 2 and 5, the large language model is the LLaMA model or the BERT model.
8. An energy consumption prediction system based on a multimodal large model and context-awareness, characterized in that, include: The multimodal time series data acquisition unit is used to collect multimodal time series data of the energy system over a period of time, including environmental sensor sequences, historical energy consumption sequences, structural features, and expert knowledge text. The word vector generation unit is used to construct structured natural language prompts based on multimodal time series data, and input the structured natural language prompts into a large language model to generate corresponding word vectors. The environmental sequence fusion unit is used to divide the environmental sensor sequence into patches, use the patches as queries and the word vectors as keys, and calculate the relevance to obtain the fused environmental sequence. The key vocabulary filtering unit is used to filter and retain a subset of key vocabulary related to the energy consumption prediction task from all word vectors of the large language model by utilizing learnable routing prototypes, environmental sensor sequences and historical energy consumption sequences. The energy consumption prediction unit is used to calculate the correlation between the fused environmental sequence and the historical energy consumption sequence using a mutual attention mechanism to obtain a fused input sequence. The fused input sequence is then input into the context-independent prediction model and the prediction branch of the large language model. The prediction results of the two are then fused to obtain the final energy consumption prediction result.
9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the computer program is loaded into the processor, it implements the energy consumption prediction method based on a multimodal large model and context awareness as described in any one of claims 1-7.
10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the energy consumption prediction method based on multimodal large model and context awareness as described in any one of claims 1-7.