Energy consumption prediction method and system based on multi-modal large model and context perception

Abstract

The application discloses a kind of based on multimodal big model and context perception's energy consumption prediction method and system, the method first acquires the multimodal time series data of energy system and constructs structured natural language prompt, corresponding text prompt word vector sequence is generated by big language model;Then environmental sensor sequence is normalized and patch division, and the fusion environment sequence is obtained by calculating correlation;Correlation calculation is carried out to text prompt word vector by using learnable routing prototype, fusion environment sequence and historical energy consumption sequence by mutual attention mechanism, and big language model word sparse is carried out;Finally, according to the fusion input sequence, two-way prediction results are obtained, and the final energy consumption prediction result is obtained after fusion.The present application can solve the entanglement problem when time series and text information are fused, realize the effective fusion of multimodal context information, and improve the accuracy of energy consumption prediction.

Description

Technical Field

[0001] This invention relates to the fields of energy consumption prediction and artificial intelligence technology, and in particular to an energy consumption prediction method and system based on multimodal large models and context-awareness. Background Technology

[0002] Traditional energy consumption forecasting tasks typically rely on historical energy consumption time series data from a single mode for modeling and prediction. While these methods have made some progress in modeling stable load patterns and short-term trends, they exhibit significant limitations in real-world, complex environments. Real-world energy consumption changes are determined not only by their own historical behavior but also by the combined influence of various heterogeneous information, including external environmental factors (such as temperature, humidity, wind speed, and solar radiation) as well as internal structural configurations, operating strategies, and user behavior habits. However, traditional models often neglect these non-energy-consuming modal intervention variables in their design, resulting in insufficient modeling of key driving factors and an inability to comprehensively characterize the dynamic evolution of the energy consumption system.

[0003] Most existing energy consumption prediction models still rely on statistical modeling methods (such as ARIMA and SARIMA) or shallow machine learning methods (such as support vector regression and random forests). These methods have advantages in handling linear patterns or short-term trends, but their performance is limited when modeling complex nonlinear relationships, long-term dependency patterns, and multivariate interaction effects. In recent years, the introduction of deep learning technologies, especially recurrent neural networks (RNNs), long short-term memory networks (LSTMs), convolutional neural networks (CNNs), and the Transformer series of models, has demonstrated superior modeling capabilities in time series prediction tasks. They can effectively uncover potential temporal dependency structures and maintain modeling accuracy in high-dimensional, multivariate scenarios. However, most of these models only model the energy consumption sequence itself and struggle to integrate external modal data such as environmental perception, structural information, or expert knowledge, thus remaining insufficient in generalization and robustness in complex scenarios. Especially when deployed in real-world environments, these models are prone to overfitting or prediction distortion.

[0004] More complexly, the dynamic changes in energy consumption systems are often jointly influenced by a large number of heterogeneous data sources. This information includes not only structural parameters (such as floor layout and equipment deployment) and operational settings (such as control strategies and air conditioning load allocation), but also user behavior habits (such as usage frequency and control preferences) and seasonal climate background. This information mostly exists in unstructured forms, such as operating instructions, user documents, and expert annotations, which traditional numerical modeling methods struggle to directly process. In recent years, Large Language Models (LLMs), represented by BERT, GPT, and LLaMA, have achieved significant breakthroughs in natural language processing tasks. Pre-trained on large-scale text corpora, they possess powerful semantic understanding, text embedding, and reasoning capabilities. This provides unprecedented possibilities for introducing unstructured text information into energy consumption modeling, laying the foundation for multimodal perception and reasoning mechanisms.

[0005] Although some studies have begun to incorporate multimodal information into energy consumption prediction models, such as jointly modeling meteorological time series and energy consumption sequences, or introducing textual knowledge to enhance model understanding, these methods still face several challenges. Firstly, modal fusion processes are often simplistically designed, frequently employing crude temporal and textual embedding concatenation methods. They lack deep alignment mechanisms for the semantic structures of different modalities, easily introducing information redundancy or semantic conflicts, leading to information entanglement between modalities and affecting downstream prediction performance. Secondly, textual information itself is characterized by high noise, significant redundancy, and weak domain adaptability, lacking mechanisms to identify and filter semantic units truly relevant to the prediction target. Furthermore, the dense word space constructed by general-purpose large language models is not entirely suitable for energy consumption scenarios. Models are often forced to perform matching in large-scale non-domain-related semantic spaces, resulting in decreased prediction accuracy, increased computational overhead, and a lack of efficient and interpretable semantic constraint mechanisms.

[0006] More importantly, current energy consumption prediction models are context-independent, often simply stacking external data as additional features without a mechanism to achieve true context awareness and multimodal semantic fusion. This design limits the model's ability to model the relationship between external semantic variables and energy consumption changes, making it difficult to capture energy consumption drivers in complex environments. Summary of the Invention

[0007] Purpose of the invention: The purpose of this invention is to provide an energy consumption prediction method and system based on a multimodal large model and context-awareness, which solves the problems of difficulty in integrating external environmental data and textual knowledge, difficulty in modal alignment, and limited prediction accuracy.

[0008] Technical solution: The energy consumption prediction method based on a multimodal large model and context awareness described in this invention includes the following steps:

[0009] Step 1: Collect multimodal time-series data of the energy system within a preset time period. The multimodal time-series data includes environmental sensor sequences, historical energy consumption sequences, structural features, and expert knowledge text.

[0010] Step 2: Construct structured natural language prompts based on the multimodal time series data, input the structured natural language prompts into a large language model for semantic encoding, and generate corresponding text prompt word vector sequences;

[0011] Step 3: Normalize the environmental sensor sequence and divide it into patches according to a preset window length and sliding step size to obtain multiple environmental sequence patches; use the environmental sequence patches as queries and the text prompt word vector sequence as keys and values, and calculate the correlation between the environmental sequence patches and the text prompt word vector sequence through a mutual attention mechanism to obtain a fused environmental sequence;

[0012] Step 4: Using the learnable routing prototype, the fusion environment sequence, and the historical energy consumption sequence, the relevance calculation of the text prompt word vectors is performed through a mutual attention mechanism to select a subset of key words related to the energy consumption prediction task and perform word domain sparsification of the large language model.

[0013] Step 5: Calculate the correlation between the fused environment sequence and the historical energy consumption sequence through a mutual attention mechanism to obtain a fused input sequence; input the fused input sequence into a context-independent prediction model and a large language model with word domain sparsification, respectively, to obtain a first prediction result and a second prediction result; fuse the first prediction result and the second prediction result through a residual connection and weighted fusion mechanism to obtain the final energy consumption prediction result.

[0014] Furthermore, step 4 specifically includes:

[0015] Initialize several learnable routing prototypes, which are used to characterize semantic filtering directions in the energy consumption prediction task;

[0016] The learnable routing prototype and the text prompt word vector sequence are input into a mutual attention mechanism to calculate the first correlation between the learnable routing prototype and each word vector in the text prompt word vector sequence, and an initial key vocabulary subset is obtained based on the first correlation.

[0017] Using the fusion environment sequence and the historical energy consumption sequence as queries, and the word vectors corresponding to the initial key vocabulary subset as keys and values, a mutual attention mechanism is used to calculate the second relevance. Based on the second relevance, the key vocabulary subset is obtained by filtering from the initial key vocabulary subset.

[0018] Furthermore, the construction of structured natural language prompts based on multimodal time-series data in step 2 includes:

[0019] Statistical features are extracted from environmental sensor sequences, and structured natural language prompts are constructed using predefined natural language prompt templates, combined with system structure information and expert knowledge. The statistical features include load mean, fluctuation amplitude, and slope of change trend.

[0020] Furthermore, in step 1, the environmental sensor sequence is the monitoring data of indoor and outdoor environmental sensors over a period of time; the historical energy consumption sequence is the energy consumption data of each sub-energy system over a period of time; and the structural features and expert knowledge text includes text data of building floor layout, equipment deployment plan, operation strategy description and energy efficiency assessment report.

[0021] Furthermore, step 3 includes normalizing the environmental sensor sequence before patching the environmental sensor sequence.

[0022] Furthermore, in step 5, the context-independent prediction model is the SPAformer model; in steps 2 and 5, the large language model is the LLaMA model or the BERT model.

[0023] The energy consumption prediction system based on a multimodal large model and context awareness described in this invention includes:

[0024] A multimodal time-series data acquisition unit is used to acquire multimodal time-series data of an energy system within a preset time period. The multimodal time-series data includes environmental sensor sequences, historical energy consumption sequences, structural features, and expert knowledge text.

[0025] The word vector generation unit is used to construct structured natural language prompts based on the multimodal time series data, input the structured natural language prompts into a large language model for semantic encoding, and generate corresponding text prompt word vector sequences.

[0026] An environment sequence fusion unit is used to normalize the environment sensor sequence and divide it into patches according to a preset window length and sliding step size to obtain multiple environment sequence patches; using the environment sequence patches as queries and the text prompt word vector sequence as keys and values, the correlation between the environment sequence patches and the text prompt word vector sequence is calculated through a mutual attention mechanism to obtain a fused environment sequence;

[0027] The key vocabulary filtering unit is used to calculate the relevance of the text prompt word vectors by using the learnable routing prototype, the fusion environment sequence and the historical energy consumption sequence through a mutual attention mechanism, and to filter out a subset of key vocabulary related to the energy consumption prediction task, thereby sparsifying the word domain of the large language model.

[0028] The energy consumption prediction unit is used to calculate the correlation between the fused environment sequence and the historical energy consumption sequence through a mutual attention mechanism to obtain a fused input sequence; input the fused input sequence into a context-independent prediction model and a large language model after word domain sparsification to obtain a first prediction result and a second prediction result; and fuse the first prediction result and the second prediction result through a residual connection and a weighted fusion mechanism to obtain a final energy consumption prediction result.

[0029] The electronic device of the present invention includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is loaded onto the processor, it implements the energy consumption prediction method based on a multimodal large model and context awareness.

[0030] The computer-readable storage medium of the present invention stores a computer program, which, when executed by a processor, implements the energy consumption prediction method based on a multimodal large model and context awareness.

[0031] The computer program product of the present invention includes a computer program that, when executed by a processor, implements the energy consumption prediction method based on a multimodal large model and context awareness.

[0032] Beneficial effects: Compared with the prior art, the advantages of the present invention are: Beneficial effects: Compared with the prior art, the advantages of the present invention are: (1) The present invention introduces multimodal semantic information and introduces a large language model to process unstructured text information, which enhances the model's comprehensive understanding of energy consumption influencing factors; (2) The present invention designs a modal alignment mechanism, which effectively solves the entanglement problem when time sequence and text information are fused, realizes the effective fusion and interpretation of multimodal context information, and improves prediction efficiency and domain adaptability; (3) The present invention performs two-stage correlation screening through learnable routing prototypes and current time sequence context, reduces irrelevant semantic information and redundant computation in the text prompt word vector sequence generated by the large language model, and improves the accuracy of energy consumption prediction results. Attached Figure Description

[0033] Figure 1 This is a schematic diagram of the overall process of the method of the present invention.

[0034] Figure 2 This is a schematic diagram illustrating the generation of multimodal data and structured natural language prompts according to the present invention.

[0035] Figure 3 This is a schematic diagram illustrating the modal alignment between the environmental sequence patch and the text prompt word vector of the present invention.

[0036] Figure 4This is a schematic diagram of the two-stage key word selection based on a learnable routing prototype according to the present invention.

[0037] Figure 5 This is a schematic diagram illustrating the multimodal context fusion and prediction result fusion of the present invention. Detailed Implementation

[0038] The technical solution of the present invention will be further described below with reference to the accompanying drawings.

[0039] like Figure 1 As shown, the energy consumption prediction method based on multimodal large model and context-awareness includes the following steps.

[0040] Step 1, Multimodal data acquisition.

[0041] like Figure 2 As shown, historical energy consumption sequences, environmental sensor time-series data, and text descriptions such as structural features and expert knowledge are collected to form multimodal metadata required for energy consumption prediction.

[0042] Step 2: Text prompt generation.

[0043] like Figure 2 As shown, based on each environmental sensor sequence and its statistical characteristics, combined with system structure information and expert knowledge, a structured natural language prompt is constructed. This prompt is then input into a large language model for semantic encoding, generating a corresponding text prompt word vector sequence. This text prompt word vector sequence is used for subsequent multimodal alignment, key word selection, and fusion.

[0044] Step 3: Time series data partitioning, patching, and modal alignment.

[0045] like Figure 3 As shown, the environment sequence is normalized and patched, with each patch serving as a query and the text prompt word vector sequence as a key and value. A mutual attention mechanism is used to calculate the correlation between the environment sequence patches and the text prompt word vector sequence, resulting in a fused multimodal environment representation. .

[0046] Specifically, suppose that the environmental sequence patches form a patch matrix after linear projection. The text prompt word vector sequence forms a matrix ;in, For the number of environment sequence patches, The number of text units obtained after word segmentation for structured natural language prompts. To unify feature dimensions, For environment sequence patch index, For text cell indexing. The Behavior No. An environment sequence patch dimensional vector, The Behavior No. Each text unit 3D word vector, denoted as The text prompt's word vector sequence serves as both key and value, referring to the same matrix. The key matrix is ​​obtained by using different trainable linear mappings. Sum matrix The query matrix is ​​obtained by linear mapping from the patch matrix. . These are the query, key, and value linear mapping matrices specifically for step 3, independent of the projection matrices in subsequent steps, as shown in equation (1). Relevance weight matrix Calculated according to equation (2), where the element Indicates the first The first environment sequence patch and the first The degree of correlation between individual text units This represents the Softmax function, which normalizes the data according to the text unit dimension corresponding to each query. The semantic enhancement matrix is ​​then obtained according to equation (3). and will With the original patch matrix After summing the residuals, layer normalization is performed to obtain the fused environment sequence. ; The representation layer normalization function.

[0047] (1)

[0048] (2)

[0049] (3)

[0050] Step 4: Sparsification of word domains in large language models.

[0051] like Figure 4 As shown, the learnable routing prototype and the text prompt word vector sequence undergo a first-stage relevance calculation through a mutual attention mechanism, and an initial subset of key words is obtained based on the relevance results. The initial subset of key terms includes terms related to building energy consumption forecasting, load changes, environmental changes, operating strategies, or equipment status.

[0052] The learnable routing prototype is a semantic filtering parameter updated during model training, used to extract semantic units related to the energy consumption prediction task from the text prompt word vector sequence. The learnable routing prototype is represented by the trainable parameter matrix shown in equation (4). ,in, For the number of routing prototypes, For the route prototype index, The Travelogue This corresponds to a potential semantic direction for energy consumption prediction. The routing prototype can be initialized randomly or based on the mean of word vectors from the energy consumption domain, and updated together with other network parameters during model training through backpropagation of the energy consumption prediction loss. During the first stage of selection, [the following will be considered]... As a query, As keys and values; These are the query and key linear mapping matrices specifically for the first stage of filtering, and are related to the matrix in equation (1). Different. The first-stage attention matrix is ​​obtained according to equation (5). Its elements Indicates the first The first routing prototype and the first The relevance of each text unit. In equation (6) For the first The maximum first relevance score of each text unit across all routing prototypes. The number of text units retained for the first stage satisfies ; This indicates the return of the highest score. The original set of indices for each text unit, therefore This is the initial set of key vocabulary indexes. Alternatively, it can be retained... Not lower than the preset threshold The text units. The "vocabulary" and "word vector" are kept in a one-to-one correspondence through word segmentation index, and the index and its corresponding word vector are retained during filtering.

[0053] (4)

[0054] (5)

[0055] (6)

[0056] Furthermore, the fusion environment sequence The historical energy consumption sequence is used as the temporal context representation of the current prediction task, and a second-stage correlation calculation is performed between the initial key vocabulary subset and the key vocabulary subset through a mutual attention mechanism, so as to further filter out the key vocabulary subset related to the current input data from the initial key vocabulary subset. By employing the two-stage screening process described above, irrelevant semantic information and redundant computation in the text prompt word vector sequences generated by the large language model can be reduced, enabling subsequent prediction branches of the large language model to generate prediction results based on semantic representations that are more relevant to the current energy consumption prediction task.

[0057] The temporal context representation of the current prediction task refers to the environmental change characteristics and historical energy consumption change characteristics corresponding to the time window to be predicted. This involves fusing the environmental sequence. Aligned with the historical energy consumption sequence obtained through patching and linear projection in the time dimension; the aligned fused environment matrix is ​​denoted as... Historical energy consumption is represented as ,in, The number of context time positions after alignment. For context time location index. When At that time, interpolation, pooling, or resampling are used to... Convert to These two matrices represent different processing stages. In equation (7), This indicates splicing along the feature dimension. To construct the projection matrix for the context, This is the context matrix. During the second stage of screening, [the following will be used]: As a query, the initial key word index set Corresponding word vector submatrix As keys and values; These are the query and key linear mapping matrices specifically for the second-stage filtering. Equation (8) yields the second-stage attention matrix. ,element Indicates the first The context time position and the original index are as follows: The correlation between candidate text units. In Equation (9) Candidate text units The highest second relevance score across all contextual time locations. The quantity reserved for the second phase must meet the following requirements. ; This is the final set of key vocabulary indexes. Alternatively, it can be retained... Not lower than the preset threshold Text units.

[0058] (7)

[0059] (8)

[0060] (9)

[0061] Step 5: Context fusion and energy consumption prediction.

[0062] like Figure 5 As shown, the multimodal environment is characterized after time alignment. Compared with historical energy consumption The common input mutual attention module is merged into a unified fused input sequence. . Use the same fused input sequence Input a context-free prediction model (such as SPAformer) and the final key vocabulary index set, respectively. The large language model prediction branch, which completes word domain sparsity for corresponding semantic units, yields the first and second prediction results, respectively. (Step 4 yields...) Semantic units used to restrict the participation of prediction branches in the computation of large language models, and serving as semantic conditions for those branches, do not replace the fused input sequence. Finally, the two outputs are integrated through residual connection and weighted fusion mechanism to form the energy consumption prediction value.

[0063] Specifically, with As a query, As keys and values, the context attention weight matrix is ​​calculated according to equation (10). And obtain the fused input sequence according to equation (11). . These are the query, key, and value linear mapping matrices specifically for step 5, and are related to the aforementioned... , and All are independent of each other. Context-free prediction function right Perform time series modeling to obtain the first scalar prediction result. For each , For matrix The original index is Text unit vectors, In order to be in The second relevance score The pooling weights are obtained by performing Softmax normalization and satisfying... ; This is a semantic convergence vector for key vocabulary. (After...) The prediction function of a large language model that completes word domain sparsity for corresponding semantic units is denoted as . , its To predict input, Based on semantic conditions, the scalar second prediction result is obtained. Specifically, as shown in equation (12). In equation (13), For the final prediction result, The fusion weights for the two predictions, These are the scalar trainable parameters that are updated during training. For the Sigmoid function. In equation (14), This represents the number of samples in the training batch. For sample index, and The first The predicted and actual values ​​of each sample. This is the mean squared error prediction loss. This loss is used to update the mutual attention parameters, routing prototype, projection matrices, and prediction head parameters.

[0064] (10)

[0065] (11)

[0066] (12)

[0067] (13)

[0068] (14)

[0069] The method described in this invention will be verified through two specific experiments below.

[0070] Experiment 1: Using a super high-rise office building with a total construction area of ​​approximately 120,000 square meters as the subject, the energy consumption prediction method based on multimodal large model and context awareness of the present invention was deployed and implemented.

[0071] (1.1) Historical energy consumption sequences were collected through the Building Energy Management Platform (BEMS), including energy consumption data from major subsystems such as the central air conditioning system, elevator group control system, lighting system, and fresh air system. The data sampling frequency was 15 minutes, and the historical observation window length was set to 30 days. At the same time, indoor and outdoor environmental parameters, including temperature, humidity, wind speed, wind direction, CO2 concentration, PM2.5 index, and solar radiation intensity, were collected in real time. Combining the structural features in text form, such as building floor layout, equipment deployment plan, operation strategy description, and energy efficiency assessment report, with expert knowledge documents, a multimodal metadata was formed that includes historical energy consumption sequences, environmental sensor time-series data, and text descriptions of structural features and expert knowledge.

[0072] (1.2) Automatically generate text prompts based on predefined natural language prompt templates. Specifically, this includes: firstly, extracting statistical features from environmental sensor time-series data, including indicators such as the average load, fluctuation range, and slope of change trends for each subsystem; secondly, combining system structure information and expert knowledge to generate structured natural language prompts such as "The load fluctuation of the office area on floors 10 to 30 has exceeded 30% in the past 24 hours" or "The outside temperature drops sharply by more than 5°C at night, which may cause a sudden change in the load of the HVAC system." The prompt is then input into a large language model, and semantic encoding is performed using the LLaMA-7B model to generate a corresponding text prompt word vector sequence for subsequent multimodal alignment, key word selection, and fusion.

[0073] (1.3) Temporal Data Partitioning and Modal Alignment. The normalized environmental sensor temporal data is partitioned into patches. Specifically, each environmental variable sequence is divided into fixed-length patches using a sliding window operation with a set window length and sliding step size to preserve its local temporal dynamics. Each patch is embedded as a fixed-dimensional vector through linear projection or convolutional coding modules and used as a query vector input to the subsequent modal alignment module. Simultaneously, the text prompt word vector sequence generated by the large language model in step (1.2) is used as the key and value, and input into the mutual attention mechanism module. By calculating the correlation between the patch vector and the text prompt word vector, the weighted representation of the environmental sensor patch in the semantic space is obtained, thereby achieving the alignment of the environmental modality and the text modality, resulting in a fused multimodal environmental sequence representation.

[0074] (1.4) Initialize multiple learnable routing prototypes, each representing the semantic filtering direction in the energy consumption prediction task. Input the learnable routing prototypes and the text prompt word vector sequence generated by the large language model into the mutual attention mechanism to perform the first-stage relevance calculation, filtering out an initial key vocabulary subset related to building energy consumption prediction, such as "peak load", "night cooling", "occupancy rate", etc. Based on this, perform the second-stage relevance calculation. Specifically, the fused environmental patch sequence and historical energy consumption sequence in step (1.3) are respectively divided into patches and uniformly embedded into fixed-length vector representations. Using these patches as queries, the word vectors corresponding to the initial key vocabulary subset selected in the first stage are used as keys and values, and the mutual attention mechanism is used to calculate the second relevance to further identify semantic units strongly related to the current input, retaining only the key vocabulary subset most relevant to the current prediction task.

[0075] (1.5) The modally aligned multimodal environment representation and the historical energy consumption sequence are fused using a mutual attention mechanism to obtain a fused input sequence. The same fused input sequence is then input into the context-independent prediction model SPAformer and the large language model prediction branch, which has undergone domain sparsification using a subset of key words. The SPAformer backbone uses a spectrum-patch attention mechanism to model long-term temporal dependencies; the large language model prediction branch uses the fused input sequence as the prediction input and outputs the energy consumption prediction result with the semantic convergence vector corresponding to the subset of key words as the condition. Finally, the two outputs are fused using a weighted average and residual connection mechanism to obtain the final energy consumption prediction result.

[0076] Experiment 2: Taking a regional energy system as the object, the platform needs to perform unified scheduling and multi-step rolling forecasting of the comprehensive energy demand of multiple building groups. The regional energy system integrates combined cooling, heating and power equipment, ground source heat pumps, photovoltaic systems, energy storage units, load aggregation controllers and energy efficiency management centers, and connects to the regional dispatch center's operation instructions, weather forecast data, energy consumption data of each building, equipment operation logs and user behavior feedback information.

[0077] (2.1) Collect multimodal data of the regional energy system.

[0078] (2.2) Based on the statistical characteristics of the load sequence, and combined with the scheduling strategy log and equipment operation description, standardized text prompts are automatically generated. For example, prompts such as "The current system is in peak shaving mode, and the load is expected to be transferred to the energy storage unit" and "Some office buildings are in energy-saving operation mode, and the air conditioning load has decreased by 10%" are combined with meteorological descriptions such as "There will be a sharp drop in temperature tonight" to construct composite natural language prompts. The prompts are input into a large language model (such as LLaMA or BERT) for semantic encoding to generate corresponding text prompt word vector sequences for subsequent multimodal alignment, key word selection and fusion.

[0079] (2.3) The historical load sequence and meteorological data are divided into time blocks (Patch) according to hourly frequency, and frequency domain smoothing is performed by Fourier transform to retain the main periodic components to reduce high-frequency noise interference. Each patch is embedded and positionally encoded and then input into a Transformer-based encoder. The system uses the load patch as the query and the text prompt word vector sequence as the key and value. The system achieves modal alignment between policy semantics and load changes through mutual attention mechanism to obtain the fused multimodal environment representation.

[0080] (2.4) Initialize multiple learnable routing prototypes. Use the learnable routing prototypes and the text prompt word vector sequence generated by the large language model to perform the first-stage correlation calculation. Adaptively extract the initial key vocabulary subset closely related to power strategy regulation, such as "demand response", "valley filling", "pre-cooling", "peakshaving", etc. Then, combine the current input load sequence and meteorological data to perform the second-stage correlation calculation. Again, filter and retain only the key vocabulary subset most relevant to the current prediction task to reduce irrelevant information and computational load.

[0081] (2.5) The modally aligned multimodal environment representation and historical load sequence are fused into a unified fusion input sequence through a mutual attention mechanism. This fusion input sequence is then input into the SPAformer backbone structure and the large language model prediction branch, which has undergone domain sparsification using a subset of key words. The large language model prediction branch is also conditional on the semantic convergence vector corresponding to the subset of key words. The final energy load prediction result is obtained by fusing the two outputs, and a historical error estimation submodule is introduced through a residual connection mechanism to dynamically correct potential cumulative biases.

[0082] The energy consumption prediction system based on a multimodal large model and context awareness described in this invention includes:

[0083] A multimodal time-series data acquisition unit is used to acquire multimodal time-series data of an energy system within a preset time period. The multimodal time-series data includes environmental sensor sequences, historical energy consumption sequences, structural features, and expert knowledge text.

[0084] The word vector generation unit is used to construct structured natural language prompts based on the multimodal time series data, input the structured natural language prompts into a large language model for semantic encoding, and generate corresponding text prompt word vector sequences.

[0085] An environment sequence fusion unit is used to normalize the environment sensor sequence and divide it into patches according to a preset window length and sliding step size to obtain multiple environment sequence patches; using the environment sequence patches as queries and the text prompt word vector sequence as keys and values, the correlation between the environment sequence patches and the text prompt word vector sequence is calculated through a mutual attention mechanism to obtain a fused environment sequence;

[0086] The key vocabulary filtering unit is used to calculate the relevance of the text prompt word vectors by using the learnable routing prototype, the fusion environment sequence and the historical energy consumption sequence through a mutual attention mechanism, and to filter out a subset of key vocabulary related to the energy consumption prediction task, thereby sparsifying the word domain of the large language model.

[0087] The energy consumption prediction unit is used to calculate the correlation between the fused environment sequence and the historical energy consumption sequence through a mutual attention mechanism to obtain a fused input sequence; input the fused input sequence into a context-independent prediction model and a large language model after word domain sparsification to obtain a first prediction result and a second prediction result; and fuse the first prediction result and the second prediction result through a residual connection and a weighted fusion mechanism to obtain a final energy consumption prediction result.

[0088] The computer-readable storage medium may include RAM, ROM, EEPROM, CD-ROM or other optical disc storage devices, magnetic disk storage devices or other magnetic storage devices, flash memory, or any other media that can be used to store program code in the form of instructions or data structures and is accessible by a computer.

[0089] The processor is used to execute a computer program stored in memory to implement the various steps in the methods described in the above embodiments.

Claims

1. An energy consumption prediction method based on a multimodal large model and context-awareness, characterized in that, Includes the following steps: Step 1: Collect multimodal time-series data of the energy system within a preset time period. The multimodal time-series data includes environmental sensor sequences, historical energy consumption sequences, structural features, and expert knowledge text. Step 2: Construct structured natural language prompts based on the multimodal time series data, input the structured natural language prompts into a large language model for semantic encoding, and generate corresponding text prompt word vector sequences; Step 3: Normalize the environmental sensor sequence and divide it into patches according to a preset window length and sliding step size to obtain multiple environmental sequence patches; use the environmental sequence patches as queries and the text prompt word vector sequence as keys and values, and calculate the correlation between the environmental sequence patches and the text prompt word vector sequence through a mutual attention mechanism to obtain a fused environmental sequence; Step 4: Using the learnable routing prototype, the fusion environment sequence, and the historical energy consumption sequence, the relevance calculation of the text prompt word vectors is performed through a mutual attention mechanism to select a subset of key words related to the energy consumption prediction task and to sparsify the word domain of the large language model. Step 5: Calculate the correlation between the fused environment sequence and the historical energy consumption sequence through a mutual attention mechanism to obtain a fused input sequence; input the fused input sequence into a context-independent prediction model and a large language model with word domain sparsification, respectively, to obtain a first prediction result and a second prediction result; fuse the first prediction result and the second prediction result through a residual connection and weighted fusion mechanism to obtain the final energy consumption prediction result.

2. The energy consumption prediction method based on a multimodal large model and context-awareness as described in claim 1, characterized in that, Step 4 specifically includes: Initialize several learnable routing prototypes, which are used to characterize semantic filtering directions in the energy consumption prediction task; The learnable routing prototype and the text prompt word vector sequence are input into a mutual attention mechanism to calculate the first correlation between the learnable routing prototype and each word vector in the text prompt word vector sequence, and an initial key vocabulary subset is obtained based on the first correlation. Using the fusion environment sequence and the historical energy consumption sequence as queries, and the word vectors corresponding to the initial key vocabulary subset as keys and values, a mutual attention mechanism is used to calculate the second relevance. Based on the second relevance, the key vocabulary subset is obtained by filtering from the initial key vocabulary subset.

3. The energy consumption prediction method based on a multimodal large model and context-awareness as described in claim 1, characterized in that, Step 2, which involves constructing structured natural language prompts based on multimodal time-series data, includes: Statistical features are extracted from environmental sensor sequences, and structured natural language prompts are constructed using predefined natural language prompt templates, combined with system structure information and expert knowledge. The statistical features include load mean, fluctuation amplitude, and slope of change trend.

4. The energy consumption prediction method based on a multimodal large model and context-awareness as described in claim 1, characterized in that, In step 1, the environmental sensor sequence is the monitoring data of indoor and outdoor environmental sensors over a period of time; the historical energy consumption sequence is the energy consumption data of each sub-energy system over a period of time; the structural features and expert knowledge text include text data of building floor layout, equipment deployment plan, operation strategy description and energy efficiency assessment report.

5. The energy consumption prediction method based on a multimodal large model and context-awareness as described in claim 1, characterized in that, Step 3, before patching the environmental sensor sequence, also includes normalizing the environmental sensor sequence.

6. The energy consumption prediction method based on a multimodal large model and context-awareness according to claim 1, characterized in that, In step 5, the context-independent prediction model is the SPAformer model; in steps 2 and 5, the large language model is the LLaMA model or the BERT model.

7. An energy consumption prediction system based on a multimodal large model and context-awareness, characterized in that, include: A multimodal time-series data acquisition unit is used to acquire multimodal time-series data of an energy system within a preset time period. The multimodal time-series data includes environmental sensor sequences, historical energy consumption sequences, structural features, and expert knowledge text. The word vector generation unit is used to construct structured natural language prompts based on the multimodal time series data, input the structured natural language prompts into a large language model for semantic encoding, and generate corresponding text prompt word vector sequences. An environment sequence fusion unit is used to normalize the environment sensor sequence and divide it into patches according to a preset window length and sliding step size to obtain multiple environment sequence patches; using the environment sequence patches as queries and the text prompt word vector sequence as keys and values, the correlation between the environment sequence patches and the text prompt word vector sequence is calculated through a mutual attention mechanism to obtain a fused environment sequence; The key vocabulary filtering unit is used to calculate the relevance of the text prompt word vectors by using the learnable routing prototype, the fusion environment sequence and the historical energy consumption sequence through a mutual attention mechanism, and to filter out a subset of key vocabulary related to the energy consumption prediction task, thereby sparsifying the word domain of the large language model. The energy consumption prediction unit is used to calculate the correlation between the fused environment sequence and the historical energy consumption sequence through a mutual attention mechanism to obtain a fused input sequence; input the fused input sequence into a context-independent prediction model and a large language model after word domain sparsification to obtain a first prediction result and a second prediction result; and fuse the first prediction result and the second prediction result through a residual connection and a weighted fusion mechanism to obtain a final energy consumption prediction result.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the computer program is loaded into the processor, it implements the energy consumption prediction method based on a multimodal large model and context awareness as described in any one of claims 1-6.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the energy consumption prediction method based on a multimodal large model and context awareness as described in any one of claims 1-6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the energy consumption prediction method based on a multimodal large model and context awareness as described in any one of claims 1-6.

Images