Time series prediction method and device based on multi-scale and multi-mode expert mixing
By employing a multi-scale, multi-modal expert hybrid approach, combining large language models and visual models to extract features and perform cross-modal fusion, the problem of low time series prediction accuracy in existing technologies is solved, achieving higher prediction accuracy.
Patent Information
- Application Number
- CN202511519400.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-10-23
- Publication Date
- 2026-02-27
AI Technical Summary
Existing time series prediction methods based on large language models and visual mask autoencoders suffer from low accuracy in long-term time series prediction, failing to meet the time series prediction needs of domain applications.
A multi-scale, multi-modal expert hybrid approach is adopted. The original time series data is downsampled, and temporal semantic features and temporal image features are extracted by combining a large language model and a visual model. Cross-modal feature fusion is performed using a multi-modal expert hybrid model, and finally weighted calculation is performed to improve prediction accuracy.
By effectively utilizing feature information, reducing information loss, ensuring timeliness, and improving the accuracy of time-series prediction of business data, the accuracy of time-series prediction has been improved.
Smart Images
Figure CN121579876A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of artificial intelligence technology, and in particular to a time series prediction method and apparatus based on multi-scale, multimodal expert hybridization. Background Technology
[0002] Long-Term Time Series Forecasting (LTSF) is an important but highly challenging task with wide applications in demand planning, energy load forecasting, and climate modeling. Although deep learning-based LTSF methods are emerging, limitations in the number of learnable parameters and insufficient training data scale restrict the performance improvement and robustness enhancement of these methods.
[0003] To alleviate the aforementioned limitations, existing technologies have proposed pre-trained base model methods. These methods, leveraging the massive knowledge of pre-trained models, mainly fall into two categories: The first is based on Large Language Models (LLMs): utilizing the robust knowledge, complex reasoning, and pattern recognition capabilities gained by LLMs during text pre-training, motivated by the similar sequential representation of time series and text. While these methods perform well, LLMs deal with discrete tokens, while time series are inherently continuous. The second is based on visual mask autoencoders (VAEs): using VAE pre-trained models to construct the base model for time series prediction, reconstructing the prediction task into an image patch-level reconstruction task, based on the similarity between time series and natural images. However, converting time series to image format may lead to information loss and disrupt the temporal dependencies crucial for LTSF. Both of these methods ultimately result in low accuracy in time series prediction, impacting the prediction process in domain applications and failing to meet the time series prediction needs of application domains. Summary of the Invention
[0004] This invention provides a time series prediction method and apparatus based on multi-scale and multimodal expert hybridization, which addresses the shortcomings of existing technologies such as large language models and visual mask autoencoders in long-term time series prediction.
[0005] This invention provides a time series prediction method based on multi-scale, multi-modal expert hybrid methods, comprising: The original time series data of the business data to be predicted is downsampled to obtain multi-scale time series data. The large language model and the visual model are invoked to extract features from the multi-scale time series data to obtain the corresponding temporal semantic features and temporal image features. A multimodal expert hybrid model is invoked to perform cross-modal feature fusion processing on the temporal semantic features and the temporal image features to obtain first time series prediction data based on text semantics and second time series prediction data based on image vision; The first time series prediction data and the second time series prediction data are weighted and calculated to obtain the time series prediction business data corresponding to the business data to be predicted.
[0006] In some embodiments, downsampling the original time series data of the business data to be predicted to obtain multi-scale time series data includes: The original time series data of the business data to be predicted is standardized, and the standardized time series data is divided into multiple sequence segments according to the time period. The sequence fragments are stacked and processed into a two-dimensional matrix; The two-dimensional matrix is subjected to mean pooling processing according to different preset step sizes to obtain multi-scale time series data.
[0007] In some embodiments, the temporal semantic features are extracted in the following manner: By invoking linear projectors of different scales in the large language model, semantic mapping processing is performed on the multi-scale time series data to obtain the corresponding temporal semantic features; The temporal image features are extracted in the following manner: The multi-scale time series data is rendered into a three-channel grayscale image; Determine the size of the visible image patch when the visual model processes pre-trained image patches, and resample the size of the three-channel grayscale image to the size of the visible image patch; The visual model is invoked to extract image features from the resampled three-channel grayscale image to obtain temporal image features.
[0008] In some embodiments, determining the visible image patch size when the visual model processes pre-trained image patches includes: The number of targets in the visible image patch processed by the visual model is calculated based on the ratio of the context length of the three-channel grayscale image to the prediction length of the multi-scale time series data. The number of segments and the target size of the pre-trained image blocks are obtained when the visual model processes them. The number of targets and the size of the targets are multiplied to obtain the size of the visible image patch when the visual model processes the pre-trained image patch.
[0009] In some embodiments, the step of invoking a multimodal expert hybrid model to perform cross-modal feature fusion processing on the temporal semantic features and the temporal image features to obtain first time series prediction data based on text semantics and second time series prediction data based on image vision includes: The temporal semantic features and the temporal image features are subjected to multi-scale splicing processing to obtain spliced features; The concatenated features are divided into h input tokens and input into the attention mechanism network included in the multimodal expert hybrid model for attention calculation, resulting in h corresponding output tokens, where h is a positive integer, and the attention network includes 2h router attention heads; Determine the routing score for each of the router attention heads; The output tokens are weighted and summed based on the routing scores to obtain the output features of the text modality and the visual modality, respectively. The output features are decoded to obtain first time series prediction data based on text semantics and second time series prediction data based on image vision.
[0010] In some embodiments, the 2h router attention heads include two shared attention heads consisting of k router attention heads, where k is a positive integer and not greater than h. Determining the routing score for each router attention head includes: When both h and k are 1, the routing score of the router's attention head is determined to be 1; When h is not 1, the activation function in the shared attention head is called to map the input token to obtain the routing score corresponding to the shared attention head, and the routing score of the router attention head that is not the shared attention head is determined to be 0.
[0011] The present invention also provides a time series prediction device based on multi-scale, multi-modal expert hybrid, comprising: The downsampling module is used to downsample the original time series data of the business data to be predicted, so as to obtain multi-scale time series data. The feature extraction module is used to call the large language model and the visual model to extract features from the multi-scale time series data, so as to obtain the corresponding temporal semantic features and temporal image features. The feature fusion module is used to call a multimodal expert hybrid model to perform cross-modal feature fusion processing on the temporal semantic features and the temporal image features to obtain first time series prediction data based on text semantics and second time series prediction data based on image vision. The time series prediction module performs weighted calculations on the first time series prediction data and the second time series prediction data to obtain the time series prediction business data corresponding to the business data to be predicted.
[0012] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the time series prediction method based on multi-scale multimodal expert hybrid as described above.
[0013] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the time series prediction method based on multi-scale multimodal expert hybrid as described above.
[0014] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the time series prediction method based on multi-scale multimodal expert hybrid as described above.
[0015] The present invention provides a time series prediction method and apparatus based on multi-scale multimodal expert hybrid model. It utilizes both large language model and visual model to extract multimodal features from time series data of business data. Furthermore, it uses a multimodal expert hybrid model to perform cross-modal feature fusion processing of temporal semantic features and temporal image features, ensuring full utilization of feature information and reducing feature loss. This ensures the temporality of business data in the prediction process and improves the accuracy of time series prediction of business data. Attached Figure Description
[0016] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced one by one below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.
[0017] Figure 1 This is a flowchart illustrating the time series prediction method based on multi-scale, multi-modal expert hybrid provided by the present invention.
[0018] Figure 2 This is a schematic diagram of the time series prediction method based on multi-scale and multi-modal expert hybrid provided by the present invention.
[0019] Figure 3 This is a schematic diagram of the time series prediction device based on multi-scale multimodal expert hybrid provided by the present invention.
[0020] Figure 4This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation
[0021] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.
[0022] The following description, in conjunction with the accompanying drawings, illustrates the time series prediction method and apparatus based on multi-scale, multi-modal expert hybrid technology of the present invention. Figure 1 This is a flowchart illustrating the time series prediction method based on multi-scale, multi-modal expert hybrid provided by the present invention, as shown below. Figure 1 As shown, the method includes the following steps 101 to 102, which are described in detail below.
[0023] Step 101: Downsample the original time series data of the business data to be predicted to obtain multi-scale time series data.
[0024] The business data to be predicted refers to the specific business data that requires time-series forecasting. This data involves specific target scenarios, such as weather, environment, vehicle traffic, power load, and equipment status. The corresponding raw time-series data represents the target scenario over a historical period or at a historical point in time; in other words, the target scenario data is time-series. Specifically, this includes meteorological data, temperature and humidity data, traffic flow data, power load data, and equipment status data.
[0025] Time series forecasting predicts data for a target scenario over a future time period or at a future point in time. Taking power load as an example, the original time series data for the business data to be predicted is the distribution of power equipment load over a historical 24-hour period. Time series forecasting then predicts the distribution of power equipment load over the next 6 hours based on this. Similarly, for temperature data, the original time series data for the business data to be predicted is meteorological data (temperature values) over a historical 12-hour period. Time series forecasting then predicts the meteorological data (temperature values) over the next 6 hours based on this. This time series data is typically displayed using a coordinate graph, with the horizontal axis representing the time point and the vertical axis representing the specific business data value for the target scenario.
[0026] Since time series data is presented as a coordinate graph, the downsampling process here is to effectively capture the temporal patterns of the original time series data at different resolutions. For example... Figure 2As shown, downsampling can yield time series data at different scales, which facilitates better feature extraction and processing in the future.
[0027] In some embodiments, downsampling is performed on the original time series data of the business data to be predicted to obtain multi-scale time series data. This can be achieved in the following ways, which are described in detail below.
[0028] The original time series data of the business data to be predicted is standardized, and the standardized time series data is divided into multiple sequence segments according to the time period.
[0029] The sequence fragments are stacked and processed into a two-dimensional matrix.
[0030] By performing mean pooling on the two-dimensional matrix with gradually increasing step sizes, multi-scale time series data can be obtained.
[0031] Standardizing time series data here is to facilitate knowledge transfer. The standardized value is calculated by taking the mean and variance of the data. The standardized time series data is denoted as... L represents the context length, which is the length of the historical time period in the original time series data. Further, the standardized time series data is divided into multiple sequence segments according to a time period, which is preset and denoted as P, for example, 2 hours or 2 minutes per period, thus obtaining L / P sequence segments.
[0032] Next, these sequence fragments are stacked and processed into a two-dimensional matrix, denoted as . P and Q represent the two-dimensional dimensions of the matrix. Then, mean pooling is performed on the two-dimensional matrix according to different preset step sizes to obtain multi-scale time series data. The different preset step sizes are denoted as P and Q. Where i represents the step index, , where h represents the upper limit of the scale.
[0033] Based on different step sizes Step-mean pooling can be performed on each pair of two-dimensional matrices to obtain multiple downsampled time series of different scales, i.e., multi-scale time series data, denoted as . .
[0034] In this embodiment of the invention, by downsampling the original time series data of the business data to be predicted, it is beneficial to fully explore the differentiated patterns of the time series at different sampling scales (such as micro-level fine-grained fluctuations and macro-level trends), and effectively capture the short-term details and long-term patterns of the time series.
[0035] Step 102: Call the large language model and the visual model to extract features from the multi-scale time series data to obtain the corresponding temporal semantic features and temporal image features.
[0036] For the downsampled multi-scale time-series data, the processing involves two different modalities: a large model branch and a visual branch. The large model branch belongs to the text semantic branch, while the visual branch belongs to the image branch. In the large model branch, a large language model is used to extract features from the multi-scale time-series data, obtaining the corresponding temporal semantic features. In the visual branch, a visual model is used to extract features from the multi-scale time-series data, obtaining the corresponding temporal image features.
[0037] In some embodiments, in the large model branch, temporal semantic features are extracted as follows: By calling linear projectors of different scales in the large language model, semantic mapping is performed on multi-scale time series data to obtain the corresponding temporal semantic features.
[0038] Scale-specific linear projectors are deployed in the large language model. It can process multi-scale time series data Embedding is performed, and then encoded and mapped to the semantic space of a large language model, with linear projectors at each scale. Applying to time series data at the corresponding scale Linear projector Essentially, it is a projection matrix. , S represents the data dimension processed by the large language model, as follows: (1) Where P represents multi-scale time series data The time period is divided, where i represents the multi-scale time series data. In mean pooling, the step size index, h, represents the upper limit of the scale.
[0039] like Figure 2 As shown, the temporal semantic features obtained by each linear projector of the large language model are denoted as... Specifically, it is expressed as , … The calculation formula is as follows: (2) in, This represents the input multi-scale time series data. This represents a linear projector, specifically a projection matrix. This indicates the bias term.
[0040] In the visual branch, temporal image features are extracted in the following way: Multi-scale time series data is rendered into a three-channel grayscale image.
[0041] The visible image patch size is determined when the visual model processes pre-trained image patches, and the size of the three-channel grayscale image is resampled to the visible image patch size.
[0042] The visual model is invoked to extract image features from the resampled three-channel grayscale image to obtain time-series image features.
[0043] Here, since the multi-scale time series data is one-dimensional, while the visual model is pre-trained and processes three-dimensional image data, a dimensionality difference exists. Therefore, a dimensionality alignment strategy is used to align the one-dimensional time series data to meet the three-dimensional input requirements of the pre-trained visual model. This can be achieved by first aligning the multi-scale time series data... Perform normalization processing, and then process the normalized time series data. The rendering process is performed as a three-channel grayscale image, denoted as... ,in Q represents the dimension of the image data processed by the visual model.
[0044] Furthermore, the size of the visible image patch is determined when the visual model processes pre-trained image patches. Specifically, the visual model is a Visual Mask Autoencoder (Visual MAE) used for mask prediction, and its input consists of the visible image portion and the mask image portion. Here, the visible image portion is a three-channel grayscale image. However, the invisible masked image portion needs to be predicted. But due to the three-channel grayscale image... The visible image patch size does not match the size of the image patch when the visual model processes pre-trained image patches, so image size alignment is required.
[0045] In some embodiments, determining the size of the visible image patch when the visual model processes pre-trained image patches can be achieved in the following ways, as detailed below.
[0046] First, based on the ratio of the context length of the three-channel grayscale image to the prediction length of the multi-scale time series data, the number of visible image patches that the visual model can process is calculated, denoted as n, and is specifically expressed by the following formula: (3) Where c represents the number of channels, This refers to the context length of the three-channel grayscale image. L represents the context length of the original time series data after standardization. N represents the number of two-dimensional image blocks used by the visual model. H represents the prediction length of the multi-scale time series data, which is the time period for which time series prediction is needed.
[0047] Next, we obtain the number of image patches the visual model divides into and the target size of the pre-trained image patches. Generally, we assume the total number of two-dimensional image patches used by the visual model is... The size of each image patch is Therefore, it can be seen that the number of visible image blocks is set to N×n, while the number of mask image blocks is set to N×(Nn).
[0048] Finally, the number of targets and their size are multiplied to obtain the size of the mask image patch in the visual model. Here, the number of targets N×n and the size of the image patch are multiplied together. Performing a multiplication operation yields the size of the visible image patch when the visual model processes pre-trained image patches, denoted as . .
[0049] In this embodiment of the invention, by determining the size of the visible image patch when the visual model processes pre-trained image patches, it is beneficial to convert time series data into image data for modeling, which enables the time series data to be effectively matched with the visual model without causing loss of feature information.
[0050] Next, the three-channel grayscale image is resampled to the size of the visible image patch. This can be done using bilinear interpolation to resample the three-channel grayscale image from... Resampling This completes the adaptation of time series data to visual models.
[0051] After size resampling, the final step is to use a visual model to extract image features from the resampled three-channel grayscale image, obtaining time-series image features. Specifically, the resampled three-channel grayscale images at each scale are used as the visible image portion for mask prediction, while the image portion of the predicted length H of the multi-scale time series data is used as the invisible mask image portion, which is then input into the visual model for image prediction. Figure 2 As shown, the visual model ultimately outputs the corresponding temporal image features, denoted as... Specifically, it is expressed as , … .
[0052] In this embodiment of the invention, when extracting features from time series data, two branches, a large language model and a visual model, are designed to achieve multimodal feature extraction of semantics and images respectively. In semantic feature extraction, time series data at each scale are processed one by one by linear projectors at each scale, achieving accurate modeling of time series features and reducing information loss. Furthermore, by adapting the size of the three-channel grayscale image to the visual model, the time series data can be effectively converted into images for modeling without destroying the inherent temporal dependencies of the time series and reducing information loss introduced by the conversion.
[0053] Step 103: Call the multimodal expert hybrid model to perform cross-modal feature fusion processing on temporal semantic features and temporal image features to obtain the first time series prediction data based on text semantics and the second time series prediction data based on image vision.
[0054] Here, this embodiment of the invention designs a multimodal expert hybrid model, which is a large attention mechanism network. The attention mechanism network includes multiple attention heads, which are specially divided by routing to effectively realize the interaction and fusion of feature information.
[0055] In some embodiments, a multimodal expert hybrid model is invoked to perform cross-modal feature fusion processing on temporal semantic features and temporal image features to obtain first time series prediction data based on text semantics and second time series prediction data based on image vision. This can be achieved in the following ways, which are explained in detail below.
[0056] First, the temporal semantic features and temporal image features are spliced together at multiple scales to obtain spliced features.
[0057] like Figure 2 As shown, the temporal semantic features output by the large model branch and the visual model branch and time-series image features Feature interaction will be performed, and multi-scale features will be concatenated from both. This unified concatenated representation allows the subsequent attention mechanism network to capture feature information from both modalities simultaneously. The concatenated feature representation is X, and the formula is as follows: (4) Where t represents the t-th input token in the subsequent partitioning of the splicing features, i represents the i-th attention head in the splicing feature input text modality-specific network and visual modality-specific network, and h represents the number of scales.
[0058] Next, the concatenated feature X is divided into h input tokens and fed into the attention mechanism network included in the multimodal expert hybrid model for attention calculation, resulting in h corresponding output tokens.
[0059] like Figure 2 As shown, the concatenated feature X is divided into two paths through routing. One path is the text modality, which is divided into h input tokens. Represented as , … The other path is the visual modality, divided into h input tokens. Represented as , … The partitioning here is based on scale, with one input token assigned to each scale. Both input tokens are fed into the attention mechanism network. Here, h is a positive integer, representing the number of scales. The attention network consists of 2h router attention heads, capable of performing attention mechanism calculations on the two h input tokens, resulting in h corresponding output tokens in the text and visual modalities, represented as... , The formula is as follows: (5) Where X represents the concatenation feature, and the routing of one text modality is represented as follows: The routing representation of the other visual modality is as follows: , , , These represent the projection matrices of the query vector, key vector, and value vector in the attention header of each router, respectively. The function representing attention calculation.
[0060] When outputting the attention mechanism network, it is also necessary to determine the routing score of each router's attention head.
[0061] Specifically, the 2h router attention heads in the attention mechanism network include two shared attention heads, each consisting of k router attention heads. Here, k is a positive integer, not greater than h, and its value is preset, typically 1. Thus, the 2h router attention heads include two shared attention heads. The shared attention heads are the active router attention heads, used for feature fusion and information interaction calculations on the input token with concatenated features. The other router attention heads are inactive and do not participate in feature fusion and information interaction calculations.
[0062] like Figure 2 As shown, input tokens for text and visual modalities and input token The input is the shared attention head of the attention mechanism network, which will perform feature fusion and information interaction calculations. The remaining input tokens are input into other router attention heads and do not participate in feature fusion and information interaction calculations.
[0063] Regarding the routing score of the router's attention head, this embodiment of the invention proposes a routing strategy to determine the routing score, for the t-th input token of two modal routing inputs. and Routing score Represented as: (6) in, This represents two modality types: text modality and visual modality, corresponding to the two routes of feature segmentation. Representing modes The routing score of the attention head of the i-th router. This indicates that the router's attention header is a shared attention header. This represents the projection matrix of the router's attention head. When the router's attention head is a shared attention head, it represents the projection matrix of the shared attention head. The activation function for the shared attention head.
[0064] In the above formula (6), i=1 indicates that there is only one router attention head in the attention mechanism network, and this router attention head is a shared attention head, that is, h and k are both 1. Therefore, the routing score of the router attention head is... The value is determined to be 1.
[0065] and This indicates that the number of router attention heads in the attention mechanism network is greater than 1, but not greater than the number of scales h, and that the router attention heads include k shared attention heads. Therefore, when h is not 1, the activation function in the shared attention head is called to map the input token, obtaining the routing score corresponding to the shared attention head. Specifically, this is done through the softmax activation function and projection matrix in the shared attention head. For the t-th input token and Feature fusion and information exchange operations are performed to obtain the corresponding routing score. .
[0066] The remaining cases are those involving non-shared attention heads, which are router attention heads that are not inactive and do not participate in feature fusion and information exchange operations. Just take 0.
[0067] In general, the attention mechanism calculation is only performed when the router attention head of the attention mechanism network is activated, and the corresponding routing score is calculated. That is, the routing score is calculated only by sharing the attention head, and other router attention heads are not included, and the routing score is 0.
[0068] The present invention implements a method for calculating routing scores by designing a shared attention head, which can dynamically balance the weights of router attention heads in an attention mechanism network.
[0069] During the output process of the attention mechanism network, the output tokens are weighted and summed based on the routing scores to obtain the output features for both the text and visual modalities. Weighting effectively outputs the attention calculation results at the corresponding scales, while summing integrates the attention calculations at each scale. The output features for the text and visual modalities are denoted as follows: as well as The formula is expressed as: (7) (8) in, and and represent the output tokens obtained after the attention mechanism calculation, where i represents the index of the attention header of the router that generated this output token. and represents the routing scores computed by the shared attention head in the text and visual modalities, respectively, where h represents the number of scales. and The projection matrix corresponding to the shared attention head is used to perform mapping calculations on the output tokens and output the attention calculation results at the corresponding scale.
[0070] Finally, the output features are decoded to obtain the first time series prediction data based on text semantics and the second time series prediction data based on image vision.
[0071] like Figure 2 As shown, the output features of the text modality and visual modality, generated by the attention mechanism network of the multimodal expert hybrid model, need to be further decoded to restore them to the format of time series data. The decoded output features of the text modality produce first-time-series prediction data based on text semantics, while the decoded output features of the visual modality produce second-time-series prediction data based on image vision. Specifically, during time series prediction, the original time series data of the business data to be predicted is standardized, and this standardization is also required after prediction, thus enabling time series prediction in both modalities.
[0072] In this embodiment of the invention, at various scales, the routing mechanism of a multimodal expert hybrid model is used to perform cross-modal feature fusion processing on temporal semantic features and temporal image features. This ensures the full utilization of feature information and reduces the loss of feature information, ensures the temporality of business data in the prediction process, and improves the temporal prediction accuracy of business data.
[0073] Step 104: Perform weighted calculations on the first time series forecast data and the second time series forecast data to obtain the time series forecast business data corresponding to the business data to be predicted.
[0074] like Figure 2 As shown, the first and second time-series prediction data obtained through decoding need to be further fused to combine the temporal prediction results of the two modalities. This is done by setting corresponding text weights and visual weights for weighted calculation. Generally, the text modality and the visual modality have the same importance, so their weights are both set to 0.5.
[0075] By weighting the first and second time-series prediction data using text weights and visual weights respectively, and then fusing the weighted results, the time-series prediction business data corresponding to the business data to be predicted can be obtained. For example, if the original time-series data of the business data to be predicted is the distribution value of power equipment load over a historical 24-hour period, then the time-series prediction business data is the distribution value of power equipment load over the next 6 hours. Or, if the original time-series data of the business data to be predicted is meteorological data (temperature values) over a historical 12-hour period, then the time-series prediction business data is the meteorological data (temperature values) predicted for the next 6 hours based on this data.
[0076] In this embodiment of the invention, a large language model and a visual model are used simultaneously to extract multimodal features from time-series business data. Furthermore, a multimodal expert hybrid model is used to perform cross-modal feature fusion processing of temporal semantic features and temporal image features, ensuring full utilization of feature information and reducing feature loss. This ensures the temporality of business data in the prediction process and improves the accuracy of time-series prediction of business data.
[0077] To verify the technical effectiveness of the above embodiments, this invention also includes verification experiments, which are evaluated on four commonly used long-term time series forecasting benchmark datasets: ETTh1, ETTh2, Electricity Consumption (ECL), and Traffic Flow. The ETTh1 and ETTh2 datasets record the oil temperature, winding temperature, and load parameters of a 110kV transformer in a power grid. ETTh1 contains standard operating condition data, while ETTh2 includes more fault condition noise and is mainly used for power equipment condition prediction and preventative maintenance. The ECL dataset is collected from hourly electricity load data of 321 industrial and commercial users in a power grid, covering residential, commercial, and industrial electricity consumption patterns. It exhibits significant spatial heterogeneity and is commonly used for electricity market bidding and distribution network optimization. The Traffic Flow dataset comes from real-time traffic flow monitoring data from 860 sensors on California highways, including congestion index, vehicle speed, and other derived features, and annotates accident time periods. It is mainly used for intelligent traffic management and road maintenance planning.
[0078] Here, the time series length L is set to 672, and four different prediction lengths H are used for testing, including 96, 192, 336, and 720 time points. The method of this invention, namely the time series prediction method based on multi-scale multimodal expert hybrid proposed in this embodiment, is compared with other benchmark models based on different methodologies, such as VisionTS based on Visual MAE mask autoencoders, which uses the knowledge of a pre-trained visual model for time series prediction; GPT4TS and Time-LLM based on large language models, which utilize the inference capabilities of LLM and apply them to the field of time series prediction through fine-tuning strategies; FEDformer based on frequency domain augmented Transformers, which captures global dependencies through Fourier transforms; DLinear using linear mappings, which verifies the basic laws of time series prediction with a minimalist architecture; and TimesNet based on multi-periodic convolutions. The evaluation metrics for the experimental comparisons are Mean Absolute Error (MAE) and Mean Squared Error (MSE). The experimental results of each model are shown in the table below: Table 1:
[0079] As shown in Table 1 above, the MAE and MSE of the method of the present invention are relatively lower than those of other benchmark models based on different methodologies on various datasets, which also proves that the time series prediction performance of the method of the present invention is higher than that of other method models on real datasets.
[0080] The following describes the time series prediction device based on multi-scale multimodal expert mixture provided by the present invention. The time series prediction device based on multi-scale multimodal expert mixture described below can be referred to in correspondence with the time series prediction method based on multi-scale multimodal expert mixture described above.
[0081] like Figure 3 As shown, the time series prediction device based on multi-scale, multimodal expert hybrid model specifically includes: a downsampling module 301, a feature extraction module 302, a feature fusion module 303, and a time series prediction module 304. Specifically, the downsampling module 301 is used to downsample the original time series data of the business data to be predicted to obtain multi-scale time series data; the feature extraction module 302 is used to call a large language model and a visual model to extract features from the multi-scale time series data to obtain corresponding time-series semantic features and time-series image features; the feature fusion module 303 is used to call a multimodal expert hybrid model to perform cross-modal feature fusion processing on the time-series semantic features and the time-series image features to obtain first time series prediction data based on text semantics and second time series prediction data based on image vision; the time series prediction module 304 performs weighted calculation on the first time series prediction data and the second time series prediction data to obtain the time series prediction business data corresponding to the business data to be predicted.
[0082] It should be noted that the beneficial effects of the time series prediction device based on multi-scale multimodal expert hybrid technology mentioned above correspond to each other, so the beneficial effects of the time series prediction device based on multi-scale multimodal expert hybrid technology will not be repeated here.
[0083] Figure 4 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 4As shown, the electronic device may include: a processor 410, a communication interface 420, a memory 430, and a communication bus 440, wherein the processor 410, the communication interface 420, and the memory 430 communicate with each other through the communication bus 440. The processor 410 can call logical instructions in the memory 430 to execute a time series prediction method based on multi-scale multimodal expert hybrid model. The method includes: downsampling the original time series data of the business data to be predicted to obtain multi-scale time series data; calling a large language model and a visual model to extract features from the multi-scale time series data to obtain corresponding temporal semantic features and temporal image features; calling a multimodal expert hybrid model to perform cross-modal feature fusion processing on the temporal semantic features and the temporal image features to obtain first time series prediction data based on text semantics and second time series prediction data based on image vision; and weighting the first time series prediction data and the second time series prediction data to obtain the time series prediction business data corresponding to the business data to be predicted.
[0084] Furthermore, the logical instructions in the aforementioned memory 430 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0085] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the time series prediction method based on multi-scale multimodal expert hybrid provided by the above methods. The method includes: downsampling the original time series data of the business data to be predicted to obtain multi-scale time series data; calling a large language model and a visual model to extract features from the multi-scale time series data to obtain corresponding temporal semantic features and temporal image features; calling a multimodal expert hybrid model to perform cross-modal feature fusion processing on the temporal semantic features and the temporal image features to obtain first time series prediction data based on text semantics and second time series prediction data based on image vision; and weighting the first time series prediction data and the second time series prediction data to obtain the time series prediction business data corresponding to the business data to be predicted.
[0086] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements the time series prediction method based on multi-scale multimodal expert hybrid provided by the above methods. The method includes: downsampling the original time series data of the business data to be predicted to obtain multi-scale time series data; calling a large language model and a visual model to extract features from the multi-scale time series data to obtain corresponding temporal semantic features and temporal image features; calling a multimodal expert hybrid model to perform cross-modal feature fusion processing on the temporal semantic features and the temporal image features to obtain first time series prediction data based on text semantics and second time series prediction data based on image vision; and weighting the first time series prediction data and the second time series prediction data to obtain the time series prediction business data corresponding to the business data to be predicted.
[0087] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.
[0088] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.
[0089] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A time series forecasting method based on multi-scale multi-modal expert mixing, characterized in that, include: The original time series data of the business data to be predicted is downsampled to obtain multi-scale time series data. The large language model and the visual model are invoked to extract features from the multi-scale time series data to obtain the corresponding temporal semantic features and temporal image features. A multimodal expert hybrid model is invoked to perform cross-modal feature fusion processing on the temporal semantic features and the temporal image features to obtain first time series prediction data based on text semantics and second time series prediction data based on image vision; The first time series prediction data and the second time series prediction data are weighted and calculated to obtain the time series prediction business data corresponding to the business data to be predicted.
2. The time series prediction method based on multi-scale, multi-modal expert hybrid method according to claim 1, characterized in that, The process of downsampling the original time series data of the business data to be predicted to obtain multi-scale time series data includes: The original time series data of the business data to be predicted is standardized, and the standardized time series data is divided into multiple sequence segments according to the time period. The sequence fragments are stacked and processed into a two-dimensional matrix; The two-dimensional matrix is subjected to mean pooling processing according to different preset step sizes to obtain multi-scale time series data.
3. The time series prediction method based on multi-scale, multi-modal expert hybrid method according to claim 1, characterized in that, The temporal semantic features are extracted in the following manner: By invoking linear projectors of different scales in the large language model, semantic mapping processing is performed on the multi-scale time series data to obtain the corresponding temporal semantic features; The temporal image features are extracted in the following manner: The multi-scale time series data is rendered into a three-channel grayscale image; Determine the size of the visible image patch when the visual model processes pre-trained image patches, and resample the size of the three-channel grayscale image to the size of the visible image patch; The visual model is invoked to extract image features from the resampled three-channel grayscale image to obtain temporal image features.
4. The time series prediction method based on multi-scale, multi-modal expert hybrid method according to claim 3, characterized in that, Determining the visible image patch size when the visual model processes pre-trained image patches includes: The number of targets in the visible image patch processed by the visual model is calculated based on the ratio of the context length of the three-channel grayscale image to the prediction length of the multi-scale time series data. The number of segments and the target size of the pre-trained image blocks are obtained when the visual model processes them. The number of targets and the size of the targets are multiplied to obtain the size of the visible image patch when the visual model processes the pre-trained image patch.
5. The time series prediction method based on multi-scale, multi-modal expert hybrid method according to claim 3, characterized in that, The process involves invoking a multimodal expert hybrid model to perform cross-modal feature fusion processing on the temporal semantic features and the temporal image features, resulting in first time-series prediction data based on text semantics and second time-series prediction data based on image vision, including: The temporal semantic features and the temporal image features are subjected to multi-scale splicing processing to obtain spliced features; The concatenated features are divided into h input tokens and input into the attention mechanism network included in the multimodal expert hybrid model for attention calculation, resulting in h corresponding output tokens, where h is a positive integer, and the attention network includes 2h router attention heads; Determine the routing score for each of the router attention heads; The output tokens are weighted and summed based on the routing scores to obtain the output features of the text modality and the visual modality, respectively. The output features are decoded to obtain first time series prediction data based on text semantics and second time series prediction data based on image vision.
6. The time series prediction method based on multi-scale, multi-modal expert hybridization according to claim 5, characterized in that, The 2h router attention heads include two shared attention heads consisting of k router attention heads, where k is a positive integer and not greater than h. Determining the routing score for each router attention head includes: When both h and k are 1, the routing score of the router's attention head is determined to be 1; When h is not 1, the activation function in the shared attention head is called to map the input token to obtain the routing score corresponding to the shared attention head, and the routing score of the router attention head that is not the shared attention head is determined to be 0.
7. A time series prediction device based on multi-scale, multi-modal expert hybrid methods, characterized in that, include: The downsampling module is used to downsample the original time series data of the business data to be predicted, so as to obtain multi-scale time series data. The feature extraction module is used to call the large language model and the visual model to extract features from the multi-scale time series data, so as to obtain the corresponding temporal semantic features and temporal image features. The feature fusion module is used to call a multimodal expert hybrid model to perform cross-modal feature fusion processing on the temporal semantic features and the temporal image features to obtain first time series prediction data based on text semantics and second time series prediction data based on image vision. The time series prediction module performs weighted calculations on the first time series prediction data and the second time series prediction data to obtain the time series prediction business data corresponding to the business data to be predicted.
8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the time series prediction method based on multi-scale multimodal expert hybrid as described in any one of claims 1 to 6.
9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the time series prediction method based on multi-scale multimodal expert hybrid as described in any one of claims 1 to 6.
10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the time series prediction method based on multi-scale multimodal expert hybrid as described in any one of claims 1 to 6.