A large model inference method and device for multivariate time series data

Abstract

The application discloses a large model inference method and device for multivariate time series data, and solves the technical problem that the mainstream processing mode of existing multivariate time series data is difficult to integrate natural language knowledge and rule constraints into the time series data processing flow, resulting in that the traditional time series model cannot accurately utilize natural language information to realize interpretable inference and strategy generation. The method comprises the following steps: acquiring multivariate time series data, performing short-time window slicing processing on the multivariate time series data, and outputting a plurality of short-time window sequences; preprocessing the plurality of short-time window sequences to generate preprocessed time series window sequences; performing image generation operation on each preprocessed time series window sequence to output a two-dimensional multichannel time series image; after performing size unification processing on each two-dimensional multichannel time series image, inputting the two-dimensional multichannel time series image into a trainable projection module respectively to output visual semantic embedding; and pretraining a multimodal large model to perform task decision according to the embedding and a preset text prompt, and outputting a task decision result.

Description

Technical Field

[0001] This invention relates to the field of multivariate time series analysis technology, and in particular to a method and apparatus for large-scale model inference for multivariate time series data. Background Technology

[0002] With the continuous development of IoT, IoV, Industrial Internet, and smart city infrastructure, multi-source sensor data such as traffic flow, vehicle speed, queue length, equipment vibration, current and voltage, temperature and humidity are constantly being collected, forming high-frequency, multivariate time-series data. This type of data is widely used in scenarios such as traffic forecasting, equipment diagnosis, anomaly detection, condition assessment, dispatch control, and intelligent decision-making, and has become a fundamental data form in modern industrial and urban operation systems. Existing research shows that the development path of time-series analysis technology has roughly gone through the process of traditional autoregressive models, recurrent neural networks, convolutional and temporal convolutional networks, Transformer, and further expansion to large-scale and basic models.

[0003] In the aforementioned application scenarios, multivariate time-series data typically exhibit the following common characteristics: First, multiple variables are coupled at the same time point; second, the same variable exhibits temporal dependencies between adjacent time points; and third, the data often contains noise, missing data, dimensional differences, local abrupt changes, and non-stationary variations. Especially in real-time scenarios such as traffic signal control and industrial closed-loop control, systems typically receive state observations on a second-level or even shorter cycle, requiring models to complete state understanding, outcome prediction, and control decisions within a short timeframe. Therefore, improving the model's ability to model multivariate dynamic characteristics within a short time window has become a crucial technical challenge in this field.

[0004] Current mainstream approaches to processing multivariate time series data are based on direct modeling of numerical sequence space. These approaches primarily rely on algorithms such as statistical learning, recurrent neural networks, convolutional networks, or Transformers to directly encode and infer multivariate time series samples, thereby extracting trend and periodic features from the data. However, this purely numerical sequence-driven approach can only capture basic time series patterns to a limited extent. When faced with local mutations within short time windows, high-frequency fluctuations, abnormal noise interference, and complex coupling relationships between multiple variables, it generally suffers from insufficient feature representation, inadequate model robustness, and limited task generalization ability. Especially in application scenarios where structured numerical time series data needs to be linked with knowledge reasoning, language prompts, instruction generation, or control strategies, the purely numerical sequence space-based approach cannot effectively adapt to the input paradigms of large language models and multimodal large models. It also struggles to integrate natural language knowledge and rule constraints into the time series data processing flow, ultimately preventing traditional time series models from accurately utilizing natural language information to achieve interpretable reasoning and strategy generation. Summary of the Invention

[0005] This invention provides a method and apparatus for large-scale model inference for multivariate time series data, which solves the technical problem that existing mainstream processing methods for multivariate time series data are unable to integrate natural language knowledge and rule constraints into the time series data processing flow, resulting in traditional time series models being unable to accurately utilize natural language information to achieve interpretable reasoning and strategy generation.

[0006] The first aspect of this invention provides a method for large-scale model inference for multivariate time-series data, comprising:

[0007] Acquire multivariate time series data, and perform short-time window slicing on the multivariate time series data to output multiple short-time window sequences;

[0008] Multiple short-time window sequences are preprocessed to generate multiple preprocessed time-series window sequences;

[0009] Perform image generation operation on each of the preprocessed time window sequences to output multiple two-dimensional multi-channel time series images;

[0010] After the dimensions of each of the two-dimensional multi-channel temporal images are uniformly processed, they are input into the trainable projection module to output multiple visual semantic embeddings.

[0011] A pre-trained multimodal large model is used to make task decisions based on multiple visual semantic embeddings and preset text prompts, and output multiple task decision results.

[0012] Optionally, the preprocessing of the multiple short-time window sequences to generate multiple preprocessed time-series window sequences includes:

[0013] Missing value imputation and outlier pruning are performed sequentially on each of the short-time window sequences, and multiple outlier-pruned short-time window observation sequences are output.

[0014] A normalization function is used to map the short-time window observation sequences after outlier clipping, and multiple preprocessed time-series window sequences are output.

[0015] Optionally, the step of performing an image generation operation on each of the preprocessed time-series window sequences to output multiple two-dimensional multi-channel time-series images includes:

[0016] The preprocessed temporal window sequences are structured and encoded to output multiple floating-point tensors;

[0017] The dimensions of each floating-point tensor are unified to obtain multiple floating-point tensors with unified dimensions;

[0018] The floating-point tensors of the unified dimension are converted to different data types to obtain multiple floating-point tensors after data type conversion.

[0019] The dimensions of the floating-point tensors after conversion of each of the aforementioned data types are normalized to obtain multiple standardized tensors;

[0020] Each of the standardized tensors is mapped to a corresponding two-dimensional multi-channel time series image.

[0021] Optionally, after the two-dimensional multi-channel temporal images are subjected to size unification processing, they are respectively input into a trainable projection module to output multiple visual semantic embeddings, including:

[0022] The dimensions of each of the two-dimensional multi-channel time-series images are uniformized to obtain multiple time-series images with uniform dimensions.

[0023] The trainable projection module is used to extract features from the time-series images after unifying the dimensions, and outputs multiple visual semantic embeddings.

[0024] Optionally, the pre-trained multimodal large model performs task decisions based on multiple visual semantic embeddings and preset text prompts, outputting multiple task decision results, including:

[0025] The preset text prompt is semantically encoded to output a semantic representation of the text prompt;

[0026] Each of the aforementioned visual semantic embeddings is mapped and matched and aligned with the semantic representation of the text prompt, resulting in multiple sets of aligned visual semantic embeddings.

[0027] The semantic representation of the text prompt is constructed into a multimodal joint input by each group of aligned visual semantic embeddings;

[0028] Multiple sets of multimodal joint inputs are fed into the pre-trained multimodal large model to perform inference calculations and output multiple task decision results.

[0029] Optionally, the optimization process of the trainable projection module specifically includes:

[0030] Extract task feedback signals from multiple task decision results;

[0031] Based on the task feedback signals, an optimization objective is constructed;

[0032] The trainable projection module is iteratively updated with the optimization objective to output the optimized trainable projection module.

[0033] A second aspect of the present invention provides a large model inference apparatus for multivariate time series data, comprising:

[0034] The acquisition module is used to acquire multivariate time series data, perform short-time window slicing processing on the multivariate time series data, and output multiple short-time window sequences;

[0035] The preprocessing module is used to preprocess the multiple short-time window sequences to generate multiple preprocessed time-series window sequences;

[0036] The image generation module is used to perform image generation operations on each of the preprocessed time window sequences and output multiple two-dimensional multi-channel time series images;

[0037] The projection module is used to perform size unification processing on each of the two-dimensional multi-channel time-series images, and then input them into the trainable projection module to output multiple visual semantic embeddings.

[0038] The decision module is used to make task decisions based on multiple visual semantic embeddings and preset text prompts using a pre-trained multimodal large model, and output multiple task decision results.

[0039] A third aspect of the present invention provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor performs the steps of the large model inference method for multivariable time series data as described above.

[0040] The fourth aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed, implements the large model inference method for multivariable time series data as described above.

[0041] The fifth aspect of the present invention provides a computer program product comprising a computer program stored on a non-transitory computer-readable storage medium, the computer program comprising program instructions, wherein, when the program instructions are executed by a computer, the computer performs the steps of the large model inference method for multivariable time series data as described above.

[0042] As can be seen from the above technical solutions, the present invention has the following advantages:

[0043] The above-mentioned technical solution of the present invention provides a large-scale model inference method for multivariate time-series data. It acquires multivariate time-series data, performs short-time window slicing on the data, and outputs multiple short-time window sequences. It preprocesses these short-time window sequences to generate multiple preprocessed time-series window sequences. It performs image generation operations on each preprocessed time-series window sequence to output multiple two-dimensional multi-channel time-series images. After unifying the size of each two-dimensional multi-channel time-series image, it inputs them into a trainable projection module to output multiple visual semantic embeddings. A pre-trained multimodal large-scale model performs task decision-making based on the multiple visual semantic embeddings and preset text prompts, outputting multiple task decision results. Based on the above solution, the present invention, by decomposing multivariate time-series data into short-time window sequences and performing preprocessing and image conversion, transforms data that was originally difficult to integrate with natural... Numerical time-series data combined with language is transformed into an image format that can be efficiently processed by a multimodal large model. Then, a trainable projection module extracts precise visual semantic embeddings, which, combined with natural language knowledge and rule constraints in preset text prompts, achieve a deep integration of time-series data processing and natural language information. This not only improves the representativeness and understandability of short-window time-series data, allowing dynamic changes, trends, and patterns within the short-window to be captured and utilized in a more stable and intuitive way, but also enables the time-series data processing workflow to effectively integrate natural language rules and knowledge through the reasoning capabilities of the multimodal large model. This allows the model to directly perform interpretable reasoning on time-series data based on natural language prompts, outputting decision results that conform to natural language constraints. At the same time, the precise feature extraction of the trainable projection module and the efficient reasoning of the multimodal large model further ensure the accuracy of the decision results. Attached Figure Description

[0044] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0045] Figure 1 This is a flowchart illustrating the steps of a large model inference method for multivariable time series data provided in Embodiment 1 of the present invention.

[0046] Figure 2 This is a flowchart illustrating a large model inference method for multivariable time series data provided in Embodiment 1 of the present invention.

[0047] Figure 3 This is a flowchart of the model training process provided in Embodiment 1 of the present invention;

[0048] Figure 4This is a structural block diagram of a large model inference device for multivariable time series data provided in Embodiment 2 of the present invention. Detailed Implementation

[0049] This invention provides a method and apparatus for large-scale model inference for multivariate time series data, which solves the technical problem that existing mainstream processing methods for multivariate time series data are unable to integrate natural language knowledge and rule constraints into the time series data processing flow, resulting in traditional time series models being unable to accurately utilize natural language information to achieve interpretable reasoning and strategy generation.

[0050] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. It should be noted that in the optional embodiments of the present invention, the object information and other related data involved require the permission or consent of the object when the embodiments of the present invention are applied to specific products or technologies, and the collection, use, and processing of related data must comply with the relevant laws, regulations, and standards of the relevant countries and regions. That is to say, if the embodiments of the present invention involve data related to the object, it needs to be obtained with the authorization and consent of the object, the authorization and consent of the relevant departments, and in compliance with the relevant laws, regulations, and standards of the country and region. If personal information is involved in the embodiments, the acquisition of all personal information requires the consent of the individual. If sensitive information is involved, the separate consent of the information subject is required, and the embodiments also need to be implemented with the authorization and consent of the object.

[0051] Terminology Explanation:

[0052] 1. Feature rearrangement: Rearrange the variable dimension, time dimension, and feature dimension according to preset rules to maintain a consistent spatial layout across different windows. The purpose of this is to make it easier for the model to stably identify "what semantics the same position represents".

[0053] 2. Structured Coding: Encode window data into tensor representations with a fixed organizational form, so that it retains both temporal relationships and the relative structure between multiple variables.

[0054] 3. Visual Semantic Embedding: The vector representation obtained after the image passes through the projection module. It is no longer pixel-level information, but a refined "semantic visual feature" used to align and integrate with text prompts at a higher level.

[0055] 4. Multimodal Semantic Space: A unified representation space where image and text information can be expressed using similar semantic coordinate systems. The goal of this paper is to project temporal images into this space so that large models can jointly understand them with cue words.

[0056] 5. Cross-modal representation differences: Here, "modality" refers to different data formats, such as images and text. This term indicates that there are inherent differences between two types of inputs in terms of representation, statistical distribution, and semantic organization.

[0057] 6. Differentiable computation graph: refers to the computational path from the output back to the input where gradient propagation is possible.

[0058] Please see Figure 1 , Figure 1 The flowchart illustrates the steps of a large model inference method for multivariable time series data provided in Embodiment 1 of the present invention.

[0059] This invention provides a large-scale model inference method for multivariate time series data, comprising:

[0060] Step 101: Obtain multivariate time series data and perform short-time window slicing on the multivariate time series data to output multiple short-time window sequences.

[0061] A short-time window sequence is a data sequence that carries multivariate observation information within a local time range, obtained by slicing data into a short-time window.

[0062] It should be noted that the multivariate time series data of the target object is obtained and processed according to a preset window length. Pair with sliding step size Multivariate time series data are sliced ​​to generate multiple short-time window samples (i.e., short-time window sequences); each short-time window sample contains observations of variables at each time step, which are used to characterize the temporal state changes within the short-time window.

[0063] The multivariate time series data can come from one or more of the following sources: geomagnetism / coil / radar / video recognition / edge computing nodes / industrial sensors / database logs / business system interfaces / simulation environment outputs, as long as it can form a multivariate observation sequence of the target object over continuous time.

[0064] It is worth noting that short-term window slicing of multivariate time series data according to a preset window length and sliding step can be implemented in various ways, including fixed-length sliding windows, overlapping or non-overlapping sliding windows, multi-scale window parallel slicing, event-triggered adaptive windows, online scrolling windows, and offline batch processing windows. It should be clarified that the core of this invention is not to limit a specific window length parameter, but to decompose continuous multivariate time series data into multiple window samples that can accurately represent short-term state changes through any of the above slicing methods, providing a structured local time series data foundation for subsequent preprocessing, image generation, and other subsequent processing steps.

[0065] In this embodiment, after acquiring multivariate time-series data, a segmentation operation is performed on the continuous multivariate time-series data according to the preset window length and sliding step size. The overall time-series data is divided into multiple data segments within a local time range, thereby completing the short-time window slicing process and outputting multiple short-time window sequences. By decomposing continuous time-series data into localized window units, the short-time dynamic features of the time-series data can be extracted, providing a structured foundation for the subsequent transformation of time-series data into an image form that can be processed by a multimodal large model. This breaks the limitation that traditional time-series data can only be modeled in numerical space and is difficult to fuse and reason with natural language information.

[0066] Step 102: Preprocess multiple short-time window sequences to generate multiple preprocessed time-series window sequences.

[0067] It should be noted that preprocessing multiple short-time window sequences involves simple operations such as removing redundant interference and regularizing data formats to optimize the quality of time-series data, generating preprocessed time-series window sequences. This provides high-quality data support for subsequent image transformation and feature extraction, ensuring the stability and accuracy of the subsequent processing flow. The preprocessing operation mainly targets and corrects invalid data and interference information in the short-time window sequences, enabling the generated preprocessed time-series window sequences to more accurately reflect the temporal change patterns and adapt to the needs of subsequent image transformation and feature extraction.

[0068] Furthermore, step 102 may include the following sub-steps:

[0069] S21. Perform missing value imputation and outlier pruning on each short-time window sequence in sequence, and output multiple short-time window observation sequences after outlier pruning;

[0070] S22. A normalization function is used to map the short-time window observation sequences after outlier clipping, and multiple preprocessed time-series window sequences are output.

[0071] It should be noted that for each short-term window sample, missing value processing, outlier processing, and normalization or standardization are performed on the multivariate observation sequence within the window to obtain the preprocessed time-series window sequence. Among them, missing value processing includes imputation or interpolation, outlier processing includes pruning, smoothing, or replacement, and normalization or standardization is used to map variables of different dimensions to a unified numerical range, and can simultaneously generate a validity mask to mark the missing and imputation positions, thereby ensuring the stability and consistency of subsequent image representation.

[0072] In this embodiment, missing value imputation is first performed on the numerical gaps in each short-time window sequence, and outlier pruning is performed on abnormal data that deviates from the normal value range to obtain the outlier-pruned short-time window observation sequence. Then, the values ​​of each outlier-pruned short-time window observation sequence are mapped to a unified value range through a normalization function, and the preprocessed time series window sequence is output. By removing data interference and normalizing the distribution of integer values, the regularity and usability of the time series data are improved.

[0073] Step 103: Perform image generation operation on each preprocessed time window sequence to output multiple two-dimensional multi-channel time series images.

[0074] It should be noted that the preprocessed time-series window sequences are arranged in a structured manner and mapped in terms of spatial dimensions to complete the image generation operation. This process converts the numerical time-series data into image format data and outputs multiple two-dimensional multi-channel time-series images. This provides a suitable input carrier for subsequent feature extraction and multimodal large model inference, facilitating the joint processing of time-series features and natural language information.

[0075] Furthermore, step 103 may include the following sub-steps:

[0076] S31. Perform structured encoding on each preprocessed time window sequence and output multiple floating-point tensors;

[0077] S32. Unify the dimensions of each floating-point tensor to obtain multiple floating-point tensors with unified dimensions;

[0078] S33. Convert the data types of the floating-point tensors of each uniform dimension to obtain floating-point tensors with multiple data type conversions;

[0079] S34. Perform dimension normalization on the floating-point tensors after conversion of various data types to obtain multiple standardized tensors;

[0080] S35. Map each standardized tensor to the corresponding two-dimensional multi-channel time series image.

[0081] It should be noted that feature rearrangement and structured encoding are involved. The preprocessed time-series window sequence is rearranged and structured according to preset feature organization rules to form a floating-point tensor with unified dimensions. The feature organization rules are used to determine the arrangement of variable dimensions and time dimensions in the structured tensor, so that the structured tensor can reflect the correlation and dynamic changes among multiple variables within a short time window with a consistent spatial layout.

[0082] Furthermore, based on floating-point tensors and a unified dimension, a graphical representation is constructed to map the multivariate dynamic information within a short time window into a multi-channel image (two-dimensional multi-channel time series image); where the pixel values ​​of the image are used to characterize the intensity, trend, or pattern of change of variables over time.

[0083] It is worth noting that this invention performs short-term window slicing on multivariate time-series data according to a preset window length and sliding step size to obtain multiple short-term window sequences. Subsequently, each short-term window sequence undergoes missing value processing, outlier processing, and normalization / standardization preprocessing to remove data redundancy and interference information. Then, the preprocessed short-term window sequences are subjected to feature rearrangement and structured encoding according to preset feature organization rules. Through encoding, the time-series data is transformed into structured data that can be mapped to images. Finally, through image mapping, a multi-channel image that can represent the dynamic changes of time series within the short-term window is generated. The focus of this invention is not on the above preprocessing steps themselves, but on the multivariate time-series data organization method for short-term windows, and the method of combining feature rearrangement, structured encoding, and image mapping to transform time-series numerical data that is originally difficult to be directly used by large models into an image representation with a fixed spatial semantic layout. Moreover, the pixel values ​​of the image can accurately express the intensity, trend, and pattern of the changes of each variable over time, realizing the transformation of time-series data into an image form that can be efficiently used by large models.

[0084] Step 104: After unifying the size of each two-dimensional multi-channel temporal image, input them into the trainable projection module to output multiple visual semantic embeddings.

[0085] It should be noted that the dimensions of each two-dimensional multi-channel time-series image are first uniformized to ensure that all images are the same size and feature aligned. Then, the processed images are input into the trainable projection module, which extracts and transforms the image features and finally outputs the corresponding visual semantic embedding, laying the foundation for subsequent reasoning based on natural language instructions.

[0086] Furthermore, step 104 may include the following sub-steps:

[0087] S41. Perform size unification processing on each two-dimensional multi-channel time series image to obtain multiple time series images with unified size;

[0088] S42. A trainable projection module is used to extract features from time-series images of uniform size and output multiple visual semantic embeddings.

[0089] It should be noted that, before inputting the temporal images into the pre-trained multimodal large model, this invention does not directly feed the temporal images into the model. Instead, it first performs feature mapping on the temporal images through a trainable projection module, transforming them into visual semantic embeddings. Then, it aligns these visual semantic embeddings with the text prompts to the same multimodal semantic space. Since the temporal images and text prompts belong to different feature spaces and are not naturally matched, a dedicated trainable alignment module is needed to achieve effective alignment between the two. The trainable projection module can use linear layers, MLP, convolutional projection, attention projection, or a combination of the above methods. The dimension of the visual semantic embedding obtained after projection is consistent with the cross-modal representation dimension of the pre-trained multimodal large model, ensuring that the two can successfully complete fusion inference later. This design is the core difference between this invention and the existing scheme of "using fixed rules to visualize temporal data and directly inputting it into the model," effectively solving the problem of difficult accurate matching and poor fusion effect between temporal features and natural language features in the existing scheme.

[0090] Specifically, a trainable projection module is input into a temporal image to obtain a visual semantic embedding. This visual semantic embedding is then mapped to a multimodal semantic space, aligning it with the semantic representation of the text prompt within the same semantic space. The trainable projection module learns the mapping relationship from the temporal image to the semantic space, reducing cross-modal representation discrepancies and enhancing the composability and reasonability between visual representations and text prompts. The size of the generated images is uniformly adjusted to meet the visual input size requirements of large multimodal models, thus obtaining a temporal image for cross-modal processing.

[0091] In this embodiment, when performing size unification processing on each two-dimensional multi-channel temporal image, the width, height dimension, and number of channels of all images are adjusted to a preset unified standard to generate multiple size-unified temporal images, thereby eliminating the interference of size differences between different images on subsequent feature extraction. Then, each size-unified temporal image is input into a trainable projection module. This module, relying on its own preset learnable parameters, performs deep feature mining and spatial mapping on the input temporal images, transforming the short-term temporal features carried in the images into vector forms that are in the same feature space as natural language semantics. Finally, multiple visual semantic embeddings are output. By unifying the image size, the feature extraction bias caused by input differences is eliminated. At the same time, the trainable projection module is used to realize the transformation of temporal features into semantically matching vector forms, providing a stable and adaptable feature foundation for subsequent pre-trained multimodal large models to accurately align temporal features with natural language information and realize interpretable reasoning.

[0092] Step 105: Use a pre-trained multimodal large model to make task decisions based on multiple visual semantic embeddings and preset text prompts, and output multiple task decision results.

[0093] It should be noted that a pre-trained multimodal large model is used, which receives multiple visual semantic embeddings and corresponding preset text prompts as joint inputs. The model performs task decision analysis based on this joint input, integrates multi-dimensional information to complete reasoning and judgment, and finally outputs multiple task decision results.

[0094] Furthermore, step 105 may include the following sub-steps:

[0095] S51. Perform semantic encoding on the preset text prompts and output the semantic representation of the text prompts;

[0096] S52. Map each visual semantic embedding separately and match and align it with the semantic representation of the text prompt, outputting multiple sets of aligned visual semantic embeddings.

[0097] S53. Construct a multimodal joint input by aligning the semantic representation of the text prompts with the visual semantic embeddings of each group;

[0098] S54. Input multiple sets of multimodal joint inputs into the pre-trained multimodal large model to perform inference calculations and output multiple task decision results.

[0099] It should be noted that the joint input is constructed by fusing the text prompts and aligned visual semantic embeddings according to the input specifications of the multimodal large model and feeding them into the multimodal large model to perform inference, outputting prediction results or decision results corresponding to the task. The fusion method includes injecting the visual semantic embeddings as prefix embeddings into the text sequence or achieving multimodal splicing through image placeholders, so that the multimodal large model can combine short-time window temporal image information and text instructions to complete understanding and inference.

[0100] In this embodiment, the preset text prompts are semantically encoded, converting the natural language text prompts into semantic feature vectors that the model can process, and outputting the semantic representation of the text prompts. Then, the visual semantic embeddings are mapped and adjusted in a unified semantic space so that they are matched and aligned with the semantic representation of the text prompts in the same space, and multiple sets of aligned visual semantic embeddings are output. Subsequently, the semantic representation of the text prompts is combined with each set of aligned visual semantic embeddings to construct a multimodal joint input that meets the model input requirements. Finally, the multiple sets of multimodal joint inputs are fed into a pre-trained multimodal large model to perform inference calculations. The model combines temporal visual features and text semantic features to complete comprehensive inference and output multiple task decision results.

[0101] Optionally, the optimization process for training the projection module is as follows:

[0102] Extract task feedback signals from the decision results of multiple tasks;

[0103] Based on task feedback signals, construct optimization objectives;

[0104] The trainable projection module is iteratively updated with an optimization objective, and the optimized trainable projection module is output.

[0105] It should be noted that the task feedback signal is obtained based on the inference output, and the optimization target is constructed based on the task feedback signal. The task feedback signal includes the real label or target value in the supervised learning scenario, and the reward signal returned by the environment or simulator in the reinforcement learning scenario. The optimization target is used to measure the difference between the inference output and the expected target, and to provide a training basis for updating the projection module parameters.

[0106] Furthermore, the parameters of the trainable projection module are updated based on the optimization objective to improve the multimodal large model's ability to understand, predict, and make decisions about short-term window time series information. The phased update includes freezing the parameters of the multimodal large model and prioritizing the training of the projection module to complete semantic alignment, thereby achieving end-to-end performance optimization and improving its applicability and generalization ability in traffic flow prediction, traffic signal control, and other industrial time series analysis scenarios.

[0107] The optimized trainable projection module, serving as the core mapping unit for subsequent decision-making tasks, does not have its parameters fixed but is incorporated into a full-link closed-loop optimization system. The execution results and feedback metrics (including task completion quality, error distribution, and scenario adaptation deviation) of each round of decision-making tasks are decomposed into training samples, driving parameter updates in the projection module. Simultaneously, dynamic changes in the business scenario (such as new variable dimensions, rule iterations, and data distribution shifts) continuously serve as optimization inputs. Through gradient propagation and parameter iteration, the feature mapping capabilities of the projection module continuously adapt to new task requirements, forming an infinite closed loop of "decision execution - feedback collection - parameter optimization - module iteration - re-decision," ensuring that the module maintains optimal feature mapping and decision support capabilities throughout the long-term task cycle.

[0108] It is worth noting that this invention does not stop at the forward reasoning stage, but after outputting the task decision result, it obtains the corresponding task feedback signal based on the reasoning result, constructs an optimization target based on the task feedback signal, and iteratively updates the parameters of the trainable projection module with the optimization target as the guide. This is also an important protection point of this invention. Among them, the task feedback signal can be the label or target value in the supervised learning scenario or the reward signal in the reinforcement learning scenario. Moreover, the optimization object focuses on the trainable projection module, rather than having to fine-tune the entire pre-trained multimodal large model. By driving the trainable projection module to learn continuously through the task feedback signal, the visual semantic representation after the temporal image conversion can be more in line with the specific task target. This design also differs from the existing technology that only relies on static supervision and lacks closed-loop feedback optimization. It effectively solves the problem of insufficient adaptability between temporal feature representation and task target in the existing solution, and further improves the accuracy and relevance of multivariate temporal data reasoning.

[0109] For example, please refer to Figures 2-3 This embodiment takes a short-window multivariate time series prediction and traffic light control of a "single four-way intersection with eight lanes (one for straight and one for left turn in each direction)" as the background. By visualizing the multivariate time series data within the short window and constructing a joint input with text prompts, the multimodal large model Gemma-12B-IT is used to output traffic light control commands. The commands are then input into the simulation system to obtain the average vehicle waiting time. The average vehicle waiting time is used to construct the optimization objective of reinforcement learning. The PPO (Proximal Policy Optimization) method is used to train only the projection module parameters and freeze the large model parameters, thereby achieving end-to-end performance improvement.

[0110] Specifically, the target intersection comprises eight lanes, denoted as follows: Detectors are deployed at intersections, or the simulator directly outputs the discrete time values ​​for each lane. Observation feature vector The observed features include at least one or more of the following for each lane: flow count, queue length, average speed, and occupancy rate, thus forming multivariate time series data. Let each lane take... If there are 10 features, then the observation at each time step can be represented as:

[0111] , ;

[0112] The sampling period is set to The short-time window length is set to Each short window then contains: One time step. The sliding step size is set to... This allows for the creation of multiple short-term window samples during continuous operation.

[0113] The multimodal large model selected is Gemma-12B-IT. To meet its visual input specifications, this embodiment uniformly encodes the temporal images as follows: The pixel size is scaled and intensity normalized in a differentiable manner, so that the gradient can be backpropagated from the logarithmic probability of the output of the large model to the projection module during PPO training.

[0114] Furthermore, the multivariate time series data of the eight lanes are analyzed according to the window length. With step size Perform a sliding slice to obtain the first... One window:

[0115] ;

[0116] in, The k-th short-time window sequence is the k-th window data obtained after slicing multivariate time series data into a sliding window. It carries multivariate observation information within a local time range and can characterize the multivariate state changes within that period. This is the observation vector representing the starting time of the k-th short-time window, corresponding to the first moment within the window's time range. Multivariate observation data (in this embodiment, it is the observation vector of the traffic state of the eight lanes at this moment, which includes the traffic characteristics of each lane). Let be the observation vector at the time step following the start time in the k-th short-time window, representing the multivariate observation data at the second time step within the window. Let be the observation vector at the end of the k-th short-time window, corresponding to the multivariate observation data at the last moment of the window's time range (T−1 time steps after the start time). This window is used to characterize the intersection in the most recent The dynamic changes in traffic conditions in each lane are used as input for subsequent image generation.

[0117] Furthermore, regarding Perform missing value imputation, outlier pruning, and standardization. Calculate the mean by each variable dimension. with standard deviation For example, a standardized sequence can be obtained. :

[0118] ;

[0119] in Used for numerical stability.

[0120] Standardized window Rearranged into structured tensors according to the "lane-time-feature" rule. :

[0121] ;

[0122] Here, 8 is the first dimension of the tensor, T is the second dimension, and F is the third dimension. The first dimension corresponds to the fixed spatial layout of eight lanes, the second dimension corresponds to the temporal order within the window, and the third dimension corresponds to the selected feature set. By using fixed rearrangement rules, the spatial semantics of different windows are ensured to be consistent, enabling the large model to learn the "change pattern of the same lane at different times" and the "correlation differences between different lanes".

[0123] Furthermore, the obtained short-window multivariate timing input Recorded as .Will Convert to floating-point tensor and unify data type as When the input is a two-dimensional shape When, it expands into a three-dimensional shape. To form batch dimensions ,get:

[0124] ;

[0125] Furthermore, along the time dimension right Calculate the mean with standard deviation For example:

[0126] ;

[0127] Furthermore, Basic raster representation mapped onto a two-dimensional image plane In this embodiment, take ( =8) (eight lanes), using the variable dimension as the "row" index and the time dimension as the "column" index, construct:

[0128] , , ;

[0129] This means that the temporal evolution sequence of each lane within the window is directly unfolded into a two-dimensional grid to ensure the spatial semantic consistency of "lane-time".

[0130] In this example, the multimodal large model Gemma-12B-IT and its internal visual encoding and cross-modal alignment parameters are frozen and not updated; only the projection module parameters are updated. The freezing strategy focuses the training objective on "making the generated images more conducive to understanding and decision-making by large models," while avoiding the training instability and resource overhead caused by large model parameter updates. This module ultimately outputs image encoding. Parameters of the trained projection module .

[0131] Furthermore, construct text prompts and images. The combined input is fed into the Gemma-12B-IT, which outputs traffic light control commands. The text prompts include at least: intersection structure description, lane numbering rules, image channel semantic description, and output format constraints. For example, the output action space is set as several phase sets. The action output by Gemma is denoted as:

[0132] ;

[0133] in For Gemma parameters, This represents the policy distribution implicitly induced by the large model; actions (task decision outcomes). It can be parsed into control commands such as "phase number / release direction and duration of the current cycle" and input into the simulation system according to the preset protocol.

[0134] Furthermore, the action Input a traffic simulation system (such as the microscopic traffic simulator Cityflow or an equivalent environment), and the simulation system outputs the average vehicle waiting time after executing the signal control. This embodiment constructs reinforcement learning rewards based on average waiting time. Defined as:

[0135] ;

[0136] Furthermore, PPO is used for trainable projection modules. Optimize. For each window sample The logarithmic probability of the action is calculated from the frozen Gemma. and utilize the advantage function Constructing the cutoff target of PPO:

[0137] ;

[0138] in, This is the truncation objective function (loss function) for the PPO algorithm. To take the expectation of all window samples k, that is, to take the average of the objective function over the window samples in the training batch, is used to stabilize the training process and reduce the influence of sample variance. Importance sampling ratio, The dominance function value corresponding to the k-th window sample. The policy distribution for the frozen Gemma large model. This is the two-dimensional multi-channel time series image corresponding to the k-th window sample. For preset text prompts, These are the old projection module parameters (parameters before the update) used when sampling trajectories, and are used to calculate the old policy probability in the importance sampling ratio. ; The cutoff factor is , The old parameters used to represent the sampling trajectory.

[0139] Through the above PPO iterative training, We learned a picture projection method for eight-lane short-window multivariate time series, which enabled the frozen Gemma-12B-IT to output better signal control commands under the same prompt constraints, thereby obtaining a lower average vehicle waiting time in the simulation system.

[0140] In this example, the key to the invention lies in: Differentiable temporal images are used as the visual input carrier for a multimodal large model. With the large model parameters frozen, reinforcement learning feedback based on average vehicle waiting time is used to directly optimize the projection module, enabling the image representation to adaptively fit the understandable patterns and decision preferences of the large model. The parameter selection, reward format, and phase set size mentioned above are merely examples and do not constitute a limitation on the scope of protection of this invention. Without departing from the spirit of this invention, the feature set can be expanded to more dimensions, and the reward can be replaced with indicators such as traffic efficiency / emissions / stop counts or their weighted combinations; all of these should fall within the scope of protection of this invention.

[0141] For comparison of technical effectiveness, existing technologies can be used as a reference. With the rapid development of IoT, IoV, industrial internet, and smart city infrastructure, multivariate time-series data such as traffic flow, vehicle speed, equipment vibration, current and voltage, temperature and humidity are continuously and frequently generated and widely used for tasks such as prediction, diagnosis, and control. This type of time-series data typically features multivariate coupling, large variability, widespread noise and missing data, and rapid dynamic changes. Especially in real-time decision-making scenarios such as traffic signal control, systems often acquire observations on a timescale of seconds or even shorter and need to complete inference and decision-making within a limited time. Therefore, how to fully utilize multivariate time-series information within a short time window to improve the model's understanding and inference capabilities regarding short-term dynamics has become an important research direction in the field of time-series intelligent analysis.

[0142] In existing technologies, the mainstream methods for multivariate time series analysis mainly include traditional statistical models (such as autoregressive models) and deep learning models (such as recurrent neural networks, temporal convolutional networks, and Transformers). These methods typically model temporal dependencies within a numerical sequence space, achieving prediction or classification to a certain extent, but still have the following shortcomings: First, dynamic information within a short time window often manifests as local mutations, sparse peaks, or high-frequency noise, making the model susceptible to missing values, outliers, and dimensional differences, leading to feature instability. Second, the coupling relationships between multiple variables may exhibit nonlinear and non-stationary characteristics within a short time window, and relying solely on sequence modeling can easily result in insufficient representational ability or limited generalization. Third, in tasks requiring integration with high-level knowledge, rule constraints, or language instructions (such as traffic control strategy generation and industrial operation suggestion generation), traditional time series models struggle to directly utilize natural language knowledge for interpretable reasoning and strategy generation.

[0143] In recent years, large language models and multimodal large models have demonstrated significant advantages in knowledge understanding, logical reasoning, instruction following, and generative decision-making, promoting a "large model-centric" paradigm for intelligent reasoning and decision-making. However, existing large model systems still face significant obstacles when processing structured numerical time-series data: On the one hand, the input of large language models is mainly text tokens. Directly expanding multivariate time-series data into text form can lead to problems such as excessively long sequences, high redundancy, insufficient numerical precision, and limited context windows, and the efficiency of capturing short-term dynamic changes is also low. On the other hand, although multimodal large models can accept visual inputs such as images, existing visual branches are mainly trained on natural images or general visual semantics. When faced with "non-natural image representations" composed of time-series data, problems such as semantic misalignment, weak coupling between visual features and text instructions, and unstable cross-modal fusion effects are prone to occur, thus limiting the application effect of large models in time-series prediction and control decision-making scenarios.

[0144] To enhance the ability of large models to utilize time-series data, existing work has attempted to discretize, symbolize, or visualize the time-series data, and then input the generated images into multimodal models to obtain predictions or decisions. However, such approaches typically suffer from several drawbacks: First, the visualization process often involves fixed-rule mapping, making it difficult to learn the optimal encoding method for specific tasks, resulting in the weakening of key information or the amplification of noise. Second, there is a lack of trainable alignment mechanisms between the generated time-series images and the multimodal semantic space, making it difficult for the model to accurately associate numerical changes in the images with text prompts and task objectives. Third, the training phase often uses only static supervision signals, lacking an end-to-end optimization path that incorporates task feedback (such as reinforcement learning rewards or simulator rewards), resulting in limited performance improvements in closed-loop decision-making tasks such as traffic signal control.

[0145] In summary, current technologies lack a unified method for short-window multivariate time-series data that can stably encode dynamic information within a short window into an image representation adapted to a multimodal model without altering the original acquisition method. This method should achieve semantic space alignment with text prompts through trainable projection, and incorporate task feedback to construct optimization objectives for joint or phased training, thereby enhancing the understanding, prediction, and decision-making capabilities of large models regarding short-window time-series information. Therefore, proposing a joint method for short-window time-series data image generation, projection alignment, and large-model inference has significant engineering value and application implications, particularly suitable for traffic flow prediction, traffic signal control, and other industrial time-series analysis and intelligent decision-making scenarios.

[0146] With the continuous development of IoT, IoV, Industrial Internet, and smart city infrastructure, multi-source sensor data such as traffic flow, vehicle speed, queue length, equipment vibration, current and voltage, temperature and humidity are constantly being collected, forming high-frequency, multivariate time-series data. This type of data is widely used in scenarios such as traffic forecasting, equipment diagnosis, anomaly detection, condition assessment, dispatch control, and intelligent decision-making, and has become a fundamental data form in modern industrial and urban operation systems. Existing research shows that the development path of time-series analysis technology has roughly gone through the process of traditional autoregressive models, recurrent neural networks, convolutional and temporal convolutional networks, Transformer, and further expansion to large-scale and basic models.

[0147] In the aforementioned application scenarios, multivariate time-series data typically exhibit the following common characteristics: First, multiple variables are coupled at the same time point; second, the same variable exhibits temporal dependencies between adjacent time points; and third, the data often contains noise, missing data, dimensional differences, local abrupt changes, and non-stationary variations. Especially in real-time scenarios such as traffic signal control and industrial closed-loop control, systems typically receive state observations on a second-level or even shorter cycle, requiring models to complete state understanding, outcome prediction, and control decisions within a short timeframe. Therefore, improving the model's ability to model multivariate dynamic characteristics within a short time window has become a crucial technical challenge in this field.

[0148] In current technologies, the mainstream approach to processing multivariate time series data still primarily relies on numerical sequence space modeling. This typically involves directly encoding and inferring from time series samples using statistical learning methods, recurrent network methods, convolutional methods, or Transformer methods. While these methods can extract trend and periodic information to some extent, they often suffer from insufficient feature representation, limited robustness, and limited task generalization ability when dealing with local mutations, high-frequency fluctuations, anomalous noise, and complex coupling relationships between variables within short time windows. Especially in scenarios requiring further integration of structured numerical time series data with knowledge reasoning, language prompts, instruction generation, or control strategies, purely numerical sequence space-based approaches are insufficient for effectively connecting with large language models or multimodal large models.

[0149] In recent years, large language models and visual language models have demonstrated strong capabilities in knowledge understanding, logical reasoning, instruction following, and generative decision-making, driving the development of intelligent analysis and decision-making technologies centered on large models. However, existing large models are mainly pre-trained on text tokens or natural images, and when faced with continuous numerical multivariate time-series data, they often still suffer from input modality mismatch. On the one hand, directly converting time-series data into text sequences can easily lead to excessively long input lengths, significant redundancy, and limited numerical precision. On the other hand, even when mapping time-series data to images, the visual branches of existing visual language models are mostly trained on the semantics of natural images, and their semantic alignment and task adaptability remain limited for non-natural image representations constructed from numerical time series.

[0150] To address the aforementioned issues, existing technologies have gradually developed a technical approach of "visual representation of time series data." This involves mapping the original time series data into a two-dimensional image according to preset rules, and then using convolutional networks, visual Transformers, or basic visual models to extract features from the image. Representative methods in this approach include Gramian Angular Field (GAF), Markov Transition Field (MTF), recurrence plots, line graphs, heatmaps, and spectrograms. Related research shows that these methods can encode the local morphology, adjacency relationships, state transition relationships, or global correlation structures in the original time series into a two-dimensional representation that can be processed by visual models, thereby improving the performance of time series classification or prediction by leveraging image modeling capabilities.

[0151] In earlier technical solutions, Wang et al. proposed encoding time series data into GAF and MTF images, and then using convolutional neural networks to complete classification tasks. The core idea of ​​this approach is to first normalize and geometrically transform the original time series data, and then generate images according to fixed rules in order to extract features using mature visual networks. This type of method proves the feasibility of "time series to image" conversion, but its image generation rules are usually manually set, mainly serving classification tasks, lacking adaptive optimization mechanisms for specific downstream tasks, and not involving joint alignment with language prompts or the semantic space of large models.

[0152] Building upon this foundation, recent research has explored approaches that utilize fundamental visual models to handle temporal tasks. Related reviews indicate that after temporal sequences are visualized, they can be further integrated with visual Transformers, large-scale visual models, or visual language models to leverage the advantages of pre-trained visual models in local block relation modeling, global context modeling, and transfer learning. Essentially, this type of approach alleviates the inconsistency between continuous temporal signals and language token spaces by introducing a visual representation space, thus providing another feasible input representation method for temporal analysis.

[0153] Furthermore, to enhance the ability of large models to utilize time-series data, existing technologies have begun to explore solutions that combine "time-series images + text descriptions + large model fusion." For example, Time-VLM proposes to jointly model time-series features, time-series image features, and textual context features, enabling multimodal visual language models to simultaneously utilize visual patterns and textual semantics to complete prediction tasks. This type of approach demonstrates that time-series data can not only be processed as images but can also be further integrated into visual language models to improve prediction performance through multimodal fusion.

[0154] Meanwhile, in specific application scenarios such as traffic signal control, existing research has attempted to directly apply large language models to traffic state understanding and control decisions. For example, Traffic-R1 models traffic signal control as a reinforcement-based large language model reasoning process, forming control strategies through expert guidance and self-exploration iteration in a simulation environment. This approach demonstrates that large models are beginning to extend from general language reasoning to closed-loop decision-making scenarios such as traffic control, highlighting the model's potential in zero-shot generalization, real-time reasoning, and interpretable decision-making.

[0155] Based on existing technologies, the most relevant solutions to the problems in this field mainly fall into two categories: one is to convert multivariate time-series data into images according to fixed rules, and then perform analysis using a visual model; the other is to input time-series features, image features, and text features into a multimodal model to enhance prediction or decision-making capabilities. The former solves the problem of time-series visualization encoding, while the latter further addresses some cross-modal fusion issues. However, existing publicly available solutions are still primarily focused on classification, prediction, or textual state reasoning. The questions of "how to generate image representations more suitable for downstream tasks from short-window multivariate time-series data, how to achieve stable semantic alignment with text prompts, and how to further serve real-time control decision-making" are still under continuous development.

[0156] Furthermore, traditional statistical models for multivariate time series analysis, as well as deep learning models such as recurrent neural networks, temporal convolutional networks, and Transformers, typically model primarily within the space of numerical sequences. While these methods can achieve certain predictions or classifications, they still suffer from the following drawbacks for the problem addressed in this invention:

[0157] 1. Insufficient representation of dynamic information within short-term windows. Within a short-term window, time-series data often exhibits local abrupt changes, sparse peaks, or high-frequency noise, and is easily affected by missing values, outliers, and dimensional differences, leading to unstable features and difficulty in fully preserving short-term dynamic change information.

[0158] 2. Limited ability to model complex coupling relationships among multiple variables. Multiple variables often exhibit nonlinear and non-stationary characteristics within a short time window. Relying solely on sequence modeling can easily lead to insufficient representational power and limited generalization ability.

[0159] 3. Difficulty in combining with natural language knowledge or rule constraints for reasoning. In tasks such as traffic control strategy generation and industrial operation suggestion generation, which require combining high-level knowledge, rule constraints, or language instructions, traditional time series models struggle to directly utilize natural language information for interpretable reasoning and strategy generation.

[0160] 4. Large language models face significant obstacles in directly processing structured time-series data. Directly expanding multivariate time-series data into text input leads to problems such as excessively long sequences, high redundancy, insufficient numerical precision, and limited context windows, and also results in low efficiency in capturing short-term dynamic changes.

[0161] In summary, this invention aims to solve the following technical problems existing in the representation, understanding, and task application of short-window time-series data in the prior art:

[0162] First, it addresses the technical problem that existing time series modeling methods are unstable in representing local dynamic changes, abrupt changes, sparse peaks, and high-frequency disturbances in short-time window scenarios, making it difficult to accurately reflect the changing patterns of multivariate time series data within a short time window.

[0163] Secondly, it addresses the technical problem that existing time series analysis methods are difficult to effectively combine with natural language knowledge, rule constraints, and task instructions, thus making it difficult to support reasoning, prediction, and decision generation based on short-window time series data.

[0164] Third, it addresses the technical problems that arise when directly texturing multivariate time-series data and inputting it into a large language model, such as lengthy input sequences, high information redundancy, insufficient numerical expression capabilities, limited context windows, and low efficiency in capturing short-term dynamic information.

[0165] Fourth, it addresses the technical problems of semantic misalignment between visual representation and text prompts, insufficient cross-modal coupling ability, and unstable fusion effect in existing multimodal large models when processing non-natural image representations generated from time-series data.

[0166] Therefore, this invention proposes a large-scale model inference method for multivariate time series data. By slicing, preprocessing, rearranging features, and visualizing the multivariate time series data within a short time window, and then using a trainable projection module to map the visual representation to a multimodal semantic space, semantic alignment is achieved with text prompts. This constructs a joint input suitable for multimodal large models, thereby improving the large model's ability to understand, infer, predict, and make decisions based on short-time window time series information.

[0167] Furthermore, the present invention also aims to:

[0168] 1. Improve the representativeness and understandability of short-term window time series data, so that short-term dynamic changes, trends and patterns can be utilized by large models in a more stable and intuitive way;

[0169] 2. Reduce cross-modal representation differences between temporal image representations and text semantics, and enhance the composability and inferability between visual representations and text cues;

[0170] 3. Overcome the problem of existing large models being unsuitable for structured time series data input, enabling multimodal large models to effectively combine short-window time series information with text commands to complete predictions or decisions;

[0171] 4. By driving projection module optimization through task feedback, a continuous optimization mechanism oriented towards specific tasks is formed, thereby improving end-to-end task performance and applicability to real-world scenarios.

[0172] Specifically, short-term window slicing is performed on multivariate time-series data; data within the window is cleaned and standardized; features are rearranged according to preset rules to form structured tensors; the structured tensors are mapped to multi-channel time-series images and their size and intensity are standardized; the time-series images are input into a trainable projection module to obtain visual semantic embeddings and mapped to a multimodal semantic space, aligned with the semantic representation of text prompts; a joint input is constructed and fed into a multimodal large model to complete inference; task feedback is obtained based on supervision labels or environmental rewards and optimization objectives are constructed; the parameters of the projection module and / or the multimodal large model are updated according to the training strategy to improve the understanding, prediction, and decision-making capabilities of short-term window time-series information.

[0173] Compared with the prior art, the present invention has the following beneficial effects:

[0174] 1. Significantly enhances the understandability and inference capabilities of large models for short-term window time-series information. Existing methods often input multivariate time-series data as numerical sequences or only perform simple statistical feature summaries, which are difficult for large language models / multimodal large models to fully utilize, especially under short-term windows where dynamic change information is more easily lost. This invention explicitly expresses the intensity, trend, and pattern of multivariate changes within a short-term window in a stable and consistent spatial layout through "short-term window slicing, cleaning and normalization, feature rearrangement and structured encoding, and image generation," enabling large models to more fully capture the dynamic relationships within the window and perform inference and decision-making.

[0175] 2. Achieve cross-modal semantic space alignment to reduce the representation gap between temporal and textual information. Existing solutions often lack dedicated alignment mechanisms, making it difficult to combine temporal features and textual prompts for reasoning in the same semantic space. This invention sets up a trainable projection module to map temporal images into visual semantic embeddings and align them with the semantic representations of textual prompts. This reduces cross-modal distribution differences, improves the composability of visual representations and textual instructions, and enables large multimodal models to effectively utilize temporal information to complete prediction or control decisions.

[0176] 3. Improve end-to-end task performance and robustness without changing the original acquisition method. This invention does not rely on changes to sensors, sampling frequencies, or acquisition protocols; enhancements are only performed at the data processing and presentation layers. Simultaneously, missing value processing and outlier processing are performed within the window, and a validity mask can be generated, improving robustness to missing values, noise, and anomalies. This ensures stable and consistent generated time-series images, making it easier to deploy and achieve more stable inference performance in real-world traffic and industrial time-series scenarios.

[0177] 4. Compatible with multiple fusion specifications and large multimodal models, offering strong engineering adaptability. This invention supports multiple fusion methods during the joint input construction stage, adapting to large multimodal models with different architectures and different input protocols, reducing dependence on specific model structures, and possessing good portability and scalability.

[0178] 5. Supports supervised and reinforcement learning feedback-driven optimization, forming a sustainable self-optimizing closed loop. Existing technologies often optimize only in offline supervised learning or lack a closed-loop mechanism with task feedback. This invention constructs optimization objectives and executes joint or phased training strategies by acquiring task feedback such as supervision labels or environmental rewards, achieving targeted updates of projection modules and / or large model parameters. This continuously improves performance on specific tasks (such as traffic flow prediction and traffic signal control) and enhances generalization ability.

[0179] In this embodiment of the invention, a large-scale model inference method for multivariate time-series data is provided. The method involves acquiring multivariate time-series data, performing short-time window slicing on the data to output multiple short-time window sequences, preprocessing these sequences to generate multiple preprocessed time-series window sequences, performing image generation on each preprocessed sequence to output multiple two-dimensional multi-channel time-series images, unifying the size of each image, and inputting them into a trainable projection module to output multiple visual semantic embeddings. A pre-trained multimodal large-scale model is then used to make task decisions based on these visual semantic embeddings and preset text prompts, outputting multiple task decision results. Based on this approach, the invention decomposes multivariate time-series data into short-time window sequences and performs preprocessing and image transformation, thus simplifying the process of combining multivariate time-series data with short-time window sequences. Numerical time-series data combined with natural language is transformed into an image format that can be efficiently processed by a multimodal large model. Then, a trainable projection module extracts precise visual semantic embeddings, which, combined with natural language knowledge and rule constraints in preset text prompts, achieve a deep integration of time-series data processing and natural language information. This not only improves the representativeness and understandability of short-window time-series data, allowing dynamic changes, trends, and patterns within the short-window to be captured and utilized in a more stable and intuitive way, but also enables the time-series data processing workflow to effectively integrate natural language rules and knowledge through the reasoning capabilities of the multimodal large model. This allows the model to directly perform interpretable reasoning on time-series data based on natural language prompts, outputting decision results that conform to natural language constraints. At the same time, the precise feature extraction of the trainable projection module and the efficient reasoning of the multimodal large model further ensure the accuracy of the decision results.

[0180] Please see Figure 4 , Figure 4 This is a structural block diagram of a large model inference device for multivariable time series data provided in Embodiment 2 of the present invention.

[0181] This invention provides a large-scale model inference device for multivariate time-series data, comprising:

[0182] The acquisition module 401 is used to acquire multivariate time series data, perform short-time window slicing on the multivariate time series data, and output multiple short-time window sequences.

[0183] Preprocessing module 402 is used to preprocess multiple short-time window sequences to generate multiple preprocessed time-series window sequences;

[0184] The image generation module 403 is used to perform image generation operations on each preprocessed time window sequence and output multiple two-dimensional multi-channel time series images;

[0185] The projection module 404 is used to perform size unification processing on each two-dimensional multi-channel time series image, and then input them into the trainable projection module to output multiple visual semantic embeddings.

[0186] The decision module 405 is used to make task decisions based on multiple visual semantic embeddings and preset text prompts using a pre-trained multimodal large model, and output multiple task decision results.

[0187] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the above-described device and module can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0188] This invention also provides a computer device, including a memory and a processor, wherein the memory stores a computer program; when the computer program is executed by the processor, the processor performs the steps of the large model inference method for multivariable time series data as described in the above embodiments.

[0189] This invention also provides a computer-readable storage medium storing a computer program / instructions thereon, which, when executed by a processor, implements the steps of the large model inference method for multivariable time-series data as described in the above embodiments.

[0190] This invention also provides a computer program product, including a computer program stored on a non-transitory computer-readable storage medium, the computer program including program instructions, wherein when the program instructions are executed by a computer, the computer performs the steps of the large model inference method for multivariable time-series data as described in the above embodiments.

[0191] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0192] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0193] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0194] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0195] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for large-scale model inference for multivariate time-series data, characterized in that, include: Acquire multivariate time series data, and perform short-time window slicing on the multivariate time series data to output multiple short-time window sequences; Multiple short-time window sequences are preprocessed to generate multiple preprocessed time-series window sequences; The preprocessed time-series window sequences are subjected to image generation operations to output multiple two-dimensional multi-channel time-series images, including: The preprocessed temporal window sequences are structured and encoded to output multiple floating-point tensors; The dimensions of each floating-point tensor are unified to obtain multiple floating-point tensors with unified dimensions; The floating-point tensors of the unified dimension are converted to different data types to obtain multiple floating-point tensors after data type conversion. The dimensions of the floating-point tensors after conversion of each of the aforementioned data types are normalized to obtain multiple standardized tensors; Each of the standardized tensors is mapped to a corresponding two-dimensional multi-channel time series image; After the dimensions of each of the two-dimensional multi-channel temporal images are uniformly processed, they are input into the trainable projection module to output multiple visual semantic embeddings. A pre-trained multimodal large model is used to make task decisions based on multiple visual semantic embeddings and preset text prompts, outputting multiple task decision results, including: The preset text prompt is semantically encoded to output a semantic representation of the text prompt; Each of the aforementioned visual semantic embeddings is mapped and matched and aligned with the semantic representation of the text prompt, resulting in multiple sets of aligned visual semantic embeddings. The semantic representation of the text prompt is constructed into a multimodal joint input by each group of aligned visual semantic embeddings; Multiple sets of multimodal joint inputs are fed into the pre-trained multimodal large model to perform inference calculations and output multiple task decision results.

2. The method for large-scale model inference for multivariate time-series data according to claim 1, characterized in that, The step of preprocessing multiple short-time window sequences to generate multiple preprocessed time-series window sequences includes: Missing value imputation and outlier pruning are performed sequentially on each of the short-time window sequences, and multiple outlier-pruned short-time window observation sequences are output. A normalization function is used to map the short-time window observation sequences after outlier clipping, and multiple preprocessed time-series window sequences are output.

3. The method for large-scale model inference for multivariate time-series data according to claim 1, characterized in that, After the dimensions of each of the two-dimensional multi-channel temporal images are uniformly processed, they are respectively input into the trainable projection module, and multiple visual semantic embeddings are output, including: The dimensions of each of the two-dimensional multi-channel time-series images are uniformized to obtain multiple time-series images with uniform dimensions. The trainable projection module is used to extract features from the time-series images after unifying the dimensions, and outputs multiple visual semantic embeddings.

4. The method for large-scale model inference for multivariate time-series data according to claim 1, characterized in that, The optimization process of the trainable projection module is as follows: Extract task feedback signals from multiple task decision results; Based on the task feedback signals, an optimization objective is constructed; The trainable projection module is iteratively updated with the optimization objective to output the optimized trainable projection module.

5. A large-scale model inference apparatus for multivariate time-series data, applied to the large-scale model inference method for multivariate time-series data as described in claim 1, characterized in that, include: The acquisition module is used to acquire multivariate time series data, perform short-time window slicing processing on the multivariate time series data, and output multiple short-time window sequences; The preprocessing module is used to preprocess the multiple short-time window sequences to generate multiple preprocessed time-series window sequences; The image generation module is used to perform image generation operations on each of the preprocessed time window sequences and output multiple two-dimensional multi-channel time series images; The projection module is used to perform size unification processing on each of the two-dimensional multi-channel time-series images, and then input them into the trainable projection module to output multiple visual semantic embeddings. The decision module is used to make task decisions based on multiple visual semantic embeddings and preset text prompts using a pre-trained multimodal large model, and output multiple task decision results.

6. An electronic device, characterized in that, It includes a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor causes the processor to perform the steps of the large model inference method for multivariable time series data as described in any one of claims 1-4.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed, it implements the large model inference method for multivariable time series data as described in any one of claims 1-4.

8. A computer program product, characterized in that, The computer program product includes a computer program stored on a non-transitory computer-readable storage medium, the computer program including program instructions, wherein when the program instructions are executed by a computer, the computer performs the steps of the large model inference method for multivariable time series data as described in any one of claims 1-4.

Images