A convex or approximately convex system time series modeling method based on an input convex transformer architecture
By constructing a convex or near-convex system temporal modeling method based on causal non-negative temporal hybridization and non-negative cross attention using an input convex Transformer architecture, this method solves the problems of insufficient prediction accuracy and convexity of traditional models under complex temporal dependencies and long time domains, and achieves prediction capabilities that balance high accuracy and convexity.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- SHANDONG UNIV
- Filing Date
- 2026-05-09
- Publication Date
- 2026-06-05
AI Technical Summary
Existing high-precision time series models are difficult to use for convex optimization control. Traditional Transformer models are difficult to maintain high prediction accuracy and convexity, resulting in insufficient prediction accuracy and optimization friendliness under complex time series dependencies and long time domains.
We adopt an input-convex Transformer-based approach to construct a convex or near-convex system time-series modeling method through causal non-negative temporal hybridization, non-negative cross-attention, and time-by-time convex nonlinear mapping. This ensures that the processing paths of state and control variables are separated, maintains the convexity or near-convexity of the model, and satisfies causal and positional encoding constraints during the prediction process.
It achieves high-precision prediction under complex temporal dependencies and long time domains, while maintaining convexity, and has stronger temporal scalability and long time series modeling capabilities, making it suitable for prediction of historical and future sequences of arbitrary length.
Smart Images
Figure CN122153608A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of Transformer architecture technology, and in particular to a method for timing modeling of convex or near-convex systems based on an input convex Transformer architecture. Background Technology
[0002] With the development of methods such as model predictive control, trajectory optimization, and data-driven control, controllers are placing increasingly higher demands on predictive models. On the one hand, predictive models need to accurately characterize the dynamic evolution of complex systems over long time domains, especially handling multivariate coupling, long history dependencies, and the continuous influence of future control sequences on the state trajectory. On the other hand, predictive models should also maintain convexity or near-convexity with respect to control decision variables as much as possible, so as to directly embed them into convex optimization controllers, ensuring online solution speed, numerical stability, and engineering feasibility.
[0003] However, there is a common contradiction between existing high-precision time series models and optimizable control models. Although traditional convex modeling methods have good optimization friendliness, they often suffer from insufficient expressive power and limited prediction accuracy in complex time series dependencies, long prediction time domains, and strongly nonlinear dynamic scenarios. Although traditional Transformers have excellent time series modeling capabilities, they are usually non-convex black-box models and are difficult to use directly for convex optimization control.
[0004] Therefore, there is an urgent need to propose a novel model structure that combines high-precision time series modeling capability, long-term dependency characterization capability, convexity preservation capability, and autoregressive rolling prediction capability, so as to simultaneously meet the requirements of "high prediction accuracy" and "solvable control optimization". Summary of the Invention
[0005] This application provides a time series modeling method for convex or near-convex systems based on the input convex Transformer architecture, in order to solve the problem that high-precision time series models are difficult to use for convex optimization control, and convex optimization-friendly models are difficult to achieve high prediction accuracy.
[0006] This application provides a method for time series modeling of convex or near-convex systems based on an input convex Transformer architecture. The method includes: Several state-control trajectories of the controlled object are collected, and the state sequence and control sequence are recorded. The state sequence consists of discrete-time states, and the control sequence consists of control inputs. Based on all state sequences and control sequences, an original trajectory set is constructed. Based on the original set of trajectories, the standardized state vector corresponding to the discrete-time state is calculated to obtain the standardized state sequence of each trajectory. Based on the standardized state sequence, the preset historical window length, and the prediction time domain length, a historical state sequence, a historical control sequence, a decoder state sequence, a decoder control sequence, and a target state sequence are constructed. Based on historical state sequences and historical control sequences, and utilizing a historical encoder, the data obtained after... After layer encoding, a history memory matrix is obtained. Based on the decoder state sequence and decoder control sequence, the causal non-negative temporal hybrid algorithm in the convex decoder is used to obtain the decoding hidden state at each decoding position. Using the decoding hidden state and the history memory matrix, the query, key, value, and bias terms of each head in the convex multi-head cross-attention mechanism are obtained, and then the context vector of each decoding position is obtained. According to the context vector and the preset global history bias, the cross-attention output is generated. The cross-attention output is added to the decoding hidden state to obtain the intermediate hidden state. The intermediate hidden states are input into the convex neural network, and after passing through a preset layer, the correction term is output. The intermediate hidden states and the correction term are added together to obtain the decoder output. Using the decoder output, the future state prediction for the next time step is obtained by non-negative output projection, thus obtaining the future state prediction sequence. Based on the mean squared error loss corresponding to the future state prediction sequence and the target state sequence, the parameters of the historical encoder and convex decoder are trained and updated, thus obtaining the trained historical encoder and convex decoder.
[0007] In one implementation of this application, several state-control trajectories of the controlled object are collected, and the state sequence and control sequence are recorded; based on all state sequences and control sequences, an original trajectory set is constructed, specifically including: Collect several state-control trajectories from the controlled object or simulation environment; For the first There are trajectories, and the state sequence and control sequence are denoted as follows: , , in, Let m represent the trajectory number, and m takes values in the range [1, M], where M represents the total number of trajectories. This represents the discrete-time state of the m-th trajectory at step t. This represents the control input at step t of the m-th trajectory. Indicates the first The effective number of control steps for a single trajectory; Based on the complete state sequence and control sequence, the original trajectory set is constructed: .
[0008] In one implementation of this application, based on the original trajectory set, the standardized state vector corresponding to the discrete-time state is calculated to obtain the standardized state sequence of each trajectory, specifically including: Through the formula: Calculate the total number of state samples ;in, Let m represent the trajectory number, and m takes values in the range [1, M], where M represents the total number of trajectories. Indicates the first The effective number of control steps for a single trajectory; Through the formula: , Calculate the state mean vector and state standard deviation vector ; in, This represents the discrete-time state of the m-th trajectory at step t. This represents element-wise multiplication. This indicates a preset positive number to prevent the denominator from being zero; Through the formula: Calculate the first The trajectory at time Standardized state vector ; The first The standardized state sequence of the trajectory is denoted as: .
[0009] In one implementation of this application, a historical state sequence, a historical control sequence, a decoder state sequence, a decoder control sequence, and a target state sequence are constructed based on a standardized state sequence, a preset historical window length, and a prediction time domain length. Specifically, this includes: Set the history window length to The predicted time domain length is For any satisfying and The moment From the first Construct the historical state sequence, historical control sequence, decoder state sequence, decoder control sequence, and target state sequence from the trajectory, respectively, as follows: , , , , , in, Represents a sequence of historical states. Indicates the historical control sequence. This represents the state sequence at the decoding end. This indicates the decoding control sequence. This represents the target state sequence.
[0010] In one implementation of this application, based on historical state sequences and historical control sequences, a historical encoder is used to obtain the data after... After layer encoding, the historical memory matrix is obtained, which specifically includes: The history encoder for the first time in the history window At any given moment, a history token is defined as: , , in, This indicates that the input projection matrix is at the encoding end. This represents the m-th historical state sequence at time [time]. The standardized state vector, This indicates that the m-th historical control sequence is at time [time]. The control input is defined as m, where m ranges from [1, M], M represents the total number of trajectories (which is consistent with the total number of historical state sequences and the total number of historical control sequences), and L represents the historical window length. Indicates the first Location encoding vectors for each historical position Indicates the first The implicit representation of each historical token; stacking all historical tokens yields the input matrix for the encoding end: , Employing a multi-layer Transformer coding structure, setting the first... Layer encoder input is and order Multi-head self-attention and feedforward mapping are: , , in, Indicates the first Multi-head self-attention operator for layer encoders, Indicates the first Layered feedforward network, Representation layer normalization operator; go through After layer encoding, the historical memory matrix is obtained:
[0011] in, Indicates the number of coding layers. Indicates the first The encoded representation of each historical position.
[0012] In one implementation of this application, based on the decoder state sequence and the decoder control sequence, the causal nonnegative temporal hybrid algorithm in the convex decoder is used to obtain the hidden decoding state at each decoding position, specifically including: Set the first in the future time domain The state input for each decoding position is The control input is The initial token for decoding is defined as follows: ； in, This belongs to the decoding end state sequence. It belongs to the decoding end control sequence, and the value of k is in the range [0, ...]. 1], Indicates the length of the prediction time domain. This represents the nonnegative state projection matrix obtained through Softplus parameterization. Represents the control projection matrix. Indicates the decoding end Location encoding vectors at each position. Indicates the initial hidden decoding state; Set the first The layer input sequence is ,in, Indicates the current decoding prefix length, for the hidden state... Each channel, causal nonnegative convolutional mixed output Defined as: , in, Indicates the length of the causal convolution kernel. Indicates the first The first channel The non-negative convolution coefficients at each time delay position, and in, This represents the corresponding original convolution parameters, if ,but ; Using causal nonnegative convolution to mix outputs Calculate the hidden decoding state at the k-th decoding position. .
[0013] In one implementation of this application, the query, key, value, and bias term of each head in the convex multi-head cross-attention mechanism are obtained by utilizing the decoded hidden state and historical memory matrix, thereby obtaining the context vector of each decoding position, specifically including: Set the first The hidden decoding state at the k-th decoding position after causal mixing is: The first in the historical memory matrix The vectors are The latent space is divided into multiple parts according to the multi-head method. There are 10 heads, and the dimension of each head is:
[0014] Define query ,key Sum The projection is:
[0015]
[0016]
[0017] in, This represents the non-negative query projection matrix obtained through Softplus parameterization. and These represent the key projection matrix and the value projection matrix, respectively. and These represent the bias vectors for keys and values, respectively. This indicates the element-wise Softplus operation; At the same time, a bias term that depends solely on the encoder output is introduced into the fractional term, set as follows: , in, This represents the fractional bias mapping matrix. This represents the fractional bias vector. Indicates taking the first The scalar component corresponding to the size; The first The first in the layer The decoding position is the first A historical position Attention Score Defined as: , Non-negative attention weights:
[0018] in, For the length of the history window, ; No. The cross-attention context vector for each head is:
[0019] in, Indicates the first The height in The context vectors for each decoding position in the history memory matrix are obtained by concatenating the entire header. .
[0020] In one implementation of this application, a cross-attention output is generated based on the context vector and a preset global history bias; the cross-attention output is added to the decoded hidden state to obtain an intermediate hidden state, specifically including: Through the formula: Calculate the global bias vector for the j-th historical position; in, This represents the global bias mapping matrix. Represents the bias vector; Through the formula: Calculate the preset global history bias ; Through the formula: Calculate the cross-attention output ; Calculate the intermediate hidden state at the k-th decoding position. .
[0021] In one implementation of this application, the intermediate hidden states are input into a convex neural network, and after passing through a preset layer of recursion, a correction term is output; the intermediate hidden states are added to the correction term to obtain the decoder output, specifically including: The first hidden variable of the input convex neural network is set as follows:
[0022] in, This represents the intermediate hidden state at the k-th decoding position. This represents the corresponding raw control input. This represents the projection matrix of the non-negative hidden states. This indicates the control direct projection matrix. This represents the bias vector. This represents the hidden layer dimension of the input convex neural network; Set the subsequent implicit variables as follows:
[0023] in, Represents a non-negative latent variable recursive matrix. This represents the non-negative hidden state mapping matrix. This indicates a control direct-connect matrix. This represents the bias vector. After... After layer recursion, the output correction term is defined as:
[0024] in, This represents a non-negative output matrix. This represents the output bias vector. Finally, the... The decoder output at the k-th decoding position of layer is: .
[0025] In one implementation of this application, the future state prediction for the next time step is obtained by using the decoder output and projecting the non-negative output, thereby obtaining a future state prediction sequence, specifically including: go through After the layer convex decoder, the final decoded representation is obtained. ; The first The next-time state prediction for each position is defined as:
[0026] in, This represents the non-negative output matrix obtained through Softplus parameterization. This represents the output bias vector. Indicates the first Prediction of time-standardized states; During the training phase, Historical state sequence and The historical control sequence is input into the historical encoder, and... Decoding end state sequence and The decoder inputs the control sequence into the convex decoder to obtain the future state prediction sequence: .
[0027] As can be seen from the above technical solutions, this application has the following advantages: For example, in the prediction chain of future sequences, causal non-negative temporal mixing, non-negative cross-attention, and time-by-time input convex nonlinear mapping are introduced sequentially. Specifically, the temporal mixing part adopts a causal convolutional structure with non-negative weight constraints, so that the current time depends only on the hidden states of the current and past time steps; the cross-attention part adopts non-negative query mapping, non-negative key-value mapping, and non-negative attention weight construction methods to maintain a non-negative aggregation relationship between historical information and the current prediction; the time-by-time nonlinear mapping part adopts an input convex neural network to perform convex correction on the hidden states of each future time step. The above multiple convex structures are connected in series to form a convex prediction end structure that is different from the traditional Transformer standard decoding layer, which is the core protection point of this application.
[0028] State variables and control variables do not undergo the same processing paths: state variables are primarily used to form temporal hidden states and participate in rolling updates, while control variables, while maintaining the original time-to-time correspondence, are continuously fed into the convex nonlinear mapping structure of each future time step through a direct pass-through. This separate processing method ensures that the model can progressively add new predictive states during autoregressive prediction, and also ensures that the influence path of future control variables on the prediction results always exists explicitly at each time step, thus helping to maintain the convexity or near-convexity of the control variables. This "state recursion, control pass-through" processing method is a key protective feature that distinguishes this application from traditional convex neural network prediction models and traditional Transformer prediction models.
[0029] The model can receive historical state-control sequences of arbitrary length at the input end and predict future state sequences of arbitrary length at the output end, while consistently satisfying causality and positional encoding constraints throughout the prediction process. Specifically, the model ensures that the prediction at any given time does not depend on future information through causal temporal mixing, and distinguishes different time locations through positional encoding, enabling the same model to handle historical windows and prediction time domains of different lengths during both the training and inference phases. Compared to traditional ICNN, ICRNN, ICLSTM, and linearized prediction models that typically rely on fixed-length input-output or fixed recursive structures, this application maintains convexity while possessing stronger temporal scalability and better long-term temporal modeling capabilities. Attached Figure Description
[0030] To more clearly illustrate the technical solution of the present invention, the accompanying drawings used in the description will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0031] Figure 1This is a flowchart of a timing modeling method for convex or near-convex systems based on an input convex Transformer architecture, provided in an embodiment of this application. Detailed Implementation
[0032] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0033] Those skilled in the art should understand that the embodiments described below are merely preferred embodiments of this disclosure and do not imply that this disclosure can only be implemented through these preferred embodiments. These preferred embodiments are merely used to explain the technical principles of this disclosure and are not intended to limit the scope of protection of this disclosure. Based on the preferred embodiments provided by this disclosure, all other embodiments obtained by those skilled in the art without creative effort should still fall within the scope of protection of this disclosure.
[0034] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.
[0035] The technical solutions proposed in the embodiments of this application will be described in detail below with reference to the accompanying drawings.
[0036] The embodiment provides a time series modeling method for convex or near-convex systems based on an input convex Transformer architecture, such as Figure 1 As shown in the embodiments of this application, the method mainly includes the following steps: Step 110: Collect several state-control trajectories of the controlled object, record the state sequence and control sequence, and construct the original trajectory set based on all state sequences and control sequences.
[0037] The state sequence consists of discrete-time states, and the control sequence consists of control inputs.
[0038] In some embodiments, the discrete-time state of the controlled object is assumed to be: The control input is ,in, Representing discrete time, Indicates the dimension of the state. This represents the dimension of the control input. First, multiple state-control trajectories are acquired from the controlled object or simulation environment. For the... Let the trajectory have a state sequence and a control sequence, respectively.
[0039] in, Indicates the trajectory number [1, M]. Indicates the first The effective control steps of a trajectory Indicates the first The trajectory at time The state vector, Indicates the first The trajectory at time The control vector. The original trajectory set is composed of all sampled trajectories.
[0040] in, This represents the total number of sampling trajectories.
[0041] Step 120: Based on the original trajectory set, calculate the standardized state vector corresponding to the discrete-time state to obtain the standardized state sequence of each trajectory.
[0042] To improve training stability and mitigate the impact of differences in the dimensions of different state components, this application preferably performs global standardization on the state variables first. After summing the state samples from all trajectories, the total number of state samples is defined as:
[0043] in, This represents the total number of state vectors across all trajectories. Further, the state mean vector is defined. and state standard deviation vector They are respectively:
[0044]
[0045] in, This represents element-wise multiplication. This prevents small positive numbers with a denominator of zero. Therefore, the first... The trajectory at time The standardized state vector is defined as:
[0046] Here, the fraction represents element-wise division. This represents the standardized state vector. Correspondingly, the th The standardized state sequence of the trajectory is denoted as: .
[0047] Step 130: Based on the standardized state sequence, preset historical window length, and prediction time domain length, construct the historical state sequence, historical control sequence, decoder state sequence, decoder control sequence, and target state sequence.
[0048] After state standardization, supervised learning samples are constructed using a sliding window approach. Let the historical window length be... The predicted time domain length is Then for any satisfying and The moment From the first Construct the historical state sequence, historical control sequence, decoder state sequence, decoder control sequence, and target state sequence from the trajectory, respectively, as follows: , , , , .
[0049] in, This represents the historical status window of the encoding input. This represents the history control window corresponding to the historical state. This represents the state sequence at the decoding end. Indicates the future Step control sequence - decoder control sequence, Indicates the future The target state sequence is generated step by step. A complete supervised dataset is constructed from all legal time points on all trajectories. .
[0050] This shows that each training sample contains historical information, future control plans, and future state evolution results, enabling the model to directly learn multi-step dynamic propagation rules during the training phase.
[0051] Step 140: Based on the historical state sequence and historical control sequence, use the historical encoder to obtain the results of the previous steps. After layer encoding, a history memory matrix is obtained. Based on the decoder state sequence and decoder control sequence, the causal non-negative temporal hybrid algorithm in the convex decoder is used to obtain the decoding hidden state at each decoding position. Using the decoding hidden state and the history memory matrix, the query, key, value and bias terms of each head in the convex multi-head cross-attention mechanism are obtained, and then the context vector of each decoding position is obtained. According to the context vector and the preset global history bias, the cross-attention output is generated. The cross-attention output is added to the decoding hidden state to obtain the intermediate hidden state.
[0052] In some embodiments, the overall model consists of a history encoder and a convex decoder. The history encoder only receives historical states and historical controls as input to extract historical context information; the convex decoder receives state tokens and control tokens (decoder-side state sequence and decoder-side control sequence) in the future time domain to construct a multi-step prediction result that maintains convexity regarding the future control sequence. Let the latent space dimension of the encoder input token be... Then for the first in the history window At any given moment, a history token is defined as:
[0053] in, This indicates that the input projection matrix is at the encoding end. Indicates the first Location encoding vectors for each historical position Indicates the first The implicit representation of each historical token. Stacking all historical tokens yields the input matrix for the encoding end: .
[0054] The encoder preferably employs a multi-layer Transformer encoding structure. Let the first... Layer encoder input is and order Then, the multi-head self-attention and feedforward mapping of this layer can be abstractly written as:
[0055]
[0056] in, Indicates the first Multi-head self-attention operator for layer encoders, Indicates the first Layered feedforward network, The layer normalization operator is represented. After... After layer encoding, the historical memory matrix is obtained:
[0057] in, Indicates the number of coding layers. Indicates the first The encoded representation of each historical position.
[0058] In the decoding phase, this application first projects the future state and future control separately. For the ... There are several positions, and their status inputs are... The control input is Then the initial token of the decoding end is defined as:
[0059] in, This represents the nonnegative state projection matrix obtained through Softplus parameterization. Represents the control projection matrix. Indicates the decoding end Location encoding vectors at each position. This represents the initial hidden decoding state. Because... Non-negative, but about Since it is an affine mapping, the initial token maintains at least an affine relationship with respect to future control variables, providing a basis for subsequent convexity preservation.
[0060] Each convex decoder layer in this application comprises three parts: causal non-negative temporal mixing, convex cross-attention, and a token-by-token input convex neural network. We first consider causal non-negative temporal mixing. Let the... The layer input sequence is ,in, This indicates the current decoding prefix length. For the hidden state... The output of the causal nonnegative convolutional mixture with 1 channel is defined as follows:
[0061] in, Indicates the length of the causal convolution kernel. Indicates the first The first channel The non-negative convolution coefficients at each time delay position, and:
[0062] in, This represents the corresponding original convolution parameters. If... Then it is agreed Therefore, this hybrid structure only utilizes the hidden states of the current and past time steps and does not use future information, thus satisfying strict causality; at the same time, since the convolution coefficients are non-negative, this mapping maintains monotonicity and convexity with respect to the input sequence.
[0063] Next, this application introduces a convex multi-head cross-attention mechanism. Unlike the softmax normalized weights used in traditional Transformers, this application employs Softplus non-negative weights and maintains convexity with respect to future control variables through special construction of the query, key, and value. Let the... The decoded hidden state of the layer after causal mixing is The first in the historical memory matrix The vectors are The latent space is divided into multiple parts according to the multi-head method. There are 10 heads, and the dimension of each head is:
[0064] in, This represents the number of attention heads. First, define the projections of the query, key, and value as:
[0065]
[0066]
[0067] in, This represents the non-negative query projection matrix obtained through Softplus parameterization. and These represent the key projection matrix and the value projection matrix, respectively. and These represent the bias vectors for keys and values, respectively. This represents the element-wise Softplus operation. Therefore, and All components are non-negative. , , After splitting by the head, let the first one be the first. The vector on the head is , and .
[0068] To enhance the expressive power of historical context, this application also introduces a bias term in the fractional term that depends solely on the encoder output. Let: , in, This represents the fractional bias mapping matrix. This represents the fractional bias vector. Indicates taking the first The scalar component corresponding to each head. Then the first... The first in the layer The decoding position is the first A historical position Attention Score Defined as: .
[0069] Therefore, non-negative attention weights are constructed:
[0070] in, This is the length of the history window, used as a scale normalization constant. Since the Softplus output is always positive, therefore... Further define the first The cross-attention context vector for each head is:
[0071] in, Indicates the first The height in The context vector at each position. Concatenating all the headers yields: .
[0072] In addition to the local attention context, this application also extracts a global bias vector from the entire historical sequence. Let:
[0073] in, This represents the global bias mapping matrix. Let the bias vector be denoted as , then the global history bias is defined as: .
[0074] Therefore, the output of cross-attention is written as: .
[0075] Adding this to the causal mixed output, we can obtain the intermediate hidden state after attention update: .
[0076] Its convexity retention mechanism is as follows: when the encoder outputs... When fixed, the key vector AND value vector All are non-negative constant vectors; query vector By the action of non-negative linear layer Therefore, for It is a monotone convex mapping; fractional term yes Regarding nonnegative constants A linear combination of these, plus an offset independent of the decoding end, therefore... It is a convex function; Softplus is a convex and monotonically non-decreasing function, therefore right It is still a convex function; because Non-negative, therefore the context vector It is a non-negative linear combination of a set of convex functions, thus preserving convexity. Therefore, with the encoder context fixed, the cross-attention output remains convex or approximately convex with respect to the decoded hidden state and further with respect to the future control sequence.
[0077] Step 150: Input the intermediate hidden state into the convex neural network, and after recursion through a preset layer, output the correction term; add the intermediate hidden state and the correction term to obtain the decoder output.
[0078] Following cross-attention, this application employs a token-by-token input convex neural network to further enhance expressive power. For the... Layer There are 10 decoding positions, and the intermediate hidden state after the cross-attention output is 1. The corresponding original control input is Define the first layer of hidden variables in the input convex neural network as:
[0079] in, This represents the projection matrix of the non-negative hidden states. This indicates the control direct projection matrix. This represents the bias vector. This represents the hidden layer dimension of the input convex neural network. For subsequent... Layer, definition:
[0080] in, Represents a non-negative latent variable recursive matrix. This represents the non-negative hidden state mapping matrix. This indicates a control direct-connect matrix. This represents the bias vector. After... After layer recursion, the output correction term is defined as:
[0081] in, This represents a non-negative output matrix. This represents the output bias vector. Finally, the... The layer decoder output is: .
[0082] This structure corresponds to TokenICNN in the code. Since the weights involving old latent variables and attention latent states in each layer are guaranteed to be non-negative through Softplus, and the activation function is Softplus, therefore... Given that the future control variables remain convex, The convexity of the control variables remains with respect to the future, and the residual connection does not violate this property.
[0083] Step 160: Using the decoder output, the future state prediction for the next time step is obtained by non-negative output projection, and then the future state prediction sequence is obtained; based on the mean square error loss of the future state prediction sequence and the target state sequence, the parameters of the historical encoder and convex decoder are trained and updated, and then the trained historical encoder and convex decoder are obtained.
[0084] go through After the layer convex decoder, the final decoded representation is obtained. This application further employs non-negative output projection to obtain future state predictions. The next-time state prediction for each position is defined as:
[0085] in, This represents the non-negative output matrix obtained through Softplus parameterization. This represents the output bias vector. Indicates the first Prediction of the standardized state at time step. Therefore, the first... The final implicit representation of each token corresponds to the prediction .
[0086] During the training phase, this application employs teacher forcing for supervised learning. Specifically, it will... and Input the history encoder, and and Input to the convex decoder to obtain multi-step prediction results: .
[0087] The corresponding target sequence is Let the total number of training samples be... The mean squared error loss in the standardized state space of the target state sequence is defined as:
[0088] in, Indicates the first The sample at the th The predicted state at a future moment. This represents the standardized target state corresponding to the target state sequence. Let represent the Euclidean norm. The model parameters are denoted as . Then the AdamW optimizer can be used according to: ; To be updated, among which, Indicates the learning rate. This represents the gradient of the loss function with respect to the parameters. During training, validation set loss, adaptive learning rate decay, gradient pruning, and early stopping mechanisms can also be combined to select the optimal model parameters.
[0089] During the inference phase, this application employs an autoregressive rolling prediction method based on multiple token prefixes, corresponding to `rollout_multi_token` in the code. Let the current time be... The most recent The standardized historical state sequence and historical control sequence are as follows:
[0090] The current standardization status is Given a future control plan:
[0091] in, This represents the future control length during the inference phase. First, the historical information only needs to be encoded once to obtain the historical memory matrix:
[0092] in, This represents the historical encoder mapping. Then, the initial state prefix is constructed: , And control prefixes: .
[0093] In the During the next rolling prediction, the existing state prefix and control prefix are denoted as follows:
[0094] in, , This indicates the predicted number of... Standardize the state at any given moment. and The entire sequence is input into the decoder to obtain the predicted output for that prefix length:
[0095] in, Indicates the length of this prefix. The predicted state corresponding to the nth token. Since the output header always includes the nth token... Each token is mapped to the state at the next time step, so taking the last output yields the new rolling prediction value: .
[0096] Will Add it to the state prefix and continue rolling in the next round until all are obtained. Step-by-step prediction sequence: .
[0097] If it is necessary to restore the original state space, then perform destandardization for each prediction time step:
[0098] in, This indicates the predicted state after restoring to the original dimensions.
[0099] Since positional encoding is not bound to the fixed window length during training, and causal nonnegative convolution and prefix decoding structures inherently support variable-length sequences, this application can use only a fixed historical window length during training. and predicted length Supervised learning is performed, with different historical lengths and future control lengths used during inference and control deployment. As long as the length of the sequence used does not exceed the preset maximum length of the position code, it is acceptable.
[0100] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A method for time series modeling of convex or near-convex systems based on an input convex Transformer architecture, characterized in that, The method includes: Several state-control trajectories of the controlled object are collected, and the state sequence and control sequence are recorded. The state sequence consists of discrete-time states, and the control sequence consists of control inputs. Based on all state sequences and control sequences, an original trajectory set is constructed. Based on the original set of trajectories, the standardized state vector corresponding to the discrete-time state is calculated to obtain the standardized state sequence of each trajectory. Based on the standardized state sequence, the preset historical window length, and the prediction time domain length, a historical state sequence, a historical control sequence, a decoder state sequence, a decoder control sequence, and a target state sequence are constructed. Based on historical state sequences and historical control sequences, and utilizing a historical encoder, the data obtained after... After layer encoding, a history memory matrix is obtained. Based on the decoder state sequence and decoder control sequence, the causal non-negative temporal hybrid algorithm in the convex decoder is used to obtain the decoding hidden state at each decoding position. Using the decoding hidden state and the history memory matrix, the query, key, value, and bias terms of each head in the convex multi-head cross-attention mechanism are obtained, and then the context vector of each decoding position is obtained. According to the context vector and the preset global history bias, the cross-attention output is generated. The cross-attention output is added to the decoding hidden state to obtain the intermediate hidden state. The intermediate hidden states are input into the convex neural network, and after passing through a preset layer, the correction term is output. The intermediate hidden states and the correction term are added together to obtain the decoder output. Using the decoder output, the future state prediction for the next time step is obtained by non-negative output projection, thus obtaining the future state prediction sequence. Based on the mean squared error loss corresponding to the future state prediction sequence and the target state sequence, the parameters of the historical encoder and convex decoder are trained and updated, thus obtaining the trained historical encoder and convex decoder.
2. The time-series modeling method for convex or near-convex systems based on an input convex Transformer architecture according to claim 1, characterized in that, Collect several state-control trajectories of the controlled object and record the state sequence and control sequence; Based on all state sequences and control sequences, the original trajectory set is constructed, specifically including: Collect several state-control trajectories from the controlled object or simulation environment; For the first There are trajectories, and the state sequence and control sequence are denoted as follows: ， ， in, Let m represent the trajectory number, and m takes values in the range [1, M], where M represents the total number of trajectories. Let represent the discrete-time state of the m-th trajectory at step t. This represents the control input at step t of the m-th trajectory. Indicates the first The effective number of control steps for a single trajectory; Based on the complete state sequence and control sequence, the original trajectory set is constructed: 。 3. The time-series modeling method for convex or near-convex systems based on an input convex Transformer architecture according to claim 1, characterized in that, Based on the original set of trajectories, the standardized state vectors corresponding to the discrete-time states are calculated to obtain the standardized state sequence of each trajectory, specifically including: Through the formula: Calculate the total number of state samples ;in, Let m represent the trajectory number, and m takes values in the range [1, M], where M represents the total number of trajectories. Indicates the first The effective number of control steps for a single trajectory; Through the formula: , Calculate the state mean vector and state standard deviation vector ; in, Let represent the discrete-time state of the m-th trajectory at step t. This represents element-wise multiplication. This indicates a pre-defined positive number to prevent the denominator from being zero; Through the formula: Calculate the first The trajectory at time Standardized state vector ; The first The standardized state sequence of the trajectory is denoted as: 。 4. The time-series modeling method for convex or near-convex systems based on an input convex Transformer architecture according to claim 1, characterized in that, Based on standardized state sequences, preset historical window lengths, and prediction time domain lengths, historical state sequences, historical control sequences, decoder-end state sequences, decoder-end control sequences, and target state sequences are constructed, specifically including: Set the history window length to The predicted time domain length is For any satisfying and The moment From the first Construct the historical state sequence, historical control sequence, decoder state sequence, decoder control sequence, and target state sequence from the trajectory, respectively, as follows: ， ， ， ， ， in, Represents a sequence of historical states. Indicates the historical control sequence. This represents the state sequence at the decoding end. This indicates the decoding control sequence. This represents the target state sequence.
5. The time-series modeling method for convex or near-convex systems based on an input convex Transformer architecture according to claim 1, characterized in that, Based on historical state sequences and historical control sequences, and utilizing a historical encoder, the data obtained after... After layer encoding, the historical memory matrix is obtained, which specifically includes: The history encoder for the first time in the history window At any given moment, a history token is defined as: ， ， in, This indicates that the input projection matrix is at the encoding end. This represents the m-th historical state sequence at time [time]. The standardized state vector, This indicates that the m-th historical control sequence is at time [time]. The control input is defined as m, where m ranges from [1, M], M represents the total number of trajectories (which is consistent with the total number of historical state sequences and the total number of historical control sequences), and L represents the historical window length. Indicates the first Location encoding vectors for each historical position Indicates the first The implicit representation of each historical token; stacking all historical tokens yields the input matrix for the encoding end: ， Employing a multi-layer Transformer coding structure, setting the first... Layer encoder input is and order Multi-head self-attention and feedforward mapping are: ， ， in, Indicates the first Multi-head self-attention operator for layer encoders, Indicates the first Layered feedforward network, Representation layer normalization operator; go through After layer encoding, the historical memory matrix is obtained: in, Indicates the number of coding layers. Indicates the first The encoded representation of each historical position.
6. The time-series modeling method for convex or near-convex systems based on an input convex Transformer architecture according to claim 1, characterized in that, Based on the decoder's state sequence and control sequence, the hidden decoder states at each decoding position are obtained using a causal non-negative temporal hybrid algorithm in the convex decoder. Specifically, these include: Set the first in the future time domain The state input for each decoding position is The control input is The initial token for decoding is defined as follows: ； in, This belongs to the decoding end state sequence. It belongs to the decoding end control sequence, and the value of k is in the range [0, ...]. 1], Indicates the length of the prediction time domain. This represents the nonnegative state projection matrix obtained through Softplus parameterization. Represents the control projection matrix. Indicates the decoding end Location encoding vectors at each position. Indicates the initial hidden decoding state; Set the first The layer input sequence is ,in, Indicates the current decoding prefix length, for the hidden state... Each channel, causal nonnegative convolutional mixed output Defined as: ， in, Indicates the length of the causal convolution kernel. Indicates the first The first channel The non-negative convolution coefficients at each time delay position, and in, This represents the corresponding original convolution parameters, if ,but ; Using causal nonnegative convolution to mix outputs Calculate the hidden decoding state at the k-th decoding position. .
7. The time-series modeling method for convex or near-convex systems based on an input convex Transformer architecture according to claim 1, characterized in that, By utilizing the decoded hidden state and historical memory matrix, the query, key, value, and bias terms of each head in the convex multi-head cross-attention mechanism are obtained, thereby yielding the context vector for each decoding position, specifically including: Set the first The hidden decoding state at the k-th decoding position after causal mixing is: The first in the historical memory matrix The vectors are The latent space is divided into multiple parts according to the multi-head method. There are 10 heads, and the dimension of each head is: Define query ,key Sum The projection is: in, This represents the non-negative query projection matrix obtained through Softplus parameterization. and These represent the key projection matrix and the value projection matrix, respectively. and These represent the bias vectors for the keys and values, respectively. This indicates the element-wise Softplus operation; At the same time, a bias term that depends solely on the encoder output is introduced into the fractional term, set as follows: ， in, This represents the fractional bias mapping matrix. This represents the fractional bias vector. Indicates taking the first The scalar component corresponding to the size; The first The first in the layer The decoding position is the first A historical position Attention Score Defined as: ， Non-negative attention weights: in, For the length of the history window, ; No. The cross-attention context vector for each head is: in, Indicates the first The height in The context vectors for each decoding position in the history memory matrix are obtained by concatenating the entire header. 。 8. The time-series modeling method for convex or near-convex systems based on an input convex Transformer architecture according to claim 1, characterized in that, Based on the context vector and the preset global history bias, a cross-attention output is generated; The cross-attention output is added to the decoded hidden state to obtain the intermediate hidden state, which specifically includes: Through the formula: Calculate the global bias vector for the j-th historical position; in, This represents the global bias mapping matrix. Represents the bias vector; Through the formula: Calculate the preset global history bias ; Through the formula: Calculate the cross-attention output ; Calculate the intermediate hidden state at the k-th decoding position. .
9. The time-series modeling method for convex or near-convex systems based on an input convex Transformer architecture according to claim 1, characterized in that, The intermediate hidden states are input into the convex neural network, and after passing through a preset layer of recursion, the correction term is output. The intermediate hidden states are added to the correction term to obtain the decoder output, which specifically includes: The first hidden variable of the input convex neural network is set as follows: in, This represents the intermediate hidden state at the k-th decoding position. This represents the corresponding raw control input. This represents the projection matrix of the non-negative hidden states. This indicates the control direct projection matrix. This represents the bias vector. This represents the hidden layer dimension of the input convex neural network; Set the subsequent implicit variables as follows: in, Represents a non-negative latent variable recursive matrix. This represents the non-negative hidden state mapping matrix. This indicates a control direct-connect matrix. Represents the bias vector; after After layer recursion, the output correction term is defined as: in, This represents a non-negative output matrix. This represents the output bias vector; ultimately, the first... The decoder output at the k-th decoding position of layer is: 。 10. The time-series modeling method for convex or near-convex systems based on an input convex Transformer architecture according to claim 1, characterized in that, Using the decoder output, a non-negative output projection is used to obtain the future state prediction for the next time step, thus obtaining the future state prediction sequence, specifically including: go through After the layer convex decoder, the final decoded representation is obtained. ; The first The next-time state prediction for each position is defined as: in, This represents the non-negative output matrix obtained through Softplus parameterization. This represents the output bias vector. Indicates the first Prediction of time-standardized states; During the training phase, Historical state sequence and The historical control sequence is input into the historical encoder, and will Decoding end state sequence and The decoder inputs the control sequence into the convex decoder to obtain the future state prediction sequence: 。