Module-level interpretability optimization method for time sequence Transform model

Abstract

The invention discloses a module-level interpretability optimization method for a time sequence Transform model, which comprises a mask generation and disturbance stage and an optimization model application stage, and is characterized in that in the mask generation and disturbance stage, a time sequence data set is acquired, a training set is divided, a to-be-optimized pre-training time sequence Transform base model is loaded, and the to-be-optimized pre-training time sequence Transform base model is loaded; by constructing a time-module fusion network, executing frequency domain input disturbance based on wavelet transform, executing module disturbance and executing joint optimization, fine-grained feature screening and model pruning of time-frequency domain dimensions are achieved, the model prediction performance is not reduced, meanwhile, the calculation cost and memory occupation are remarkably reduced, and the calculation efficiency is improved. The problems that an existing X-formers model is serious in framework redundancy, insufficient in anti-noise capacity and single in interpretability are solved.

Description

Technical Field

[0001] This invention relates to the field of time series analysis and deep learning model optimization technology, and in particular to a module-level interpretability optimization method for time series Transformer models. Background Technology

[0002] Time series forecasting has wide applications in energy management, traffic flow forecasting, and meteorological analysis. In recent years, numerous variant models based on the Transformer architecture (collectively referred to as X-formers) have been proposed and have demonstrated excellent predictive performance. However, these performance improvements are often accompanied by significant architectural redundancy. Research shows that many modules in X-formers not only increase computational overhead but may also weaken the model's ability to capture key features. Although some studies have attempted to simplify the attention mechanism using CNNs or structured matrices, or proposed non-Transformer models (such as DLinear and TimesNet), removing redundant components while retaining the advantages of the Transformer architecture remains a challenge.

[0003] Current research on model interpretability is mainly divided into two categories: ante-hoc methods and post-hoc methods. Ante-hoc methods are represented by interpretability-driven neural architecture search (INAS), but they struggle to balance architectural interpretability with model performance and have limited generalization ability. Post-hoc methods are mainly based on perturbation (such as Gatemask and Dynamask), which typically treat the model as a black box and mostly perform masking perturbations on the input data in a single time dimension. This single-scale perturbation method cannot effectively distinguish between low-frequency components carrying long-term trends and high-frequency noise containing random perturbations in time series data, easily leading to interpretation results that confuse key signals with noise interference and lacking the ability to perform refined analysis of the frequency domain characteristics of the data.

[0004] Therefore, existing technologies cannot effectively reveal the deep interaction between the internal modules of X-formers and the multi-scale time-frequency features of the input data, making it difficult to achieve robust and fine-grained model optimization. There is an urgent need for a method that can simultaneously optimize multi-scale frequency domain feature selection and model module structure, reducing model complexity and improving interpretability while ensuring predictive performance and suppressing high-frequency noise interference. Summary of the Invention

[0005] To address the aforementioned technical problems, this invention provides a module-level interpretability optimization method for time series Transformer models. This method identifies redundant modules in the model through joint time-frequency domain analysis and utilizes multi-channel masks to accurately locate key input features (such as long-term trends and short-term fluctuations) at different frequency components. This improves the computational efficiency, robustness, and interpretability of the model in time series prediction tasks, overcoming the shortcomings of existing X-formers models, such as redundant architecture, high computational overhead, and the inability of existing interpretability methods to precisely assess module importance.

[0006] This invention is achieved using the following technical solution: A module-level interpretability optimization method for time series Transformer models includes a mask generation and perturbation stage and an optimization model application stage. The mask generation and perturbation stage includes the following steps: Step A1: Obtain the time series dataset and divide it into training sets, then load the pre-trained time series Transformer base model to be optimized; Step A2: Construct a time-module fusion network and generate a highly interpretable multi-channel input mask by utilizing the different frequency components of the input data and the model structure information; Step A3: Perform frequency domain input perturbation based on wavelet transform. Using the generated multi-channel input mask, refine the perturbation of the time series in the wavelet domain to generate a perturbation input that highlights key features. Step A4: Perform module perturbation, construct a learnable module mask, and decide whether to retain the original module or replace it with an identity mapping based on the state of the module mask to generate a perturbation model; Step A5: Perform joint optimization by feeding the perturbation input into the perturbation model, calculating the difference between the output and the original model output, constructing the total loss function by combining the mask sparsity regularization term, and updating the parameters of the input mask and the module mask in parallel.

[0007] Specifically, step A2 includes the following steps: Step A21: Construct a feature fusion intermediate layer to process the input time series data. , ,in For time step, For the feature dimension; mapped to the module mask through a linear transformation. The appropriate dimension is then multiplied element-wise with the module mask to generate a hybrid feature that integrates the structural information of the model. , is represented as: ; Step A22: Utilizing hybrid features The query matrix is ​​generated through two independent linear transformations. Bond matrix ; directly on the raw input data Perform linear transformations to generate value matrices , respectively represented as: ; ; ; Step A23: Calculate the scaled dot product attention and input mask generation based on the query matrix. Key matrix Sum matrix Calculate the attention distribution by querying the matrix. Bond matrix The interaction captures the data and the correlation between modules, and applies this attention weight to the value matrix. The attention output is processed by a feedforward neural network (FFN) to obtain intermediate features used to generate the input mask. , is represented as: ; Step A24: Based on intermediate features ,pass Each independent projection head generates a corresponding set of frequency domain masks in parallel. ;in Corresponding to the low-frequency approximation component, Corresponding to high-frequency detail components at each level; for Any number within the range Each frequency component has its corresponding mask. The calculation formula is: .

[0008] Specifically, step A3 includes the following steps: Step A31: Based on discrete wavelet transform, convert the time series data... conduct Level decomposition, yielding A set of frequency components , is represented as: ; Step A32: Use the mask generated in step A24 Combined with the reference perturbation value corresponding to the frequency band generated by the bidirectional GRU For each frequency component Perform independent perturbation processing; for Any number within the range One component, the perturbed component The calculation formula is: ; Step A33: Add all perturbated frequency components The final perturbation input data is obtained by reconstructing the time domain using inverse discrete wavelet transform. , is represented as: .

[0009] Specifically, step A4 includes the following steps: Step A41: Define an updatable module parameter matrix Calculate the binarized module mask , is represented as: ; Step A42: For the first in the model Each module, if If it is, then replace it with an identity mapping; if If so, the original module will be retained; Step A43: Construct the perturbation model Its output is .

[0010] Specifically, the loss function for joint optimization in step A5 is expressed as: ; in, This indicates a measure of the difference between the output of the original model and the perturbation model; This is the output of the original model; For sparsity regularization, promote mask sparsity to identify the most critical parts; Ensure the validity of the reference disturbance values ​​corresponding to the frequency band generated by the bidirectional GRU.

[0011] Specifically, the optimization model application stage includes the following steps: Step B1: Obtain the trained module mask and identify redundant modules with a mask value of 0; Step B2: Prune and reconstruct the original time series Transformer model, permanently replacing the identified redundant modules with identity mappings or other more efficient modules to obtain a lightweight optimized model; Step B3: Input the time series data to be predicted into the optimization model to obtain the final prediction result.

[0012] The beneficial effects of this invention are as follows: (1) Enhanced multi-scale time-frequency interpretability: This invention introduces wavelet decomposition technology, breaking through the limitation of existing technologies that can only interpret data in a single time dimension. By generating multi-channel frequency domain masks, this invention can not only reveal the time steps that the model is interested in, but also further distinguish whether the model depends on low-frequency long-term trend components or high-frequency detail fluctuation components. This joint time-frequency analysis provides deeper theoretical support for understanding the behavior of the model when dealing with data of different characteristics (such as stationary data and abrupt data).

[0013] (2) Superior noise resistance and robustness: Thanks to the independent perturbation mechanism in the frequency domain, the present invention can automatically identify and suppress high-frequency noise interference in the input data. During the optimization process, by applying sparsity constraints to the high-frequency detail components, the model can learn to ignore non-informative random noise, thereby making the optimized model more robust and predictive stable than the original model when facing noisy environments or scenarios with poor data quality.

[0014] (3) Significantly improve computational efficiency and reduce memory usage: Through joint optimization strategies, redundant modules (such as redundant attention heads or feedforward layers) in the X-formers architecture are accurately identified and eliminated. Without sacrificing or even improving the model's prediction performance, the number of model parameters and computational load are significantly reduced, and the memory usage during inference is greatly reduced, making it more suitable for deployment on resource-constrained edge devices.

[0015] (4) Strong versatility: It has good generalization ability and is applicable to a variety of Transformer-based time series prediction models (such as Informer, Autoformer, FEDformer, etc.). The wavelet decomposition level can be flexibly adjusted according to the data characteristics, making it easy to promote and apply in time series analysis tasks in different fields. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the structures shown in these drawings without creative effort.

[0017] Figure 1 This is a diagram illustrating the overall framework for module-level interpretability optimization of the Transformer model in this embodiment of the invention. Figure 2 This is a schematic diagram of the feature fusion network model architecture in an embodiment of the present invention; Figure 3 This is a schematic diagram of the mask generation and perturbation stages in an embodiment of the present invention; Figure 4 This is a schematic diagram of the process for the application stage of the optimization model in an embodiment of the present invention. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0019] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0020] The following is in conjunction with the appendix Figures 1-4 The following describes some embodiments of the present invention in detail. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0021] This invention proposes a module-level interpretability optimization method for time series Transformer models. By jointly optimizing multi-channel frequency domain input masks and module masks, it evaluates the importance of each module in the X-formers (time series Transformer model) architecture and the impact of data characteristics under different frequency components on these modules, thereby identifying redundant modules and key signals and optimizing them. This method comprises a mask generation and perturbation stage and an optimized model application stage. In the mask generation and perturbation stage, an improved multi-scale time-module fusion network is used to integrate the dynamic features of the data and the model structure information to generate multi-channel frequency domain input masks corresponding to different wavelet decomposition levels. Subsequently, discrete wavelet transform is used to decompose the input data into different frequency components. Each component is independently perturbed according to the generated frequency domain mask, and the time domain signal is reconstructed through inverse wavelet transform to identify key long-term trends and suppress high-frequency noise. Simultaneously, module masks are generated to evaluate the contribution of each component, and all mask parameters are updated in parallel while minimizing output differences through a joint optimization strategy. In the optimized model application stage, the module masks obtained from training are identified and replaced with identity mappings to construct a lightweight optimized model for final prediction of time series data, thereby improving computational efficiency while maintaining performance.

[0022] In one specific embodiment, the overall architecture of the module-level interpretability optimization method for time series Transformer models is as follows: Figure 1 As shown, the method mainly includes a mask generation and perturbation stage and an optimization model application stage.

[0023] In this embodiment, the mask generation and perturbation stages are as follows: Figure 3 Specifically, it includes: S1. Data Preparation and Model Initialization. Obtain time-series datasets (such as electricity, traffic, weather, etc.), represented as... ,in For time step, For feature dimensions. Load the pre-trained X-former model to be analyzed. .

[0024] S2, Build Time-Module Fusion Network, Model Architecture as follows: Figure 2 As shown, a highly interpretable multi-channel input mask is generated using the different frequency component features of the input data and model structure information; in this embodiment, this step specifically includes: S21. Construct a feature fusion intermediate layer. First, input time series data... Mapped to the module mask via linear transformation The appropriate dimension is then multiplied element-wise with the module mask to generate hybrid features that incorporate model structure information. : ; S22. Generate the query matrix Key matrix Sum matrix Using the hybrid features obtained in step A21 The query matrix is ​​generated through two independent linear transformations. Bond matrix At the same time, directly processing the raw input data Perform linear transformations to generate value matrices : , , ; S23. Calculate the scaled dot product attention and generate the input mask. Based on the above generated... , , Calculate attention distribution, by and The interaction captures data and the correlation between modules, and applies this attention weight to... Subsequently, the attention output is processed by a feedforward neural network (FFN) to obtain intermediate features used to generate the input mask. : ; S24, Based on intermediate features ,pass Each independent projection head generates a corresponding set of frequency domain masks in parallel. .in Corresponding to the low-frequency approximation component, Corresponding to each level of high-frequency detail components. For any... Each frequency component ( ), and its corresponding mask The calculation formula is: .

[0025] S3. Perform frequency domain input perturbation based on wavelet transform. Using the generated multi-channel input mask, refine the perturbation of the time series in the wavelet domain to generate a perturbation input that highlights key features. In this embodiment, this step specifically includes: S31. Using discrete wavelet transform, time series data... conduct Level decomposition, yielding A set of frequency components : ; S32. Using the corresponding mask generated in step A2 and the reference perturbation value corresponding to the frequency band generated using a bidirectional GRU For each frequency component Perform independent perturbation processing. For any given... Each component ( ), the perturbed components The calculation formula is: ; S33, All perturbed frequency components The final perturbation input data is obtained by reconstructing the time domain using inverse discrete wavelet transform. : .

[0026] S4. Perform module perturbation, construct a learnable module mask, and determine whether to retain the original module or replace it with an identity block based on the module mask state, generating a perturbation model; in this embodiment, this step specifically includes: S41. Define an updatable module parameter matrix. Calculate the binarized module mask : ; S42, for the first in the model Each module, if If it is, then replace it with an identity mapping; if If so, the original module will be retained.

[0027] S5. Perform joint optimization by inputting the perturbation into the perturbation model, calculating the difference between the output and the original model output, constructing the total loss function by combining the mask sparsity regularization term, and updating the parameters of the input mask and module mask in parallel: .

[0028] See the optimization model application stage in this embodiment. Figure 4 It includes: B1. Obtain the trained module mask and identify redundant modules with a mask value of 0; B2. Prune and reconstruct the original X-formers model, permanently replacing the identified redundant modules with identity mappings or other more efficient modules to obtain a lightweight optimized model. B3. Input the time series data to be predicted into the optimization model to obtain the final prediction result.

[0029] This method consists of a Time-Module Fusion Network (TMFN), a wavelet transform-based frequency domain input perturbation module, a module perturbation module, and a joint optimization strategy. In its implementation, the TMFN integrates dynamic data features with model structure information to generate multi-channel frequency domain masks and module masks corresponding to different frequency components (approximate and detail components). During the input perturbation stage, the time series is decomposed using discrete wavelet transform, and each scale component is independently perturbed and reconstructed based on the frequency domain mask, effectively suppressing high-frequency noise interference while preserving key trend information. In the module perturbation stage, the contribution of each internal component is analyzed through the module mask. In the joint optimization stage, all mask parameters are updated in parallel. This method enables fine-grained feature selection and model pruning in the time and frequency domains, significantly reducing computational cost and memory usage without compromising model prediction performance. It addresses the problems of severe redundancy, insufficient noise resistance, and limited interpretability in existing X-formers model architectures.

[0030] For the foregoing embodiments, in order to simplify the description, they are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, because according to this application, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions involved are not necessarily essential to this application.

[0031] The above embodiments describe the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Modifications and variations made by those skilled in the art without departing from the spirit and scope of the invention should be within the protection scope of the appended claims.

Claims

1. A module-level interpretability optimization method for time series Transformer models, characterized in that, The model includes a mask generation and perturbation stage and an optimization model application stage. The mask generation and perturbation stage includes the following steps: Step A1: Obtain the time series dataset and divide it into training sets, then load the pre-trained time series Transformer base model to be optimized; Step A2: Construct a time-module fusion network and generate a highly interpretable multi-channel input mask by utilizing the different frequency components of the input data and the model structure information; Step A3: Perform frequency domain input perturbation based on wavelet transform. Using the generated multi-channel input mask, refine the perturbation of the time series in the wavelet domain to generate a perturbation input that highlights key features. Step A4: Perform module perturbation, construct a learnable module mask, and decide whether to retain the original module or replace it with an identity mapping based on the state of the module mask to generate a perturbation model; Step A5: Perform joint optimization by feeding the perturbation input into the perturbation model, calculating the difference between the output and the original model output, constructing the total loss function by combining the mask sparsity regularization term, and updating the parameters of the input mask and the module mask in parallel.

2. The module-level interpretability optimization method for time series Transformer models as described in claim 1, characterized in that, Step A2 specifically includes the following steps: Step A21: Construct a feature fusion intermediate layer to process the input time series data. , ,in For time step, For the feature dimension; mapped to the module mask through a linear transformation. The appropriate dimension is then multiplied element-wise with the module mask to generate a hybrid feature that integrates the structural information of the model. , is represented as: ； Step A22: Utilizing hybrid features The query matrix is ​​generated through two independent linear transformations. Bond matrix ; directly on the raw input data Perform linear transformations to generate value matrices , respectively represented as: ； ； ； Step A23: Calculate the scaled dot product attention and input mask generation based on the query matrix. Key matrix Sum matrix Calculate the attention distribution by querying the matrix. Bond matrix The interaction captures the data and the correlation between modules, and applies this attention weight to the value matrix. The attention output is processed by a feedforward neural network (FFN) to obtain intermediate features used to generate the input mask. , is represented as: ； Step A24: Based on intermediate features ,pass Each independent projection head generates a corresponding set of frequency domain masks in parallel. ;in Corresponding to the low-frequency approximation component, Corresponding to high-frequency detail components at each level; for Any number within the range Each frequency component has its corresponding mask. The calculation formula is: 。 3. The module-level interpretability optimization method for time series Transformer models as described in claim 2, characterized in that, Step A3 specifically includes the following steps: Step A31: Based on discrete wavelet transform, convert the time series data... conduct Level decomposition, yielding A set of frequency components , represented as: ； Step A32: Use the mask generated in step A24 Combined with the reference perturbation value corresponding to the frequency band generated by the bidirectional GRU For each frequency component Perform independent perturbation processing; for Any number within the range One component, the perturbed component The calculation formula is: ； Step A33: Add all perturbated frequency components The final perturbation input data is obtained by reconstructing the time domain using inverse discrete wavelet transform. , represented as: 。 4. The module-level interpretability optimization method for time series Transformer models as described in claim 3, characterized in that, Step A4 specifically includes the following steps: Step A41: Define an updatable module parameter matrix Calculate the binarized module mask , represented as: ； Step A42: For the first in the model Each module, if If it is, then replace it with an identity mapping; if If so, the original module will be retained; Step A43: Construct the perturbation model Its output is .

5. The module-level interpretability optimization method for time series Transformer models as described in claim 4, characterized in that, The loss function for joint optimization in step A5 is expressed as: ； in, This indicates a measure of the difference between the output of the original model and the perturbation model; This is the output of the original model; For sparsity regularization, promote mask sparsity to identify the most critical parts; Ensure the validity of the reference disturbance values ​​corresponding to the frequency band generated by the bidirectional GRU.

6. The module-level interpretability optimization method for time series Transformer models as described in claim 5, characterized in that, The application phase of the optimization model includes the following steps: Step B1: Obtain the trained module mask and identify redundant modules with a mask value of 0; Step B2: Prune and reconstruct the original time series Transformer model, permanently replacing the identified redundant modules with identity mappings or other more efficient modules to obtain a lightweight optimized model; Step B3: Input the time series data to be predicted into the optimization model to obtain the final prediction result.

Images