A Transformer model evolution method and system

Abstract

The present invention provides a Transformer model evolution method and system, the method comprising: collecting transformer operation data, selecting training samples, and dividing to obtain historical data sets X 0 and the data set to be updated X 1; Build a time series feature fusion module to transform the sample data set X 0 and the data set to be updated X 1. Extract and fuse data features to obtain advanced fusion features Y 1. Build a data feature processing module and use the time series data difference method to obtain new feature fusion results Z ; Build a Transformer time series prediction model for similar feature datasets Y in Perform time series forecasting to obtain forecast results Z out , and update the weight parameters of the Transformer time series prediction model through incremental learning. The present invention solves the technical problems of poor model stability and flexibility, reduced prediction accuracy over time, and loss of feature information.

Description

Technical Field

[0001] The present invention relates to the field of power grid control, and in particular to a Transformer model evolution method and system. Background Art

[0002] The transformers of UHV DC converter stations play an important role in the power transmission process and are expensive. However, once manufactured by the manufacturer, their structure is fixed, and the positions and number of sensors that can be arranged are also determined, resulting in limited data collected by the sensors. For example, the transformer oil chromatography and the volume fraction of gas in the transformer oil change over time, which may cause digital drift and model performance degradation. The established digital twin transformer model is difficult to meet the current mapping relationship. Incremental learning of the model is performed to update the backward network parameters and adapt to the continuous evolution of the model over time, which is beneficial to the subsequent hidden fault detection of the transformer model.

[0003] It is a sequence-to-sequence model based on the self-attention mechanism. It has better parallelism and the ability to capture long-distance dependencies. It establishes the association between different positions in the input sequence through the self-attention mechanism without relying on step-by-step cyclic calculations. It has high performance advantages and uses new data features to compare with historical data. In incremental learning, it can improve the accuracy of the entire model through the time difference method and realize the dynamic evolution of the model. For example, in the existing invention patent application document "A temporal convolutional network model, model training and target recognition method and device" with publication number CN111898564A, the temporal convolutional network model adopted by the existing method includes: an input layer, a multi-convolution structure, a feature fusion layer and an output layer; the multi-convolution structure includes an image feature extraction substructure and a temporal feature extraction substructure; the input layer is used to receive the input image of the temporal convolutional network model, and import the image feature extraction substructure and the temporal feature extraction substructure respectively; the image feature extraction substructure is used to extract image features; the temporal feature extraction substructure is used to extract temporal features; the feature fusion layer is used to fuse image features and temporal features to obtain data features; the output layer is used to classify the data features and output the classification results. And the existing invention patent application document "Clustering method and device based on dual-dimensional multivariable time series feature fusion" with publication number CN118035778A, the existing method includes: enhancing the original time series data through the data enhancement module to construct sample pairs for contrastive learning; encoding the data through the encoding module to obtain feature representation, the encoding module includes a time series dimension feature extraction module and a variable dimension feature extraction module, which are respectively used to mine the dependency between time series according to the time series dimension and the dependency between variables according to the variable dimension; the deep clustering module is used to cluster the encoded vectors and evaluate the model effect. In the entire model training process, the multi-task training method is adopted to unify the contrastive learning and clustering tasks into the same training process.

[0004] However, the models in the aforementioned existing solutions suffer from catastrophic forgetting of the neural network, resulting in poor model stability and flexibility; at the same time, measurement data drift will cause performance degradation of the model, reducing the model accuracy, and process data drift, mutation, and strong nonlinear factors will cause the model prediction accuracy to decrease over time; at the same time, the feature data extracted by the encoder is high-level feature information with a high semantic information capacity, and an excessively high expansion rate can easily lead to the loss of feature information.

[0005] In summary, the existing technology has technical problems such as poor model stability and flexibility, reduced prediction accuracy over time, and loss of feature information. Summary of the invention

[0006] The technical problem to be solved by the present invention is: how to solve the technical problems in the prior art of poor model stability and flexibility, reduced prediction accuracy over time, and loss of feature information.

[0007] The present invention adopts the following technical solutions to solve the above technical problems: A Transformer model evolution method includes:

[0008] S1. Collect transformer operation data according to the preset time series, select training samples from the transformer operation data, and divide the training samples into historical data sets X 0. Dataset to be updated X 1;

[0009] S2. Build a time series feature fusion module to perform historical data sets X 0 and the data set to be updated X 1. Perform data feature extraction and fusion operations to obtain advanced fusion features Y 1;

[0010] S3. Build a data feature processing module and use the time series data difference method to process advanced fusion features Y 1. Get the new feature fusion result Z , obtain the changing trend of new and old data characteristics;

[0011] S4. Based on advanced fusion features Y 1. New feature fusion results Z , processed to obtain similar feature data sets Y in , build a Transformer time series prediction model, and use similar feature datasets Y in Perform time series prediction and obtain prediction results Z out , perform incremental learning to update the weight parameters of the Transformer time series prediction model.

[0012] The present invention can use the Transformer model evolution method to realize the model prediction update and dynamic evolution of transformer operation data. The historical data set and the data set to be updated are feature fused through the time series feature fusion module, and the similarity between the historical data set features and the data set features to be updated is measured and judged, and the similarity is input into the Transformer time series prediction model. Incremental learning and prediction are performed in the continuously updated time series data features to realize the dynamic evolution result of the Transformer time series prediction model.

[0013] In a more specific technical solution, S2 includes:

[0014] S21. Build a time series feature fusion module based on the GRU model to transform the historical data set X 0 is input to the low-level feature layer to obtain and output the low-level features of historical samples and save the shallow network parameter weights ω 1. Output high-level features of historical samples through the high-level feature layer Y 0, save the deep network parameter weights ω 2;

[0015] S22. Using the data set to be updated X 1. Reuse the low-level feature extraction neural network of the time series feature fusion module, and save and initialize the deep network parameter weights when extracting the deep feature neural network. ω 2. Obtain and use the new data set to train the deep feature extraction neural network again to obtain the new sample high-level feature layer and the new deep feature layer weights ω 3;

[0016] S23, using the update gate and weight gate of the time series feature fusion module, the historical data set X 0.0 deep network parameter weights ω 2. With the dataset to be updated X New deep feature layer weights for new samples in 1 ω 3. As the input data of the time series feature fusion module, the time series feature fusion module is used to generate the fusion feature layer weights ω 4; Use fusion feature layer weights ω 4 Replace the deep network parameter weights ω 2. Use the final feature extraction network to generate and output high-level fusion features Y 1.

[0017] The present invention constructs a temporal feature fusion module. While alleviating the catastrophic forgetting of the neural network, the feature fusion module can adaptively fuse the high-level feature representations of new and old tasks, thereby ensuring the stability and flexibility of the model.

[0018] In a more specific technical solution, in S21, the following formula is used to construct a temporal feature fusion module based on the GRU model:

[0019] (1)

[0020] (2)

[0021] (3)

[0022] (4)

[0023] In the formula, Represents the probability vector corresponding to the features of the original sample, Represents the probability vector corresponding to the features of the new sample; is the probability vector corresponding to the features of the original sample The first pre-trained weights of is the probability vector corresponding to the features of the original sample The second pre-trained weights of is the probability vector corresponding to the features of the original sample The third pre-trained weight of ; The probability vector corresponding to the features of the new sample The first pre-trained weights of The probability vector corresponding to the features of the new sample The second pre-trained weights of The probability vector corresponding to the features of the new sample The third pre-trained weight of ; It is the initial fusion feature; For the initial fusion feature The first pre-trained weights of For the initial fusion feature The second pre-trained weights of For the initial fusion feature The third pre-trained weight of ; for function; For the The high-level feature layer weights of the task input, For the i+1 The high-level feature layer weights of the task input, is the number of tasks; is the new weight after feature fusion, Represents the feature fusion operation.

[0024] In a more specific technical solution, S3 includes:

[0025] S31. Fusion results of new features Z The feature data in is processed by standardized difference to obtain high-level fusion features Y 1 as input, according to the target time t Corresponding high-level fusion data feature input Y(t) , New feature fusion feature output Z(t) And before preset i Data corresponding to each task, calculate the target time t Inside front i High-level fusion data feature input corresponding to each task Y(t - i) With target time t Inside front iThe new features corresponding to the task are fused with data features to output Z(t - i) time difference variables, intermediate time difference variables and characteristics of data to be processed;

[0026] S32. Advanced features from historical samples Y 0, select and advanced fusion features Y 1 The sample with the highest similarity builds the new feature fusion result Z .

[0027] The present invention utilizes the time series data differencing method to reduce the impact of measurement data drift on model performance degradation, effectively capture the dynamic changes of process data over time, and update model parameters in real time to ensure model accuracy, thus solving the problem of model prediction accuracy decreasing over time due to process data drift, mutation, and strong nonlinearity.

[0028] In a more specific technical solution, in S31, the following logic is used to calculate the time difference variable, the intermediate time difference variable and the characteristics of the data to be processed:

[0029] (5)

[0030] (6)

[0031] (7)

[0032] (8)

[0033] In the formula, Target time t and before i The time difference variable corresponding to each task, in Represents input; Target time t and before i Each task corresponds to the intermediate time difference variable of the high-level fusion feature. Target time t and before i The intermediate time difference variable of the new feature fusion result corresponding to each task; for Time series forecasting models; e To calculate the error vector.

[0034] The time series data difference method adopted in the present invention has the ability to analyze complex nonlinear data, can continuously update model parameters according to new data, and can learn and optimize model parameters in real-time data processing, thereby realizing dynamic modeling and real-time prediction of data.

[0035] In a more specific technical solution, S32 includes:

[0036] Using the Euclidean distance determination logic, the local model parameter φ is obtained based on the local polynomial fitting, and the high-level features of the historical samples are used to determine the local model parameter φ. Y Advanced Fusion Data in 0 Y , process the new feature fusion results Z :

[0037] Z= Y φ;

[0038] When new features are input, the following logic is used for iterative operation:

[0039] (9)

[0040] In the formula, High-level features for historical samples Target time t Corresponding data feature input; Advanced fusion features The current j feature input, n Represents the total number of feature inputs; For the j The Euclidean distance corresponding to the feature input; is the norm of 2.

[0041] In a more specific technical solution, S4 includes:

[0042] S41, using a one-dimensional convolutional network with a convolution kernel of 3, the similar feature data set Y in Project it into the model space, and then perform the fusion operation of time encoding and feature encoding to obtain the input encoding vector:

[0043] (10)

[0044] In the formula, is the feature data of similar feature datasets, is the timestamp corresponding to the feature data, is the input encoding vector;

[0045] S42, constructing a Transformer time series prediction model, wherein the Transformer time series prediction model includes: an encoder layer, an adaptive multi-scale attention layer, and a decoder layer, adding dilated convolution to the backbone network of the Transformer time series prediction model, and using the decoder layer to perform upsampling in the preset convolution layer; setting a jump component in the encoder layer and the decoder layer;

[0046] S43, input encoding vector x inInput into the Transformer time series prediction model to obtain the model prediction results. According to the model prediction results and the actual results, the Transformer time series prediction model is obtained and trained using the binary cross entropy loss.

[0047] S44. Add the output results of the Transformer time series prediction model to the historical data set X 0, continuously update historical data sets X 0, iteratively execute S2, S3 and S4 to perform incremental learning and update the weight parameters of the Transformer time series prediction model.

[0048] In a more specific technical solution, in S42, in the adaptive multi-scale attention layer, the input encoding vector is processed using the following logic x in Process the difference scale feature information:

[0049] (11)

[0050] (12)

[0051] (13)

[0052] (14)

[0053] (15)

[0054] (16)

[0055] (17)

[0056] In the formula, The input encoding vector x in The corresponding results of the first group of channels are: The input encoding vector x in The corresponding results of the second group of channels are: The input encoding vector x in The corresponding results of the third group of channels are: The input encoding vector x in The corresponding results for the 4th group of channels are: The input encoding vector x in The corresponding results for the 5th group of channels are: The input encoding vector x inThe corresponding results of the 6th group of channels; represents a 1×1 convolutional layer; represents a 3×1 convolutional layer with a dilation rate of 2; represents a 1×3 convolutional layer with a dilation rate of 2; represents a 3×1 convolutional layer with a dilation rate of 4. represents a 1×3 convolutional layer with a dilation rate of 4; represents a 3×1 convolutional layer with a dilation rate of 6. represents a 1×3 convolutional layer with a dilation rate of 6; represents a 3×1 convolutional layer with a dilation rate of 8. Represents a 1×3 convolutional layer with a dilation rate of 8; Indicates batch normalization operation on the output of convolution; Indicates that the normalized output Activation function processing; Indicates the result of data splicing; Indicates data concatenation.

[0057] The adaptive multi-scale attention layer constructed by the present invention optimizes the feature extraction operation for global feature-related information through multiple channels; since the feature data extracted by the encoder layer is high-level feature information with high semantic information capacity, too high expansion rate can easily lead to the loss of feature information, so the processing of high-level fusion features is refined by reducing the expansion rate, and the number of expansion convolution layers is increased to obtain rich feature information, thereby improving the ability to extract feature information.

[0058] In a more specific technical solution, in step S43, the binary cross entropy loss is determined using the following logic: :

[0059] (18)

[0060] In the formula, is the number of data, for i In the tasks, from 0 to The cumulative result of For the i The data features of the tasks are the real results. Indicates i The logarithm of the actual results of the data features of the tasks, For the i The data feature prediction results of each task, Represents the real result, Represents the prediction result, Indicates i The input encoding vector corresponding to each task, Represents high-level fusion features Middle i The feature input corresponding to each task.

[0061] In a more specific technical solution, a Transformer model evolution system includes:

[0062] The training sample collection and division module is used to collect transformer operation data according to the preset time series, select training samples from the transformer operation data, and divide the training samples into historical data sets X 0. Dataset to be updated X 1;

[0063] Time series feature fusion module, used to fusion historical data sets X 0 and the data set to be updated X 1. Perform data feature extraction and fusion operations to obtain advanced fusion features Y 1. The time series feature fusion module is connected to the training sample collection and division module;

[0064] Data feature processing module, used to process advanced fusion features using time series data difference method Y 1. Get the new feature fusion result Z , obtain the changing trend of new and old data features, and connect the data feature processing module with the time series feature fusion module;

[0065] Model parameter update module to update the model according to the high-level fusion features Y 1. New feature fusion results Z , processed to obtain similar feature data sets Y in , build a Transformer time series prediction model, and use similar feature datasets Y in Perform time series prediction and obtain prediction results Z out , perform incremental learning to update the weight parameters of the Transformer time series prediction model, and the model parameter update module is connected to the data feature processing module.

[0066] Compared with the prior art, the present invention has the following advantages:

[0067] The present invention can use the Transformer model evolution method to realize the model prediction update and dynamic evolution of transformer operation data. The historical data set and the data set to be updated are feature fused through the time series feature fusion module, and the similarity between the historical data set features and the data set features to be updated is measured and judged, and the similarity is input into the Transformer time series prediction model. Incremental learning and prediction are performed in the continuously updated time series data features to realize the dynamic evolution result of the Transformer time series prediction model.

[0068] The present invention constructs a temporal feature fusion module. While alleviating the catastrophic forgetting of the neural network, the feature fusion module can adaptively fuse the high-level feature representations of new and old tasks, thereby ensuring the stability and flexibility of the model.

[0069] The present invention utilizes the time series data differencing method to reduce the impact of measurement data drift on model performance degradation, effectively capture the dynamic changes of process data over time, and update model parameters in real time to ensure model accuracy, thus solving the problem of model prediction accuracy decreasing over time due to process data drift, mutation, and strong nonlinearity.

[0070] The time series data difference method adopted in the present invention has the ability to analyze complex nonlinear data, can continuously update model parameters according to new data, and can learn and optimize model parameters in real-time data processing, thereby realizing dynamic modeling and real-time prediction of data.

[0071] The adaptive multi-scale attention layer constructed by the present invention optimizes the feature extraction operation for global feature-related information through multiple channels; since the feature data extracted by the encoder layer is high-level feature information with high semantic information capacity, too high expansion rate can easily lead to the loss of feature information, so the processing of high-level fusion features is refined by reducing the expansion rate, and the number of expansion convolution layers is increased to obtain rich feature information, thereby improving the ability to extract feature information.

[0072] The present invention solves the technical problems existing in the prior art of poor model stability and flexibility, reduced prediction accuracy over time, and loss of feature information. BRIEF DESCRIPTION OF THE DRAWINGS

[0073] Figure 1 A schematic diagram of basic steps of a Transformer model evolution method according to Embodiment 1 of the present invention;

[0074] Figure 2 This is a schematic diagram of specific steps for performing time series feature fusion according to Embodiment 1 of the present invention;

[0075] Figure 3 This is a schematic diagram of the structure of a temporal feature fusion module according to Embodiment 1 of the present invention;

[0076] Figure 4 This is a schematic diagram of specific steps of modeling operation in the process of building a time series data difference module in Example 1 of the present invention;

[0077] Figure 5 This is a schematic diagram of the structure of a time series data difference module according to Embodiment 1 of the present invention;

[0078] Figure 6A schematic diagram of specific steps of result prediction and weight update learning in Example 1 of the present invention;

[0079] Figure 7 This is a schematic diagram of the structure of a Transformer timing prediction module according to Embodiment 1 of the present invention;

[0080] Figure 8 This is a schematic diagram of the structure of the adaptive multi-scale attention module of Example 1 of the present invention. DETAILED DESCRIPTION

[0081] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in combination with the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0082] Example 1

[0083] like Figure 1 As shown in Figure 1, a Transformer model evolution method includes the following basic steps:

[0084] Step S1: Collect transformer operation data according to a preset time series, select training samples from the transformer operation data, and divide the training samples into historical data sets X 0. Dataset to be updated X 1;

[0085] In this embodiment, the operating data of the transformer is collected as a historical data set. X 0; In this embodiment, the transformer includes but is not limited to: a DC converter transformer; after the DC converter transformer runs for a period of time, a new data set is collected as the data set to be updated X 1; In this embodiment, the historical data set X 0 as the training set and the data set to be updated X 1 as the test set;

[0086] In this embodiment, the operating data includes but is not limited to: oil temperature, gas concentration;

[0087] Step S2: Construct a time series feature fusion module to perform X 0 and the data set to be updated X 1. Perform data feature extraction and fusion operations to obtain advanced fusion features Y 1;

[0088] like Figure 2 and Figure 3As shown, in this embodiment, step S2 of performing time series feature fusion also includes the following specific steps:

[0089] Step S21: construct a time series feature fusion module based on the GRU model to combine the historical data set X 0 is input to the low-level feature layer to obtain and output the low-level features of historical samples and save the shallow network parameter weights ω 1. Output high-level features of historical samples through the high-level feature layer Y 0, save the deep network parameter weights ω 2;

[0090] In this embodiment, the feature extraction neural network CNN is used as a carrier and is divided into two layers of convolutional neural networks, shallow and deep. In this embodiment, the first few layers extract shallow features and the last few layers extract deep features.

[0091] Step S22: Using the data set to be updated X 1. Reuse the low-level feature extraction neural network of the time series feature fusion module, and save and initialize the deep network parameter weights when extracting the deep feature neural network. ω 2. Obtain and use the new data set to train the deep feature extraction neural network again to obtain the new sample high-level feature layer and the new deep feature layer weights ω 3;

[0092] In this embodiment, the new data set to be updated X 1Reuse the low-level feature extraction neural network, save the current high-level feature layer weights during the deep feature extraction process, and then randomly initialize the high-level feature layer weights to obtain the deep network parameter weights ω 2. Use the new data set to retrain the deep feature extraction neural network to obtain the new sample high-level feature layer and the new deep feature layer weights ω 3;

[0093] Step S23: Use the update gate and weight gate of the time series feature fusion module to transform the historical data set X 0.0 deep network parameter weights ω 2. With the dataset to be updated X New deep feature layer weights for new samples in 1 ω 3. As the input data of the time series feature fusion module, the time series feature fusion module is used to generate the fusion feature layer weights ω 4; Use fusion feature layer weights ω 4 Replace the deep network parameter weights ω 2. Use the final feature extraction network to generate and output high-level fusion features Y 1;

[0094] In this embodiment, a time series feature fusion module is constructed based on the GRU model, and the saved historical data set is converted into X 0 samples' deep network parameter weights ω 2 and the currently learned data set to be updated X New deep feature layer weights for new samples in 1 ω 3. As two inputs of the time series feature fusion module, they are input into the time series feature fusion module to generate the fusion feature layer weights ω 4; Use fusion feature layer weights ω 4Replace historical data set X 0 samples' deep network parameter weights ω 2. Obtain and use the final feature extraction network to output high-level fusion features Y 1.

[0095] In this embodiment, the following formula is used to construct a temporal feature fusion module based on the GRU model:

[0096] (1)

[0097] (2)

[0098] (3)

[0099] (4)

[0100] in, Represents the probability vector corresponding to the features of the original sample, Represents the probability vector corresponding to the features of the new sample; for function; For the The high-level feature layer weights of the task input, For the i+1 The high-level feature layer weights of the task input, is the number of tasks.

[0101] Step S3: Construct a data feature processing module and use the time series data difference method to process advanced fusion features. Y 1. Get the new feature fusion result Z , obtain the changing trend of new and old data characteristics;

[0102] like Figure 4 and Figure 5 As shown, in this embodiment, the modeling operation in the process of constructing the time series data difference module includes the following specific steps:

[0103] Step S31: Fusion results of new features Z The feature data in is processed by standardized difference to obtain high-level fusion features Y 1 as input, according to the target time t Corresponding high-level fusion data feature input Y(t) , New feature fusion feature output Z (t) And before preset Data corresponding to each task, calculate the target time t Inside front i High-level fusion data feature input corresponding to each task Y(t - i) With target time t Inside front i The new features corresponding to the task are fused with data features to output Z(t - i) time difference variables, intermediate time difference variables and characteristics of data to be processed;

[0104] In this embodiment, the feature data is subjected to standardized difference processing, and the high-level fusion feature output in the above step S2 is taken. Y 1 as input, calculate the target time Inside front High-level fusion data feature input corresponding to each task Y(t - i) With target time Inside front The new features corresponding to the task are fused with data features to output Z(t - i) The time difference variable, the intermediate time difference variable and the characteristics of the data to be processed; in this embodiment, the target time Inside front High-level fusion data feature input corresponding to each task Y(t - i) Fusion of data features with new features output Z(t - i) The time difference variable and the intermediate time difference variable are expressed as follows: (5), (6) and (7), and then the intermediate time difference variable and the characteristics of the data to be processed are expressed as follows: (5)

[0105] (6)

[0106] (7)

[0107] (8).

[0108] Step S32: High-level features from historical samples Y 0, select and advanced fusion features Y 1 The sample with the highest similarity builds the new feature fusion result Z ;

[0109] In this embodiment, high-level features of historical samples are Y 0 selection and advanced fusion features Y 1. The sample with the highest similarity builds a similar data set as the new feature fusion result Z , the similarity evaluation standard is the Euclidean distance, and the evaluation formula is as follows (9):

[0110] (9).

[0111] Use local polynomial fitting to find the local model parameter φ and calculate the new feature fusion result corresponding to the current input Z = Y φ; When a new feature is input, the above step S31 is iterated repeatedly.

[0112] Step S4: Based on high-level fusion features Y 1. New feature fusion results Z , processed to obtain similar feature data sets Y in , build a Transformer time series prediction model, and use similar feature datasets Y in Perform time series prediction and obtain prediction results Z out , perform incremental learning to update the weight parameters of the Transformer time series prediction model.

[0113] like Figure 6 and Figure 7 As shown, in this embodiment, the step S4 of result prediction and updating weight learning also includes the following specific steps:

[0114] Step S41: Use a one-dimensional convolutional network with a convolution kernel of 3 to transform the similar feature data set Y in Project it into the model space, then perform the time coding and feature coding fusion operation to obtain the input coding vector;

[0115] In this embodiment, a fusion of time coding and feature coding is used as input coding. First, a one-dimensional convolutional network with a convolution kernel of 3 is used to transform similar feature data sets. Y in Project it into the model space, and then fuse the time code with the feature code to get the input code vector:

[0116] (10).

[0117] Step S42, constructing a Transformer time series prediction model, wherein the Transformer time series prediction model includes: an encoder layer, an adaptive multi-scale attention layer, and a decoder layer, adding dilated convolution to the backbone network of the Transformer time series prediction model, and using the decoder layer to perform upsampling in the preset convolution layer; setting a jump component in the encoder layer and the decoder layer;

[0118] like Figure 8 As shown, in this embodiment, the encoder layer can use, for example, ResNet50, as a multi-scale feature extraction backbone network, and add dilated convolution, the step size can be set to, for example, 8, and the decoder layer can be upsampled in, for example, 4 groups of convolutional layers; jump connection technology is added to the corresponding layers of the encoder and decoder.

[0119] In this embodiment, the adaptive multi-scale attention layer passes the data extracted by the encoder layer through 1×1 convolution and 3×3 dilated convolution respectively. The dilation rate can be set to, for example, 2, 4, 6, and 8. A global average pooling is used for inference to splice 6 groups of features into one group of data. Multi-scale features are extracted and fused, and a self-attention mechanism is added to further fuse multi-scale features. In the adaptive multi-scale attention layer of this embodiment, 6 groups of feature information of different scales are used for processing. The mathematical descriptions of these 6 groups of modules of different scales are as follows: (11) to (16):

[0120] (11)

[0121] (12)

[0122] (13)

[0123] (14)

[0124] (15)

[0125] (16)

[0126] (17).

[0127] Step S43: Input the encoding vector x in Input into the Transformer time series prediction model to obtain the model prediction results. According to the model prediction results and the actual results, the Transformer time series prediction model is obtained and trained using the binary cross entropy loss.

[0128] In this embodiment, the binary cross entropy loss is defined as the following formula (18):

[0129] (18).

[0130] Step S44: Add the output results of the Transformer time series prediction model to the historical data set X 0, continuously update historical data sets X 0, iteratively execute S2, S3 and S4 to perform incremental learning and update the weight parameters of the Transformer time series prediction model.

[0131] In summary, the present invention can use the Transformer model evolution method to realize the model prediction update and dynamic evolution of transformer operation data. The historical data set and the data set to be updated are feature fused through the time series feature fusion module, and the similarity between the historical data set features and the data set features to be updated is measured and judged, and input into the Transformer time series prediction model, and incremental learning prediction is performed in the continuously updated time series data features to realize the dynamic evolution result of the Transformer time series prediction model.

[0132] The present invention constructs a temporal feature fusion module. While alleviating the catastrophic forgetting of the neural network, the feature fusion module can adaptively fuse the high-level feature representations of new and old tasks, thereby ensuring the stability and flexibility of the model.

[0133] The present invention utilizes the time series data differencing method to reduce the impact of measurement data drift on model performance degradation, effectively capture the dynamic changes of process data over time, and update model parameters in real time to ensure model accuracy, thus solving the problem of model prediction accuracy decreasing over time due to process data drift, mutation, and strong nonlinearity.

[0134] The time series data difference method adopted in the present invention has the ability to analyze complex nonlinear data, can continuously update model parameters according to new data, and can learn and optimize model parameters in real-time data processing, thereby realizing dynamic modeling and real-time prediction of data.

[0135] The adaptive multi-scale attention layer constructed by the present invention optimizes the feature extraction operation for global feature-related information through multiple channels; since the feature data extracted by the encoder layer is high-level feature information with high semantic information capacity, too high expansion rate can easily lead to the loss of feature information, so the processing of high-level fusion features is refined by reducing the expansion rate, and the number of expansion convolution layers is increased to obtain rich feature information, thereby improving the ability to extract feature information.

[0136] The present invention solves the technical problems existing in the prior art of poor model stability and flexibility, reduced prediction accuracy over time, and loss of feature information.

[0137] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A Transformer model evolution method, characterized in that: The method comprises: S1. Collect transformer operation data according to a preset time series, select training samples from the transformer operation data, and divide the training samples into historical data sets. X 0. Dataset to be updated X 1; S2. Construct a time series feature fusion module to perform X 0 and the data set to be updated X 1. Perform data feature extraction and fusion operations to obtain advanced fusion features Y 1; Wherein, the S2 includes: S21, construct the time series feature fusion module based on the GRU model, and transform the historical data set X 0 is input to the low-level feature layer to obtain and output the low-level features of historical samples and save the shallow network parameter weights ω 1. Output high-level features of historical samples through the high-level feature layer Y 0, save the deep network parameter weights ω 2; S22, using the data set to be updated X 1. Reuse the low-level feature extraction neural network of the temporal feature fusion module, and save and initialize the deep network parameter weights when extracting the deep feature neural network. ω 2. Obtain and use the new data set to train the deep feature extraction neural network again to obtain the new sample high-level feature layer and obtain the new deep feature layer weights ω 3; S23, using the update gate and weight gate of the temporal feature fusion module to transform the historical data set X 0 in the deep network parameter weights ω 2. With the data set to be updated X 1 The new deep feature layer weights of the new samples in ω 3, as the input data of the temporal feature fusion module, so as to generate the fusion feature layer weight by using the temporal feature fusion module ω 4; Use the fusion feature layer weights ω 4 Replace the deep network parameter weights ω 2. Use the final feature extraction network to generate and output the advanced fusion features Y 1; S3. Construct a data feature processing module and use the time series data difference method to process the advanced fusion features. Y 1. Get the new feature fusion result Z , obtain the changing trend of new and old data characteristics; Wherein, the S3 includes: S31, fusion result of the new feature Z The feature data in is processed by standardized difference, and the advanced fusion feature is obtained. Y 1 as input, according to the target time t Corresponding high-level fusion data feature input Y(t) , New feature fusion feature output Z (t) And before preset i The data corresponding to each task is used to calculate the target time. t Inside front i The high-level fusion data feature input corresponding to the task Y(ti) With the target time t Inside front i The new features corresponding to the task are fused into data feature output Z(ti) time difference variables, intermediate time difference variables and characteristics of data to be processed; S32. Advanced features from historical samples Y 0, select the advanced fusion feature Y 1 The sample with the highest similarity is used to construct the new feature fusion result Z ; S4. According to the high-level fusion features Y 1. The new feature fusion results Z , processed to obtain similar feature data sets Y in , build a Transformer time series prediction model, and use the similar feature dataset Y in Perform time series prediction and obtain prediction results Z out , perform incremental learning to update the weight parameters of the Transformer time series prediction model.

2. A Transformer model evolution method according to claim 1, characterized in that: In S21, the temporal feature fusion module is constructed based on the GRU model using the following formula: (1) (2) (3) (4) In the formula, Represents the probability vector corresponding to the features of the original sample, A probability vector representing the features of the new sample; is the probability vector corresponding to the feature of the original sample The first pre-trained weights of is the probability vector corresponding to the feature of the original sample The second pre-trained weights of is the probability vector corresponding to the feature of the original sample The third pre-trained weight of ; is the probability vector corresponding to the feature of the new sample The first pre-trained weights of is the probability vector corresponding to the feature of the new sample The second pre-trained weights of is the probability vector corresponding to the feature of the new sample The third pre-trained weight of ; It is the initial fusion feature; The initial fusion feature The first pre-trained weights of The initial fusion feature The second pre-trained weights of The initial fusion feature The third pre-trained weight of ; for function; For the The high-level feature layer weights of the task input, For the i+1 The high-level feature layer weights of the task input, is the number of tasks; is the new weight after feature fusion, Represents the feature fusion operation.

3. A Transformer model evolution method according to claim 1, characterized in that: In S31, the time difference variable, the intermediate time difference variable and the feature of the data to be processed are calculated using the following logic: (5) (6) (7) (8) In the formula, The target time t and before i The time difference variable corresponding to each of the tasks, in Represents input; The target time t and before i The intermediate time difference variables of the tasks corresponding to the high-level fusion features, The target time t and before i The intermediate time difference variable of the new feature fusion result corresponding to each of the tasks; for Time series forecasting models; e To calculate the error vector.

4. The Transformer model evolution method according to claim 1, characterized in that: The S32 includes: S321, using the Euclidean distance determination logic, the local model parameter φ is obtained according to the local polynomial fitting, and the high-level features of the historical samples are used to determine the local model parameter φ. Y Advanced Fusion Data in 0 Y , process the new feature fusion result Z : Z= Y φ; S322. When a new feature is input, the following logic is used for iterative operation: (9) In the formula, The high-level features of the historical samples The target time stated in t Corresponding data feature input; For the high-level fusion feature The current j feature input, n Represents the total number of feature inputs; For the j The Euclidean distance corresponding to the feature input; is the norm of 2.

5. The Transformer model evolution method according to claim 1, characterized in that: The S4 includes: S41, using a one-dimensional convolutional network with a convolution kernel of 3, the similar feature data set Y in Project it into the model space, and then perform the fusion operation of time encoding and feature encoding to obtain the input encoding vector: (10) In the formula, is the feature data of the similar feature data set, is the timestamp corresponding to the feature data, encoding a vector for the input; S42, constructing the Transformer time series prediction model, wherein the Transformer time series prediction model includes: an encoder layer, an adaptive multi-scale attention layer and a decoder layer, adding dilated convolution to the backbone network of the Transformer time series prediction model, and using the decoder layer to perform upsampling in a preset convolution layer; and setting a jump component in the encoder layer and the decoder layer; S43, the input encoding vector x in Input into the Transformer time series prediction model to obtain the model prediction result, and according to the model prediction result and the actual result, obtain and train the Transformer time series prediction model using the binary cross entropy loss; S44, adding the output result of the Transformer time series prediction model to the historical data set X 0, continuously update the historical data set X 0, iteratively execute S2, S3 and S4 to perform incremental learning and update the weight parameters of the Transformer time series prediction model.

6. A Transformer model evolution method according to claim 5, characterized in that: In S42, in the adaptive multi-scale attention layer, the input encoding vector is processed using the following logic: x in Process the difference scale feature information: (11) (12) (13) (14) (15) (16) (17) In the formula, The input encoding vector x in The corresponding results of the first group of channels are: The input encoding vector x in The corresponding results of the second group of channels are: The input encoding vector x in The corresponding results of the third group of channels are: The input encoding vector x in The corresponding results for the 4th group of channels are: The input encoding vector x in The corresponding results for the 5th group of channels are: The input encoding vector x in The corresponding results of the 6th group of channels; represents a 1×1 convolutional layer; represents a 3×1 convolutional layer with a dilation rate of 2; represents a 1×3 convolutional layer with a dilation rate of 2; represents a 3×1 convolutional layer with a dilation rate of 4. represents a 1×3 convolutional layer with a dilation rate of 4; represents a 3×1 convolutional layer with a dilation rate of 6. represents a 1×3 convolutional layer with a dilation rate of 6; represents a 3×1 convolutional layer with a dilation rate of 8. represents a 1×3 convolutional layer with a dilation rate of 8; Indicates batch normalization operation on the output of convolution; Indicates that the normalized output Activation function processing; Indicates the result of data splicing; Indicates data concatenation.

7. A Transformer model evolution method according to claim 5, characterized in that: In S43, the binary cross entropy loss is determined using the following logic: : (18) In the formula, is the number of data, for i In the tasks, from 0 to The cumulative result of For the i The data characteristics of the tasks are the real results. Indicates i The logarithm of the true results of the data features of the task, For the i The prediction results of the data features of the task, Represents the real result, Represents the prediction result, Indicates i The input encoding vector corresponding to the task, Represents high-level fusion features Middle i The feature input corresponding to the task.

8. A Transformer model evolution system, used to execute a Transformer model evolution method according to any one of claims 1 to 7, characterized in that: The system comprises: The training sample collection and division module is used to collect transformer operation data according to a preset time series, select training samples from the transformer operation data, and divide the training samples into historical data sets. X 0. Dataset to be updated X 1; A time series feature fusion module is used to fusion the historical data set X 0 and the data set to be updated X 1. Perform data feature extraction and fusion operations to obtain advanced fusion features Y 1. The temporal feature fusion module is connected to the training sample acquisition and division module; A data feature processing module is used to process the advanced fusion features using a time series data difference method. Y 1. Get the new feature fusion result Z , obtaining the changing trend of new and old data features, the data feature processing module is connected with the time series feature fusion module; A model parameter updating module is used to update the model parameters according to the high-level fusion features. Y 1. The new feature fusion results Z , processed to obtain similar feature data sets Y in , build a Transformer time series prediction model, and use the similar feature dataset Y in Perform time series prediction and obtain prediction results Z out , perform incremental learning to update the weight parameters of the Transformer time series prediction model, and the model parameter updating module is connected to the data feature processing module.

Images