A method, apparatus and storage medium for establishing a transfer learning model for an industrial control system
By using the VAE module and multi-head interactive attention module of the TransNet model, the data of the industrial control system is mapped to the same feature space and the trend features are extracted, which solves the problem of the lack of transferability between different industrial control system models and improves the accuracy and consistency of anomaly detection.
Patent Information
- Application Number
- CN202311226787.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-09-21
- Publication Date
- 2026-11-17
- Estimated Expiration
- 2043-09-21
AI Technical Summary
Different industrial control systems have different feature spaces and complex time dimensions, resulting in models that are not transferable and difficult to maintain detection accuracy on different industrial control systems.
The TransNet model is adopted, and the source and target domain data are mapped to the same feature space through the VAE module. The multi-head interactive attention module is used to extract and cross-extract the changing trend features of industrial control data, and a loss function is constructed for training to generate a transferable industrial control system transfer learning model.
It realizes the transferability of the transfer learning model of industrial control system, and improves the accuracy and consistency of anomaly detection on different industrial control systems.
Smart Images

Figure CN117371514B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of industrial control system anomaly detection technology, specifically to a method, apparatus, and storage medium for establishing a transfer learning model for an industrial control system. Background Technology
[0002] Industrial control systems are widely used in various fields, including power, chemical, petroleum, and natural gas. As these systems increasingly connect to the internet, their security is threatened by malicious parameter injection, malicious status commands, and control process disruptions. Therefore, improving the security of industrial control systems is urgently needed. Anomaly detection in industrial operating data is a crucial branch of industrial control system security research. Regardless of the type of attack, an industrial control system will ultimately experience operational data that deviates from the system's logic, leading to various dangers. Here, industrial control system anomaly detection refers to the timely discovery of operational data anomalies and the provision of early warnings in the event of a hazard.
[0003] While anomaly detection trained on specific field data may perform well, its accuracy on similar systems is difficult to guarantee. This is mainly due to two reasons: 1. The diversity and complexity of industrial control systems themselves; 2. The difficulty in obtaining attacked data in the field. A full dataset of an industrial control system is collected at fixed time intervals, resulting in a two-dimensional sample of the system's operational data, which contains the system's business logic. However, even within the same type of industrial control system, significant differences exist due to variations in scenarios, brands, sensor value ranges, and the sorting of sensor data during acquisition. The diversity manifests in the spatial dimension—different industrial control systems have different feature spaces—while the complexity lies in the temporal dimension, as the internal sensors of an industrial control system dynamically change over time during different control processes. That is, under different control strategies, the correlation between sensors and the distribution of sensor data change with the industrial control business. The diverse spatial dimensions make it difficult for the model to extract potential common features between the operational data of similar industrial control systems, while the complex temporal dimension makes it difficult for the model to understand the business logic constituted by the system's control strategies, ultimately resulting in a model lacking transferability. Summary of the Invention
[0004] In view of this, the purpose of this invention is to provide a method, apparatus and storage medium for establishing a transfer learning model for an industrial control system, in order to solve the problem that in the prior art, different industrial control systems have different feature spaces and are highly complex in the time dimension. During the different control processes of the industrial control system, the internal sensors will exhibit dynamic changes over time. That is, under different control strategies, the correlation between sensors and the distribution of sensor data will change with the changes in industrial control business, resulting in the trained model not having transferability.
[0005] According to a first aspect of the present invention, a method for establishing a transfer learning model for an industrial control system is provided, the method comprising:
[0006] Obtain the source domain dataset and the target domain dataset respectively. Randomly select a portion of the source domain dataset as the source domain training set and a portion of the target domain dataset as the target domain training set. Label the data in the source domain training set and the target domain training set respectively to obtain the labeled dataset of the source domain and the labeled dataset of the target domain.
[0007] The samples from the labeled dataset of the source domain and the labeled dataset of the target domain are respectively input into the VAE module of the pre-built TransNet model;
[0008] The VAE module first extracts the spatial variation features of the input sample and maps the spatial variation features to the same feature space. Then, it extracts the latent vectors of the source domain spatial variation features and the target domain spatial variation features in the same feature space.
[0009] The VAE module reconstructs the latent vectors of the source domain spatial variation features and the target domain spatial variation features respectively, to obtain source domain restored features and target domain restored features of fixed size;
[0010] Based on the fixed-size source domain reconstruction features and target domain reconstruction features, a loss function for the VAE module is constructed. The VAE module is trained with the goal of minimizing the loss, and a trained VAE module is obtained.
[0011] The fixed-size source domain restored features and target domain restored features are input into the multi-head interactive attention module of the pre-built TransNet model;
[0012] The multi-head interactive attention module extracts fixed-size source domain restored features and target domain restored features containing trend features.
[0013] Then, the time change features in the change trend features of the source domain and the time change features in the change trend features of the target domain are extracted through the multi-head attention interaction layer of the multi-head interaction attention module. The time change features of the source domain and the time change features of the target domain are mapped to time change features of fixed size respectively. The mapped time change features of the source domain and the time change features of the target domain are added together to obtain the attention map.
[0014] The attention map is input into the classifier, and the classifier outputs the predicted label. The predicted label and the standard label are cross-entropy loss is calculated to obtain the prediction loss. The multi-head interaction attention module is trained with the goal of minimizing the prediction loss to obtain the trained multi-head interaction attention module.
[0015] The trained VAE module and the multi-head interaction attention module are concatenated to obtain the trained TransNet model.
[0016] Preferably,
[0017] The VAE module first extracts the spatial variation features of the input samples and maps these features to the same feature space. The latent vectors extracted from the same feature space for both the source domain spatial variation features and the target domain spatial variation features include:
[0018] The VAE module extracts the source domain spatial variation features of the samples in the labeled dataset of the input source domain through the first convolutional layer of the first encoder, fixes the magnitude of the source domain spatial variation features through the first linear layer of the first encoder, and outputs the mean and variance of the source domain spatial variation features. Based on the mean and variance of the source domain spatial variation features, the latent vector of the source domain spatial variation features is obtained through the reparameter method.
[0019] The VAE module extracts the target domain spatial variation features of the labeled dataset samples in the input target domain through the second convolutional layer of the second encoder. The second linear layer of the second encoder fixes the magnitude of the target domain spatial variation features and outputs the mean and variance of the target domain spatial variation features. Based on the mean and variance of the target domain spatial variation features, the latent vector of the target domain spatial variation features is obtained through the reparameter method.
[0020] Preferably,
[0021] The VAE module reconstructs the latent vectors of the source domain spatial variation features and the target domain spatial variation features respectively, obtaining source domain restored features and target domain restored features of fixed size, including:
[0022] The VAE module reconstructs the latent vectors of the source domain spatial variation features and the target domain spatial variation features through the transposed convolutional layer of the decoder, respectively, to obtain source domain restored features and target domain restored features of fixed size.
[0023] Preferably,
[0024] The loss function for constructing the VAE module based on the fixed-size source domain restored features and target domain restored features includes:
[0025] The similarity between the restored features of the source domain and the restored features of the target domain is calculated using MSE loss.
[0026] The difference between the latent vector of the source domain spatial variation characteristics output by the VAE module and the samples of the labeled dataset of the source domain before inputting into the VAE module is calculated by KL divergence.
[0027] The difference between the latent vector of the target domain spatial variation features output by the VAE module and the samples of the labeled dataset of the target domain before inputting into the VAE module is calculated by KL divergence.
[0028] The loss function of the VAE module is obtained by adding the MSE loss, the KL divergence of the source domain, and the KL divergence of the target domain.
[0029] Preferably,
[0030] The step of extracting the fixed-size source domain restored features and the target domain restored features from the input through the multi-head interactive attention module also includes:
[0031] The change trend features of the restored source domain features of the input are extracted through the first multi-head self-attention layer of the multi-head interactive attention module;
[0032] The change trend feature of the restored feature of the source domain is subjected to layer normalization processing through the first normalization layer of the multi-head interactive attention module. The change trend feature of the restored feature of the source domain after layer normalization processing is added to the restored feature of the source domain to obtain the first residual feature.
[0033] The first residual feature is mapped to a fixed-size first residual feature through the first feedforward layer of the multi-head interactive attention module;
[0034] The second multi-head self-attention layer of the multi-head interactive attention module extracts the trend features of the restored features of the input target domain.
[0035] The change trend features of the restored features of the target domain are subjected to layer normalization processing through the second normalization layer of the multi-head interactive attention module. The change trend features of the restored features of the target domain after layer normalization processing are added to the restored features of the target domain to obtain the second residual features.
[0036] The second residual feature is mapped to a fixed-size second residual feature through the second feedforward layer of the multi-head interactive attention module.
[0037] Preferably,
[0038] The step of cross-extracting time change features from the source domain's trend features and the target domain's trend features through the multi-head attention interaction layer of the multi-head interaction attention module, and mapping the time change features of the source domain and the target domain to time change features of fixed size, includes:
[0039] The third multi-head self-attention layer of the multi-head interactive attention module extracts the variation trend features of the second residual feature of a fixed size from the input;
[0040] The third normalization layer of the multi-head interactive attention module performs layer normalization processing on the changing trend features of the input second residual feature;
[0041] The third residual feature is obtained by adding the trend feature of the second residual feature after layer normalization to the second residual feature.
[0042] The third residual feature and the two first residual features are input together into the first multi-head attention interaction layer of the multi-head interaction attention module to extract the first time change feature;
[0043] The first time change feature is processed by the fifth normalization layer of the multi-head interactive attention module, and the first time change feature after the layer normalization is added to the third residual feature to obtain the fifth residual feature.
[0044] The fifth residual feature is mapped to a fixed-size fifth residual feature through the third feedforward layer of the multi-head interactive attention module;
[0045] The fourth multi-head self-attention layer of the multi-head interactive attention module extracts the trend features of the change of the first residual feature of a fixed size from the input.
[0046] The fourth normalization layer of the multi-head interactive attention module performs layer normalization processing on the changing trend features of the input first residual feature;
[0047] The fourth residual feature is obtained by adding the trend feature of the first residual feature after layer normalization to the first residual feature.
[0048] The fourth residual feature and the two second residual features are input together into the second multi-head attention interaction layer of the multi-head interaction attention module to extract the second time change feature;
[0049] The second time-varying feature is processed by the sixth normalization layer of the multi-head interactive attention module, and the second time-varying feature after the layer normalization process is added to the fourth residual feature to obtain the sixth residual feature.
[0050] The sixth residual feature is mapped to a fixed-size sixth residual feature through the fourth feedforward layer of the multi-head interactive attention module.
[0051] According to a second aspect of the present invention, an apparatus for establishing a transfer learning model for an industrial control system is provided, the apparatus comprising:
[0052] Training set acquisition module: used to acquire source domain dataset and target domain dataset respectively, randomly select a part of source domain dataset as source domain training set, randomly select a part of target domain dataset as target domain training set, label the data in source domain training set and target domain training set respectively, and obtain labeled dataset of source domain and labeled dataset of target domain.
[0053] First input module: used to input samples from the labeled dataset of the source domain and the labeled dataset of the target domain into the VAE module of the pre-built TransNet model;
[0054] Latent vector acquisition module: This module is used to first extract the spatial variation features of the input sample, and then map the spatial variation features to the same feature space. In the same feature space, latent vectors of the source domain spatial variation features and the target domain spatial variation features are extracted respectively.
[0055] Feature Reconstruction Module: Used by the VAE module to reconstruct the latent vectors of the source domain spatial variation features and the target domain spatial variation features respectively, to obtain source domain reconstructed features and target domain reconstructed features of fixed size;
[0056] VAE module training module: used to construct the loss function of the VAE module based on the source domain restored features and target domain restored features of the fixed size, and to train the VAE module with the goal of minimizing the loss, so as to obtain the trained VAE module;
[0057] The second input module is used to input the fixed-size source domain restored features and target domain restored features into the multi-head interactive attention module of the pre-built TransNet model.
[0058] Trend feature acquisition module: used to extract the trend features contained in the fixed-size source domain restored features and target domain restored features of the input through the multi-head interactive attention module;
[0059] Attention map acquisition module: used to extract time change features from the change trend features of the source domain and time change features from the change trend features of the target domain through the multi-head attention interaction layer of the multi-head interaction attention module, map the time change features of the source domain and the time change features of the target domain to time change features of fixed size respectively, and add the mapped fixed-size time change features of the source domain and the time change features of the target domain to obtain the attention map.
[0060] Multi-head interaction attention module training module: used to input the attention map into the classifier, output the predicted label through the classifier, calculate the cross-entropy loss between the predicted label and the standard label to obtain the prediction loss, and train the multi-head interaction attention module with the goal of minimizing the prediction loss to obtain the trained multi-head interaction attention module;
[0061] TransNet model acquisition module: used to concatenate the trained VAE module and the multi-head interaction attention module to obtain the trained TransNet model.
[0062] According to a third aspect of the present invention, a storage medium is provided, the storage medium storing a computer program, which, when executed by a host controller, implements the steps of the above-described method.
[0063] The technical solutions provided by the embodiments of the present invention may include the following beneficial effects:
[0064] This application uses a VAE module and an attention interaction module to understand the complex semantic information of industrial control data. The semantic information of industrial control data is mainly reflected in the correlation between features in the horizontal direction and the temporal sequence in the vertical direction. The VAE module of the model maps the data of the source domain and the target domain to the same space by aligning the distribution of the data and carrying the common semantic information. The attention interaction module first encodes the semantic information through self-attention, and then decodes the semantic information encoded by the data of the two domains through attention interaction and learns the similarity between them. When applied to the target domain, the input of the attention interaction module becomes the data mapped by the VAE module and its transpose. This allows the attention interaction module to perform attention interaction from both horizontal and vertical perspectives. Finally, the output is achieved through a fusion strategy, making the trained model transferable.
[0065] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit the invention. Attached Figure Description
[0066] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.
[0067] Figure 1 This is a flowchart illustrating a method for establishing a transfer learning model for an industrial control system according to an exemplary embodiment;
[0068] Figure 2 This is the overall structure of the TransNet model as illustrated in an exemplary embodiment;
[0069] Figure 3This is a schematic diagram of the structure of a VAE module according to an exemplary embodiment;
[0070] Figure 4 This is a schematic diagram of the structure of a multi-head interactive attention module according to an exemplary embodiment;
[0071] Figure 5 This is a schematic diagram of a transfer learning model building device for an industrial control system, according to another exemplary embodiment.
[0072] In the attached diagram: 1-Training set acquisition module, 2-First input module, 3-Latent vector acquisition module, 4-Reconstructed feature acquisition module, 5-VAE module training module, 6-Second input module, 7-Change trend feature acquisition module, 8-Attention map acquisition module, 9-Multi-head interaction attention module training module, 10-TransNet model acquisition module. Detailed Implementation
[0073] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Rather, they are merely examples of apparatuses and methods consistent with some aspects of the invention as detailed in the appended claims.
[0074] Example 1
[0075] Figure 1 This is a flowchart illustrating a method for establishing a transfer learning model for an industrial control system according to an exemplary embodiment. The method includes:
[0076] S1. Obtain the source domain dataset and the target domain dataset respectively. Randomly select a portion of the source domain dataset as the source domain training set and a portion of the target domain dataset as the target domain training set. Label the data in the source domain training set and the target domain training set respectively to obtain the labeled dataset of the source domain and the labeled dataset of the target domain.
[0077] S2, input the samples from the labeled dataset of the source domain and the labeled dataset of the target domain into the VAE module of the pre-built TransNet model respectively;
[0078] S3, the VAE module first extracts the spatial variation features of the input sample and maps the spatial variation features to the same feature space, and extracts the latent vectors of the source domain spatial variation features and the target domain spatial variation features in the same feature space respectively;
[0079] S4, the VAE module reconstructs the latent vectors of the source domain spatial variation features and the target domain spatial variation features respectively to obtain source domain restored features and target domain restored features of fixed size;
[0080] S5. Based on the source domain reconstruction features and target domain reconstruction features of the fixed size, construct the loss function of the VAE module, and train the VAE module with the goal of minimizing the loss to obtain the trained VAE module.
[0081] S6, input the fixed-size source domain restored features and target domain restored features into the multi-head interactive attention module of the pre-built TransNet model;
[0082] S7 extracts fixed-size source domain reconstruction features and target domain reconstruction features from the input through a multi-head interactive attention module;
[0083] S8. The time change features in the change trend features of the source domain and the time change features in the change trend features of the target domain are extracted through the multi-head attention interaction layer of the multi-head interaction attention module. The time change features of the source domain and the time change features of the target domain are mapped to time change features of fixed size respectively. The time change features of the source domain and the time change features of the target domain after mapping are added to obtain the attention map.
[0084] S9, input the attention map into the classifier, output the predicted label through the classifier, calculate the cross-entropy loss between the predicted label and the standard label to obtain the prediction loss, train the multi-head interaction attention module with the goal of minimizing the prediction loss, and obtain the trained multi-head interaction attention module.
[0085] S10, the trained VAE module and the multi-head interaction attention module are concatenated to obtain the trained TransNet model;
[0086] Understandably, as shown in the attached document Figure 2 As shown, TransNet can be structurally divided into a VAE module and a multi-head interactive attention module. The VAE module's encoder designs convolutional layers based on the different sizes of the source and target domain data. The decoder takes the latent vectors sampled from the common latent semantic information distribution of the source and target domains as input, and outputs fixed-size sampled data through transposed convolutional layers. The forward propagation process of the attention interaction module can be divided into an encoding stage and a decoding stage. In the encoding stage, the inputs from the source and target domains are used to encode the semantic information characteristics within each domain using self-attention. In the decoding stage, the output from the encoding stage is used to exchange different semantic information between domains using interactive attention. These represent the source and target domain datasets, respectively. This indicates a labeled dataset from the source domain, and This represents an unlabeled dataset from the target domain. For a dataset in the target domain consisting of a small amount of labeled data, use This indicates that our goal is to use X. s X t To train the TransNet model to learn the common latent semantic information representation of the two domains, so that in X t` This application achieves good results in downstream tasks. It uses a VAE module and an attention interaction module to understand the complex semantic information of industrial control data. The semantic information of industrial control data is mainly reflected in the correlation between features in the horizontal direction and the temporal sequence in the vertical direction. The VAE module of the model maps the data of the source domain and the target domain to the same space by aligning the distribution of the data and carrying the common semantic information. The attention interaction module first encodes the semantic information through self-attention, and then decodes the semantic information encoded by the data of the two domains through attention interaction and learns the similarity between them. When applied to the target domain, the input of the attention interaction module becomes the data mapped by the VAE module and its transpose. This allows the attention interaction module to perform attention interaction from both horizontal and vertical perspectives. Finally, the results are output through a fusion strategy, making the trained model transferable.
[0087] Preferably,
[0088] The VAE module first extracts the spatial variation features of the input samples and maps these features to the same feature space. The latent vectors extracted from the same feature space for both the source domain spatial variation features and the target domain spatial variation features include:
[0089] The VAE module extracts the source domain spatial variation features of the samples in the labeled dataset of the input source domain through the first convolutional layer of the first encoder, fixes the magnitude of the source domain spatial variation features through the first linear layer of the first encoder, and outputs the mean and variance of the source domain spatial variation features. Based on the mean and variance of the source domain spatial variation features, the latent vector of the source domain spatial variation features is obtained through the reparameter method.
[0090] The VAE module extracts the target domain spatial variation features of the labeled dataset samples in the input target domain through the second convolutional layer of the second encoder, fixes the magnitude of the target domain spatial variation features through the second linear layer of the second encoder, and outputs the mean and variance of the target domain spatial variation features. Based on the mean and variance of the target domain spatial variation features, the latent vector of the target domain spatial variation features is obtained through the reparameter method.
[0091] Understandably, as shown in the attached document Figure 3The VAE module architecture details are shown below. This module consists of separate encoders for the source and target domains, and a shared decoder. The encoders are constructed using multiple convolutional layers, normalization layers, and ReLU activation functions. The decoder is constructed using multiple transposed convolutional layers. The main goal of the VAE module is to map the original inputs of the source and target domains to the same feature space, sample latent vectors from the feature space for both domains, and finally decode these latent vectors into the input of the multi-head interactive attention module. The acquisition of latent vectors involves: extracting spatial variation features of the source and target domains through convolutional layers in the respective encoders of the source and target domains; fixing the magnitude of these spatial variation features through their respective linear layers, i.e., outputting the mean and variance of the spatial variation features of the source and target domains; and finally obtaining the latent vectors of the spatial variation features through a reparameterization method. The reparameterization technique changes the sampling method, sampling a random variable ε from a standard normal distribution to ensure randomness while making the sampling process differentiable. The expression for the latent vectors is shown below.
[0092] Z = μ + σ*ε
[0093] In the formula, z represents the latent vector, μ represents the mean of the spatial variation characteristics of the source or target domain, and σ represents the variance of the spatial variation characteristics of the source or target domain.
[0094] Preferably,
[0095] The VAE module reconstructs the latent vectors of the source domain spatial variation features and the target domain spatial variation features respectively, obtaining source domain restored features and target domain restored features of fixed size, including:
[0096] The VAE module reconstructs the latent vectors of the source domain spatial variation features and the target domain spatial variation features through the transposed convolutional layer of the decoder, respectively, to obtain source domain restored features and target domain restored features of fixed size.
[0097] Understandably, as shown in the attached document Figure 3 As shown, the obtained latent vectors are input into a shared decoder. The latent vectors of the source domain spatial variation features and the target domain spatial variation features are reconstructed through the transposed convolutional layer of the decoder, respectively, to obtain source domain restored features and target domain restored features of fixed size.
[0098] Preferably,
[0099] The loss function for constructing the VAE module based on the fixed-size source domain restored features and target domain restored features includes:
[0100] The similarity between the restored features of the source domain and the restored features of the target domain is calculated using MSE loss.
[0101] The difference between the latent vector of the source domain spatial variation characteristics output by the VAE module and the samples of the labeled dataset of the source domain before inputting into the VAE module is calculated by KL divergence.
[0102] The difference between the latent vector of the target domain spatial variation features output by the VAE module and the samples of the labeled dataset of the target domain before inputting into the VAE module is calculated by KL divergence.
[0103] The loss function of the VAE module is obtained by adding the MSE loss, the KL divergence of the source domain, and the KL divergence of the target domain.
[0104] Understandably, the optimization direction of this module first needs to ensure that the encoder can extract features common to both the source and target domains. Therefore, MSE loss is used to measure the similarity of the output data features of the source and target domains as part of the optimization of this module.
[0105]
[0106] In the formula, Let f(·) represent the MSE loss, and f(·) represent the shared decoder. This represents the latent vector sampled from the source domain. This represents the latent vector sampled from the target domain;
[0107] This solution considers that the module needs to ensure that the data distribution before encoding and after reparameter sampling is as similar as possible in order to preserve the data distribution variations within the neighborhood. Based on this, the application uses KL divergence to calculate the distribution difference between the latent vector after sampling and the data before sampling.
[0108]
[0109] Where p(·) and q(·) represent the probability distributions before and after the transformation. In actual optimization, calculating the difference between the two probability distributions is complex. Directly sampling the samples in the probability distribution will lose parameter information when calculating the gradient, resulting in the inability to update the parameters, as shown in the implicit vector calculation formula above. Here, the reparameter technique is used to solve this problem. The reparameter technique changes the sampling method. By sampling a random variable ε from the standard normal distribution, randomness is guaranteed, while the sampling process is made differentiable in the following way. After rearranging the above two formulas, the second part of the VAE module optimization can be obtained as follows:
[0110]
[0111] In the formula, Denotes the KL divergence of the source domain. The mean squared value represents the spatial variation characteristics of the source domain. The square of the variance representing the spatial variation characteristics of the source domain;
[0112] Similarly, the loss before and after the target domain transformation is:
[0113]
[0114] In the formula, Represents the KL divergence of the target domain. This represents the square of the mean of the spatial variation characteristics of the source domain. The square of the variance representing the spatial variation characteristics of the source domain;
[0115] By reorganizing the above formulas, we can derive the loss function of the VAE module as follows:
[0116]
[0117] In the formula, The loss of the VAE module is represented by α, β, and γ, which are coefficients used to adjust the interaction between the loss terms.
[0118] Preferably,
[0119] The step of extracting the fixed-size source domain restored features and the target domain restored features from the input through the multi-head interactive attention module also includes:
[0120] The change trend features of the restored source domain features of the input are extracted through the first multi-head self-attention layer of the multi-head interactive attention module;
[0121] The change trend feature of the restored feature of the source domain is subjected to layer normalization processing through the first normalization layer of the multi-head interactive attention module. The change trend feature of the restored feature of the source domain after layer normalization processing is added to the restored feature of the source domain to obtain the first residual feature.
[0122] The first residual feature is mapped to a fixed-size first residual feature through the first feedforward layer of the multi-head interactive attention module;
[0123] The second multi-head self-attention layer of the multi-head interactive attention module extracts the trend features of the restored features of the input target domain.
[0124] The change trend features of the restored features of the target domain are subjected to layer normalization processing through the second normalization layer of the multi-head interactive attention module. The change trend features of the restored features of the target domain after layer normalization processing are added to the restored features of the target domain to obtain the second residual features.
[0125] The second residual feature is mapped to a fixed-size second residual feature through the second feedforward layer of the multi-head interactive attention module;
[0126] Understandably, as shown in the attached document Figure 4 As shown, the forward propagation process of the multi-head interactive attention module first encodes the association embedding between the features of the source domain and the target domain and their private semantic information through the multi-head self-attention layer. Then, the attention interaction module is used to find the common parts between the association embeddings, mainly through the calculation of the multi-head attention interaction score. Then, the similarity between the data changes of the source domain and the target domain is learned to complete the transfer of inter-domain knowledge. Finally, the fusion strategy is applied to superimpose the two results as the output of the module.
[0127] Specifically, the multi-head self-attention layer of the multi-head interactive attention module extracts the trend features of the restored features of the input source or target domain. It is worth noting that after the restored features enter the multi-head self-attention layer, they are divided into multiple blocks. Taking the source domain as an example, the extraction of the trend features of each block is as follows:
[0128]
[0129] In the formula, This represents the attention score of the i-th attention head in the source domain (i.e., the restored feature of the source domain after a single cut). These represent the query, key-value transformation, and d of the attention head, respectively. K x represents the dimension of the input data. s This represents the source domain input data (i.e., the restored features of the source domain);
[0130] The sum of the trend characteristics of all the cut blocks yields the trend characteristics of the entire restored feature, as shown below:
[0131]
[0132] W represents the attention score of multi-head attention in the source domain (i.e., the sum of the trend features of all the cut blocks). H (·) represents the transformation matrix for summing multiple attentions, and Concat(·) represents summing the attentions. Represents the attention score of the first attention head in the source domain. Represents the attention score of the i-th attention head in the source domain.
[0133] The acquired trend features are each processed by a normalization layer to remove abnormal interference values. The trend features of the restored features after the normalization process are then processed by residual processing, i.e., the original input is added to obtain residual features. Finally, the residual features of the source domain and the target domain are mapped to a fixed size through their respective feedforward layers.
[0134] Preferably,
[0135] The step of cross-extracting time change features from the source domain's trend features and the target domain's trend features through the multi-head attention interaction layer of the multi-head interaction attention module, and mapping the time change features of the source domain and the target domain to time change features of fixed size, includes:
[0136] The third multi-head self-attention layer of the multi-head interactive attention module extracts the variation trend features of the second residual feature of a fixed size from the input;
[0137] The third normalization layer of the multi-head interactive attention module performs layer normalization processing on the changing trend features of the input second residual feature;
[0138] The third residual feature is obtained by adding the trend feature of the second residual feature after layer normalization to the second residual feature.
[0139] The third residual feature and the two first residual features are input together into the first multi-head attention interaction layer of the multi-head interaction attention module to extract the first time change feature;
[0140] The first time change feature is processed by the fifth normalization layer of the multi-head interactive attention module, and the first time change feature after the layer normalization is added to the third residual feature to obtain the fifth residual feature.
[0141] The fifth residual feature is mapped to a fixed-size fifth residual feature through the third feedforward layer of the multi-head interactive attention module;
[0142] The fourth multi-head self-attention layer of the multi-head interactive attention module extracts the trend features of the change of the first residual feature of a fixed size from the input.
[0143] The fourth normalization layer of the multi-head interactive attention module performs layer normalization processing on the changing trend features of the input first residual feature;
[0144] The fourth residual feature is obtained by adding the trend feature of the first residual feature after layer normalization to the first residual feature.
[0145] The fourth residual feature and the two second residual features are input together into the second multi-head attention interaction layer of the multi-head interaction attention module to extract the second time change feature;
[0146] The second time-varying feature is processed by the sixth normalization layer of the multi-head interactive attention module, and the second time-varying feature after the layer normalization process is added to the fourth residual feature to obtain the sixth residual feature.
[0147] The sixth residual feature is mapped to a fixed-size sixth residual feature through the fourth feedforward layer of the multi-head interactive attention module;
[0148] Understandably, as shown in the attached document Figure 4 As shown, multi-head interactive attention is implemented based on the aforementioned self-attention. Its calculation method is roughly the same as that of self-attention, but the input data is different. The goal of this application is to extract and transfer the similar change trends contained in the changes of industrial control strategies in different industrial control systems. Assuming that the encoder output retains the global feature changes of their respective domains through multi-head self-attention, the transfer of these changes is completed through multi-head attention interaction. Taking the source domain decoder as an example, the query of multi-head attention interaction is defined as the output of the target domain encoder after the decoder self-attention layer, and the key and value are both the output of the source domain encoder. In the source domain part, the change trend features of the fixed-size second residual feature (i.e., the residual feature of the target domain mentioned above) of the input are extracted through the multi-head self-attention layer; then the change trend features of the input second residual feature are subjected to layer normalization processing through the normalization layer; then the change trend features of the second residual feature after layer normalization processing are added to the second residual feature (i.e., the residual feature of the target domain obtained above) to obtain the third residual feature, as shown in the appendix. Figure 4 As shown, in the source domain portion (i.e., the appendix) Figure 4 The upper half of the data is used to input the third residual feature and two identical first residual features (i.e., the residual features of the source domain obtained above) into the multi-head attention interaction layer to extract time-varying features. Similarly, the input data entering the multi-head attention interaction layer will also be divided into multiple blocks, and the time-varying features of each block are calculated as follows:
[0149]
[0150] This represents the attention interaction score of the j-th attention head (i.e., the temporal variation feature of the residual feature after a single cut). This represents the query, key, and value transformation matrix of the attention head. The attention score represents the multi-head attention in the target domain. d represents the attention score of multi-head attention in the source domain. K Indicates the dimension of the input data;
[0151] The final calculation of the time-varying characteristics is as follows:
[0152]
[0153] W represents the attention score (i.e., the final temporal variation feature) of the multi-head attention interaction layer. H(·) represents the transformation matrix for summing multiple attention heads, and Concat(·) represents summing the attention heads. The attentional interaction score of the first attentional head. This represents the attention interaction score of the j-th attention head;
[0154] The acquired time-varying features are then further processed by a normalization layer. The normalized time-varying features are then subjected to residual processing (i.e., a third residual feature is added). Finally, a feedforward layer maps the residual-processed time-varying features to a fixed-size feature, yielding the output of the source domain. Similarly, as shown in the appendix... Figure 4 As shown in the lower half, the principle of obtaining the target domain output is as described above. The attention map is obtained by adding the outputs of the source and target domains. This attention map is then input into the classifier to obtain the predicted label. Using multi-class cross-entropy loss as an optimization method, the loss of the multi-head interaction attention module is defined as:
[0155]
[0156] C(·) denotes the classifier, and exp(·) denotes the exponential function with base e.
[0157] Finally, the trained VAE module and the multi-head interaction attention module are concatenated to obtain the attached... Figure 2 The TransNet model shown.
[0158] Understandably, to verify this solution, this embodiment provides a comparative experiment between this solution and existing technologies. In selecting the dataset, five datasets from different industrial control systems were used, and their specific information is shown in Table 1 below:
[0159] Table 1 provides detailed information about the datasets used.
[0160]
[0161] As shown in Table 1, Dataset A is collected from a self-built miniature simulation platform for oil depot operations. This platform simulates the implementation of oil storage and transportation processes and consists of three parts: a physical sand table module, a data acquisition and control module, and a human-machine interface module. The physical sand table module is constructed from physical entities such as water tanks, level sensors, temperature sensors, valves, and water pumps. The data acquisition and control module uses a PLC to control the various components of the physical sand table. The human-machine interface module uses WinCC to build a simulation control system and implement a graphical interface. The abnormal samples were collected after attacks on the platform, and a total of 13 types of attacks targeting industrial control systems were designed. Dataset B is a water distribution network attack dataset from the Cybersecurity Research Center of Singapore University of Technology. Its features include nine PLC sensors and actuators, such as tank water level, outlet pressure, flow rate, and status. Each PLC also has a corresponding control sensor, which provides the information required to operate the actuator, and includes 13 types of anomalies. Dataset C is the 2014 Industrial Control System Human Intrusion Detection and Assessment Dataset from the Mississippi State University Critical Infrastructure Protection Center. It helps researchers evaluate the performance of Monitoring and Data Acquisition Intrusion Detection Systems (SCADA IDS) using real-world SCADA attack patterns. It simulates and performs attacks on a natural gas pipeline control system, featuring 26 characteristics and anomalies categorized into 7 types. Datasets D and E are derived from real-world operational data from oil depots in Beijing and Chongqing, respectively.
[0162] Of the five datasets mentioned above, A, B, and C all come from a physical miniature simulation system, while D and E come from runtime data of two different real oil depot systems. Anomalies in the former are generated by attacking the simulation system and the categories of the anomaly samples are labeled. In the latter, due to the need for safe production at the oil depot, anomalies cannot be generated by attacking the real oil depot system; therefore, a generative adversarial network is used to construct anomaly samples from the collected data.
[0163] The transfer tasks on five datasets were divided into ten groups: A→B, A→C, A→D, A→E, B→A, B→C, B→D, B→E, C→D, and C→E. The left side of the arrow represents the source domain dataset used for learning, and the right side represents the target domain dataset. The experiment used accuracy as the indicator, and the purpose was to verify whether the model could extract the common semantic information of different industrial control systems through the accuracy of the transfer tasks, and to judge the anomaly category in the test set based on this. Four adversarial transfer methods were used for comparison: DANN, DSN, ADDA, and IADA.
[0164] The experimental results are shown in Table 2:
[0165] Table 2. Migration Task Accuracy Rate (%)
[0166]
[0167] Table 2 presents the experimental results on ten transfer tasks. The data shows that TransNet's average accuracy is higher than the other four methods, improving accuracy by approximately 24.69%, 4.97%, 57.48%, and 0.9% respectively compared to the four comparison algorithms. The IADA method is similar to TransNet, achieving an average accuracy of 62.43%, followed by DSN with an average accuracy of 58.36%. DANN then achieves an accuracy of 38.64%, while ADDA performs poorly with an average accuracy of 5.85%.
[0168] Example 2:
[0169] Figure 5 This is a schematic diagram of a transfer learning model building device for an industrial control system according to another exemplary embodiment, the device comprising:
[0170] Training set acquisition module 1: used to acquire source domain dataset and target domain dataset respectively, randomly select a part of source domain dataset as source domain training set, randomly select a part of target domain dataset as target domain training set, label the data in source domain training set and target domain training set respectively, and obtain labeled dataset of source domain and labeled dataset of target domain.
[0171] First input module 2: used to input the samples from the labeled dataset of the source domain and the labeled dataset of the target domain into the VAE module of the pre-built TransNet model respectively;
[0172] Latent vector acquisition module 3: is used by the VAE module to first extract the spatial variation features of the input sample, and map the spatial variation features to the same feature space, and extract the latent vectors of the source domain spatial variation features and the target domain spatial variation features in the same feature space respectively;
[0173] Feature Reconstruction Module 4: Used by the VAE module to reconstruct the latent vectors of the source domain spatial variation features and the target domain spatial variation features respectively, to obtain source domain reconstructed features and target domain reconstructed features of fixed size;
[0174] VAE module training module 5: used to construct the loss function of the VAE module based on the source domain restored features and target domain restored features of the fixed size, and to train the VAE module with the goal of minimizing the loss, so as to obtain the trained VAE module;
[0175] Second input module 6: used to input the fixed-size source domain restored features and target domain restored features into the multi-head interactive attention module of the pre-built TransNet model;
[0176] Trend feature acquisition module 7: used to extract the trend features contained in the fixed-size source domain restored features and target domain restored features of the input through the multi-head interactive attention module;
[0177] Attention map acquisition module 8: is used to extract time change features from the change trend features of the source domain and time change features from the change trend features of the target domain through the multi-head attention interaction layer of the multi-head interaction attention module, map the time change features of the source domain and the time change features of the target domain to time change features of fixed size respectively, and add the mapped fixed-size time change features of the source domain and the time change features of the target domain to obtain the attention map.
[0178] Multi-head interaction attention module training module 9: is used to input the attention map into the classifier, output the predicted label through the classifier, calculate the cross-entropy loss between the predicted label and the standard label to obtain the prediction loss, and train the multi-head interaction attention module with the goal of minimizing the prediction loss to obtain the trained multi-head interaction attention module.
[0179] TransNet model acquisition module 10: used to concatenate the trained VAE module and the multi-head interaction attention module to obtain a trained TransNet model;
[0180] Understandably, the training set acquisition module 1 is used to acquire the source domain dataset and the target domain dataset respectively, randomly selecting a portion of the source domain dataset as the source domain training set and a portion of the target domain dataset as the target domain training set. The data in both the source and target domain training sets are then labeled to obtain the labeled datasets for the source and target domains. The first input module 2 is used to input the samples from the labeled datasets for the source and target domains into the pre-built VAE module of the TransNet model. The latent vector acquisition module 3 is used by the VAE module to first extract the spatial variation features of the input samples, and... The spatial variation features are mapped to the same feature space, and latent vectors of the source domain spatial variation features and the target domain spatial variation features are extracted from the same feature space. The feature reconstruction module 4 is used by the VAE module to reconstruct the latent vectors of the source domain spatial variation features and the target domain spatial variation features, respectively, to obtain fixed-size source domain reconstructed features and target domain reconstructed features. The VAE module training module 5 is used to construct the loss function of the VAE module based on the fixed-size source domain reconstructed features and the target domain reconstructed features, and trains the VAE module with the goal of minimizing the loss, obtaining a trained VAE module. The second input module 6 then... The fixed-size source domain restored features and target domain restored features are input into the pre-built TransNet model's multi-head interactive attention module; the trend feature acquisition module 7 is used to extract the trend features contained in the input fixed-size source domain restored features and target domain restored features through the multi-head interactive attention module; the attention map acquisition module 8 is used to extract the time change features in the trend features of the source domain and the trend features of the target domain through the multi-head attention interaction layer of the multi-head interactive attention module, and map the time change features of the source domain and the time change features of the target domain to fixed-size time change features respectively. The attention map is obtained by adding the time-varying features of the source domain and the target domain after mapping to a fixed size. The attention map is then input into a classifier through a multi-head interaction attention module training module 9. The classifier outputs a predicted label, and the predicted label is compared with the standard label to calculate the cross-entropy loss. The multi-head interaction attention module is trained with the goal of minimizing the predicted loss. Finally, the TransNet model acquisition module 10 is used to concatenate the trained VAE module and the multi-head interaction attention module to obtain a trained TransNet model.This application utilizes a VAE (Visual Augmentation Module) module and an attention interaction module to understand the complex semantic information of industrial control data. The semantic information of industrial control data is mainly reflected in the horizontal correlation between features and the vertical temporal sequence. The model's VAE module maps the distribution of source and target domain data to the same space, carrying shared semantic information. The attention interaction module first encodes this semantic information through self-attention, then decodes the encoded semantic information from both domains through attention interaction and learns their similarities. When applied to the target domain, the input to the attention interaction module becomes the data mapped by the VAE module and its transpose. This allows the attention interaction module to perform attention interactions from both horizontal and vertical perspectives. Finally, a fusion strategy is used to output the results, making the trained model transferable.
[0181] Example 3:
[0182] This embodiment provides a storage medium storing a computer program, which, when executed by a host controller, implements the various steps in the above method.
[0183] It is understood that the storage medium mentioned above can be a read-only memory, a hard disk, or an optical disk, etc.
[0184] It is understood that the same or similar parts in the above embodiments can be referred to each other, and the contents not described in detail in some embodiments can be referred to the same or similar contents in other embodiments.
[0185] It should be noted that in the description of this invention, the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance. Furthermore, in the description of this invention, unless otherwise stated, "a plurality of" means at least two.
[0186] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process, and the scope of the preferred embodiments of the invention includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as will be understood by those skilled in the art to which embodiments of the invention pertain.
[0187] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.
[0188] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.
[0189] Furthermore, the functional units in the various embodiments of the present invention can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.
[0190] The storage media mentioned above can be read-only memory, disk, or optical disk, etc.
[0191] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.
[0192] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.
Claims
1. A method for establishing a transfer learning model for an industrial control system, characterized in that, The method includes: Obtain the source domain dataset and the target domain dataset respectively. Randomly select a portion of the source domain dataset as the source domain training set and a portion of the target domain dataset as the target domain training set. Label the data in the source domain training set and the target domain training set respectively to obtain the labeled dataset of the source domain and the labeled dataset of the target domain. The samples from the labeled dataset of the source domain and the labeled dataset of the target domain are respectively input into the VAE module of the pre-built TransNet model; The VAE module first extracts the spatial variation features of the input sample and maps the spatial variation features to the same feature space. Then, it extracts the latent vectors of the source domain spatial variation features and the target domain spatial variation features in the same feature space. The VAE module reconstructs the latent vectors of the source domain spatial variation features and the target domain spatial variation features respectively, to obtain source domain restored features and target domain restored features of fixed size; Based on the fixed-size source domain reconstruction features and target domain reconstruction features, a loss function for the VAE module is constructed. The VAE module is trained with the goal of minimizing the loss, and a trained VAE module is obtained. The fixed-size source domain restored features and target domain restored features are input into the multi-head interactive attention module of the pre-built TransNet model; The multi-head interactive attention module extracts fixed-size source domain restored features and target domain restored features containing trend features, including: The change trend features of the restored source domain features of the input are extracted through the first multi-head self-attention layer of the multi-head interactive attention module; The change trend feature of the restored feature of the source domain is subjected to layer normalization processing through the first normalization layer of the multi-head interactive attention module. The change trend feature of the restored feature of the source domain after layer normalization processing is added to the restored feature of the source domain to obtain the first residual feature. The first residual feature is mapped to a fixed-size first residual feature through the first feedforward layer of the multi-head interactive attention module; The second multi-head self-attention layer of the multi-head interactive attention module extracts the trend features of the restored features of the input target domain. The change trend features of the restored features of the target domain are subjected to layer normalization processing through the second normalization layer of the multi-head interactive attention module. The change trend features of the restored features of the target domain after layer normalization processing are added to the restored features of the target domain to obtain the second residual features. The second residual feature is mapped to a fixed-size second residual feature through the second feedforward layer of the multi-head interactive attention module; Then, the time change features in the change trend features of the source domain and the time change features in the change trend features of the target domain are extracted through the multi-head attention interaction layer of the multi-head interaction attention module. The time change features of the source domain and the time change features of the target domain are mapped to time change features of fixed size respectively. The mapped time change features of the source domain and the time change features of the target domain are added together to obtain the attention map. The attention map is input into the classifier, and the classifier outputs the predicted label. The predicted label and the standard label are cross-entropy loss is calculated to obtain the prediction loss. The multi-head interaction attention module is trained with the goal of minimizing the prediction loss to obtain the trained multi-head interaction attention module. The trained VAE module and the multi-head interaction attention module are concatenated to obtain the trained TransNet model.
2. The method according to claim 1, characterized in that, The VAE module first extracts the spatial variation features of the input samples and maps these features to the same feature space. The latent vectors extracted from the same feature space for both the source domain spatial variation features and the target domain spatial variation features include: The VAE module extracts the source domain spatial variation features of the samples in the labeled dataset of the input source domain through the first convolutional layer of the first encoder, fixes the magnitude of the source domain spatial variation features through the first linear layer of the first encoder, and outputs the mean and variance of the source domain spatial variation features. Based on the mean and variance of the source domain spatial variation features, the latent vector of the source domain spatial variation features is obtained through the reparameter method. The VAE module extracts the target domain spatial variation features of the labeled dataset samples in the input target domain through the second convolutional layer of the second encoder. The second linear layer of the second encoder fixes the magnitude of the target domain spatial variation features and outputs the mean and variance of the target domain spatial variation features. Based on the mean and variance of the target domain spatial variation features, the latent vector of the target domain spatial variation features is obtained through the reparameter method.
3. The method according to claim 2, characterized in that, The VAE module reconstructs the latent vectors of the source domain spatial variation features and the target domain spatial variation features respectively, obtaining source domain restored features and target domain restored features of fixed size, including: The VAE module reconstructs the latent vectors of the source domain spatial variation features and the target domain spatial variation features through the transposed convolutional layer of the decoder, respectively, to obtain source domain restored features and target domain restored features of fixed size.
4. The method according to claim 3, characterized in that, The loss function for constructing the VAE module based on the fixed-size source domain restored features and target domain restored features includes: The similarity between the restored features of the source domain and the restored features of the target domain is calculated using MSE loss. The difference between the latent vector of the source domain spatial variation characteristics output by the VAE module and the samples of the labeled dataset of the source domain before inputting into the VAE module is calculated by KL divergence. The difference between the latent vector of the target domain spatial variation features output by the VAE module and the samples of the labeled dataset of the target domain before inputting into the VAE module is calculated by KL divergence. The loss function of the VAE module is obtained by adding the MSE loss, the KL divergence of the source domain, and the KL divergence of the target domain.
5. The method according to claim 1, characterized in that, The step of cross-extracting time change features from the source domain's trend features and the target domain's trend features through the multi-head attention interaction layer of the multi-head interaction attention module, and mapping the time change features of the source domain and the target domain to time change features of fixed size, includes: The third multi-head self-attention layer of the multi-head interactive attention module extracts the variation trend features of the second residual feature of a fixed size from the input; The third normalization layer of the multi-head interactive attention module performs layer normalization processing on the changing trend features of the input second residual feature; The third residual feature is obtained by adding the trend feature of the second residual feature after layer normalization to the second residual feature. The third residual feature and the two first residual features are input together into the first multi-head attention interaction layer of the multi-head interaction attention module to extract the first time change feature; The first time change feature is processed by the fifth normalization layer of the multi-head interactive attention module, and the first time change feature after the layer normalization is added to the third residual feature to obtain the fifth residual feature. The fifth residual feature is mapped to a fixed-size fifth residual feature through the third feedforward layer of the multi-head interactive attention module; The fourth multi-head self-attention layer of the multi-head interactive attention module extracts the trend features of the change of the first residual feature of a fixed size from the input. The fourth normalization layer of the multi-head interactive attention module performs layer normalization processing on the changing trend features of the input first residual feature; The fourth residual feature is obtained by adding the trend feature of the first residual feature after layer normalization to the first residual feature. The fourth residual feature and the two second residual features are input together into the second multi-head attention interaction layer of the multi-head interaction attention module to extract the second time change feature; The second time-varying feature is processed by the sixth normalization layer of the multi-head interactive attention module, and the second time-varying feature after the layer normalization process is added to the fourth residual feature to obtain the sixth residual feature. The sixth residual feature is mapped to a fixed-size sixth residual feature through the fourth feedforward layer of the multi-head interactive attention module.
6. A device for establishing a transfer learning model for an industrial control system, characterized in that, The device includes: Training set acquisition module: used to acquire source domain dataset and target domain dataset respectively, randomly select a part of source domain dataset as source domain training set, randomly select a part of target domain dataset as target domain training set, label the data in source domain training set and target domain training set respectively, and obtain labeled dataset of source domain and labeled dataset of target domain. First input module: used to input samples from the labeled dataset of the source domain and the labeled dataset of the target domain into the VAE module of the pre-built TransNet model; Latent vector acquisition module: This module is used to first extract the spatial variation features of the input sample, and then map the spatial variation features to the same feature space. In the same feature space, latent vectors of the source domain spatial variation features and the target domain spatial variation features are extracted respectively. Feature Reconstruction Module: Used by the VAE module to reconstruct the latent vectors of the source domain spatial variation features and the target domain spatial variation features respectively, to obtain source domain reconstructed features and target domain reconstructed features of fixed size; VAE module training module: used to construct the loss function of the VAE module based on the source domain restored features and target domain restored features of the fixed size, and to train the VAE module with the goal of minimizing the loss, so as to obtain the trained VAE module; The second input module is used to input the fixed-size source domain restored features and target domain restored features into the multi-head interactive attention module of the pre-built TransNet model. Trend feature acquisition module: Used to extract the trend features implied in the fixed-size source domain restored features and target domain restored features of the input through a multi-head interactive attention module, including: The change trend features of the restored source domain features of the input are extracted through the first multi-head self-attention layer of the multi-head interactive attention module; The change trend feature of the restored feature of the source domain is subjected to layer normalization processing through the first normalization layer of the multi-head interactive attention module. The change trend feature of the restored feature of the source domain after layer normalization processing is added to the restored feature of the source domain to obtain the first residual feature. The first residual feature is mapped to a fixed-size first residual feature through the first feedforward layer of the multi-head interactive attention module; The second multi-head self-attention layer of the multi-head interactive attention module extracts the trend features of the restored features of the input target domain. The change trend features of the restored features of the target domain are subjected to layer normalization processing through the second normalization layer of the multi-head interactive attention module. The change trend features of the restored features of the target domain after layer normalization processing are added to the restored features of the target domain to obtain the second residual features. The second residual feature is mapped to a fixed-size second residual feature through the second feedforward layer of the multi-head interactive attention module; Attention map acquisition module: used to extract time change features from the change trend features of the source domain and time change features from the change trend features of the target domain through the multi-head attention interaction layer of the multi-head interaction attention module, map the time change features of the source domain and the time change features of the target domain to time change features of fixed size respectively, and add the mapped fixed-size time change features of the source domain and the time change features of the target domain to obtain the attention map. Multi-head interaction attention module training module: used to input the attention map into the classifier, output the predicted label through the classifier, calculate the cross-entropy loss between the predicted label and the standard label to obtain the prediction loss, and train the multi-head interaction attention module with the goal of minimizing the prediction loss to obtain the trained multi-head interaction attention module; TransNet model acquisition module: used to concatenate the trained VAE module and the multi-head interaction attention module to obtain the trained TransNet model.
7. A storage medium, characterized in that, The storage medium stores a computer program, which, when executed by the main controller, implements each step of the method for establishing a transfer learning model for an industrial control system as described in any one of claims 1-5.
Citation Information
Patent Citations
Method and terminal for detecting time series data exception by combining attention mechanism and LSTM (Long Short Term Memory)
CN115983087A
System and Method for Automated Transfer Learning with Domain Disentanglement
US20230162023A1
