Federal learning sensor anomaly detection method and system based on Bert model

Abstract

The invention discloses a federated learning sensor anomaly detection method and system based on a Bert model, which can carry out distributed anomaly detection and is suitable for carrying out anomaly detection through effective time sequence data generated by a distributed sensor under the condition of low communication cost. The method specifically comprises the following steps: arranging corresponding temperature sensors on key nodes, and connecting the temperature sensors to an anomaly detection terminal; and the central server initializes parameters and issues the parameters to each anomaly detection terminal. And the anomaly detection terminals train the local model by using local data, then the gradient adjustment parameters of the local model are sent back to the central server, the central server updates the global model parameters and sends the global model parameters back to the anomaly detection terminals, and the local model is updated at the anomaly detection terminals so as to carry out the next round of training. And after the overall training is completed, each detection terminal performs anomaly detection on local data, and a detection result is transmitted back to the central server.

Description

Technical Field

[0001] The present invention relates to the field of federated learning technology, and in particular to a federated learning sensor anomaly detection method and system based on a Bert model. Background Art

[0002] With the development and maturity of industrial systems, the traditional solution of relying on pure manpower to manage system operation has been unable to adapt to the needs of productivity development. In order to further reduce production costs and improve production efficiency, more and more sensor systems and management control systems are connected to industrial production systems. However, these sensor systems and management control systems will generate a large amount of time series data. These data reflect the operating status of the industrial production system at all times and are an objective manifestation of the system status. Therefore, anomaly detection of these data can timely discover abnormal status of equipment operation, so as to further maintain abnormal equipment to reduce production losses.

[0003] In recent years, environmental problems caused by industrial production have given rise to two new era goals of carbon peak and carbon neutrality in my country. The best way to achieve the two goals is to find renewable clean energy to replace the consumption of traditional energy. Combined with my country's actual situation, wind energy and solar energy are the core forces in the new energy industry in recent years. By the end of 2023, the cumulative installed capacity of wind power in China has reached 441 million kilowatts, a year-on-year increase of 20.7%; the national solar power generation installed capacity is about 610 million kilowatts, a year-on-year increase of 55.2%, and the total installed capacity of renewable energy has reached more than 1,000GW.

[0004] A new type of industrial application scenario has emerged in this type of industrial production environment. This production environment has a consistent system feature. Taking a wind power plant as an example, at the end of each wind turbine, there will be detection sensors for wind turbines and various equipment. Each sensor generates time series data that can reflect the operating status of the wind turbine. In addition to the sensor, there are generally some small computing terminals for data collection at the end. Since wind farms cover a large area, the cost of connecting each wind turbine to a high-bandwidth network is extremely high. At the same time, due to the special properties of the power system, certain privacy protection requirements are required for the transmitted data. In order to effectively monitor the operating status of each wind turbine and to be able to timely discover abnormal operating conditions in a large number of wind turbines at a low labor cost, it is necessary to perform anomaly detection on the time series data generated by each terminal.

[0005] By abstracting and summarizing the above example of wind farms, it is not difficult to find the system characteristics of this type of time series anomaly detection application scenario, that is, there are a large number of small, low-cost, and low-power terminals distributed over a long distance in the space. Due to various factors, it is impossible to provide a high-quality and high-bandwidth communication network for each terminal, and there is a certain need for privacy protection of the internally transmitted data.

[0006] However, if the algorithm is deployed on a central server, a large amount of bandwidth is required to transmit data to the central service and bring the risk of privacy leakage; then, if traditional distributed non-neural network time series anomaly detection algorithms are deployed on the data terminal, such as distributed K-means, distributed probability models, etc., it is necessary to transmit data features to the central server through an unreliable network, which will still bring the problem of data privacy leakage; if only neural network-based time series anomaly detection algorithms, such as TranAD, USAD, etc., are used and deployed on the data terminal, the algorithm can only use local data and cannot effectively perceive the global situation.

[0007] Therefore, in order to adapt to this special physical environment, how to effectively utilize data from multiple sources in a low-bandwidth data transmission environment and provide anomaly detection solutions for time series data on all terminal devices is an urgent problem to be solved. Summary of the invention

[0008] In view of this, the present invention provides a federated learning sensor anomaly detection method and system based on the Bert model, which can perform distributed anomaly detection and is suitable for anomaly detection through time series data generated by effective distributed sensors in low communication cost situations.

[0009] To achieve the above-mentioned purpose, the present invention provides a federated learning sensor anomaly detection method based on the Bert model, which specifically comprises: using at least one anomaly detection terminal and a central server to perform anomaly detection, wherein each anomaly detection terminal maintains a local model, and the central server maintains a global model, and both the local model and the global model adopt the Bert model. The process of the anomaly detection method specifically includes the following steps:

[0010] Step 1: Arrange corresponding temperature sensors at key nodes according to the requirements of wind farm temperature detection, and connect the temperature sensors to the anomaly detection terminal.

[0011] Step 2: The central server initializes the following parameters according to the wind farm environment, including the number of rounds of training, learning rate and detection window, and sends the initialized parameters to each anomaly detection terminal.

[0012] Step 3: The anomaly detection terminal trains the local model using local data according to the initialization parameters sent by the central server.

[0013] Step 4: After completing the local training rounds with parameter settings, the anomaly detection terminal sends the local model gradient adjustment parameters back to the central server, which processes them and updates the global model parameters and sends them back to each anomaly detection terminal for the next training step.

[0014] Step 5: After the anomaly detection terminal receives the global model parameters sent back by the central server, it updates the local model for the next round of training.

[0015] Step 6: The central server and the anomaly detection terminal repeat steps 1-4 in a loop until all local models and the global model achieve ideal detection results, completing the overall training.

[0016] Step 7: After the overall training is completed, each detection terminal performs anomaly detection on the local data and transmits the detection results back to the central server as the results of the anomaly detection of all sensors in the wind farm.

[0017] Furthermore, key nodes include wind turbines and transformers.

[0018] Furthermore, step 3 is specifically as follows:

[0019] The anomaly detection terminal first has two parts of local data built in. One part is the sensor data collected during the period when the system is running normally, which is the clean data; the other part is the sensor data collected under the normal operation state of the system. This part of the data contains the anomalies that occur during the operation of the system, including abnormal temperature changes and speed changes, and the anomalies are marked. This part of the data is the dirty data; clean data and dirty data are collectively referred to as training data and are obtained through human supervision.

[0020] Both clean data and dirty data are generated and stored at the anomaly detection terminal. Each anomaly detection terminal uses local training data to train the local anomaly detection algorithm based on the Bert model. The specific data used is selected according to the different training stages of the model.

[0021] In the model training phase, the abnormal terminal uses clean data to learn the patterns and knowledge of time series data through unsupervised training to complete several training tasks of the Bert model. The model training phase of the Bert model is divided into two sub-phases: the MDP sub-phase and the NSP sub-phase, as follows:

[0022] MDP sub-stage: In the MDP task, the Bert model is trained to predict the data hidden in the input sequence; specifically, for each input sequence, a certain proportion of data points are randomly selected and replaced with a special mask symbol [MASK]; the task of the Bert model is to guess the specific data that is masked; the MDP stage enables the Bert model to learn to understand the contextual information of time series data by training the model's ability to predict different time series data points at different positions.

[0023] NSP sub-stage: In the NSP task, the Bert model is trained to determine whether two sequences are continuous; given an input of a sequence pair, one of which starts with a [CLS] marker and the other ends with a [SEP] marker; the task of the Bert model is to predict whether the two sequences are continuous, outputting 0 if the two sequences are continuous and outputting 1 if the two sequences are discontinuous; through the NSP task, the Bert model learns to capture long-distance dependencies between time series.

[0024] The model training phase outputs the initially trained Bert model.

[0025] During the model fine-tuning stage, the anomaly detection terminal uses dirty data to further fine-tune the output of the initially trained BERT model, so that the initially trained BERT model can more accurately detect system anomalies contained in sensor data.

[0026] Furthermore, in step 4, the central server executes the following process:

[0027] S4.1. The central server receives the gradient information obtained after training of all anomaly detection terminals.

[0028] S4.2. The central server receives the gradient information of all anomaly detection terminals and aggregates them.

[0029] S4.3. The central server updates the parameters of the global model based on the aggregated gradient information. The purpose of the central server updating the global model parameters is to allow the global model to be shared among all anomaly detection terminals and to adapt to the data distribution of all anomaly detection terminals.

[0030] S4.4. After the central server updates the global model parameters, it distributes the updated global model to all anomaly detection terminals. After receiving the global model, the anomaly detection terminal updates the parameters of the local model to make the local model consistent with the global model. At the beginning of the next training round, all anomaly detection terminals use the updated global model for training.

[0031] Further, step 5, the specific steps are as follows:

[0032] S5.1. The anomaly detection terminal first verifies the received global model to ensure the integrity and correctness of the global model. After the verification, the anomaly detection terminal compares the global model with the local model.

[0033] S5.2. If the version number of the global model is consistent with the version number of the local model, the anomaly detection terminal uses the global model to overwrite the local model; if the version number of the global model is inconsistent with the version number of the local model, the anomaly detection terminal merges the local model and the global model to ensure that the local model is consistent with the global model;

[0034] S5.3. After completing the model synchronization, the anomaly detection terminal updates the local model so that subsequent training and prediction tasks are based on the latest global model.

[0035] Another embodiment of the present invention also provides a federated learning sensor anomaly detection system based on the Bert model, including at least one anomaly detection terminal and a central server; wherein each anomaly detection terminal maintains a local model, and the central server maintains a global model, and both the local model and the global model adopt the Bert model.

[0036] The anomaly detection terminal is used to connect to the corresponding temperature sensors arranged on the key nodes; the local Bert module, the local model training module, the local model parameter transmission module and the anomaly determination module are set on the anomaly detection terminal.

[0037] The local Bert module contains the structure of the Bert model, which is used to learn the features in local time series data and make abnormal judgments on subsequent data.

[0038] The local model training module is used to train the local Bert model by using local data combined with the global model update parameters sent by the central server.

[0039] The local model parameter transmission module is used to send the local model gradient adjustment parameters back to the global model training module of the central server after completing the local training round.

[0040] The local model updating module is used to update the local model after receiving the global model parameters sent back by the central server.

[0041] An initialization module, a global model training module, a global model parameter transmission module and an anomaly detection result storage module are set on the central server.

[0042] The initialization module initializes the following parameters according to the wind farm environment, including the number of rounds of training, learning rate and detection window, and sends the initialized parameters to the local model training module of each anomaly detection terminal.

[0043] After receiving the local model gradient adjustment parameters sent by each anomaly detection terminal, the global model training module trains the global model and updates the global model parameters.

[0044] The global model parameter transmission module is used to send the updated global model parameters back to the local model training module of each anomaly detection terminal.

[0045] The local model training module of the anomaly detection terminal updates the local model after receiving the global model parameters sent back by the central server for the next round of training.

[0046] The global model training module of the central server and the local model training module of the anomaly detection terminal cyclically execute the training process until each local model and the global model achieve the ideal detection results, completing the overall training.

[0047] After the overall training is completed, each anomaly detection terminal performs anomaly detection on the local data and transmits the detection results back to the anomaly detection result storage module of the central server to store the results of the anomaly detection of all sensors in the wind farm.

[0048] Furthermore, key nodes include wind turbines and transformers.

[0049] Furthermore, the local model training module specifically adopts the following method to perform local model training.

[0050] The anomaly detection terminal first has two parts of local data built in. One part is the sensor data collected during the period when the system is running normally, which is the clean data; the other part is the sensor data collected under the normal operation state of the system. This part of the data contains the anomalies that occur during the operation of the system, including abnormal temperature changes and speed changes, and the anomalies are marked. This part of the data is the dirty data; clean data and dirty data are collectively referred to as training data and are obtained through human supervision.

[0051] Both clean data and dirty data are generated and stored at the anomaly detection terminal. Each anomaly detection terminal uses local training data to train the local anomaly detection algorithm based on the Bert model. The specific data used is selected according to the different training stages of the model.

[0052] In the model training phase, the abnormal terminal uses clean data to learn the patterns and knowledge of time series data through unsupervised training to complete several training tasks of the Bert model. The model training phase of the Bert model is divided into two sub-phases: the MDP sub-phase and the NSP sub-phase, as follows:

[0053] MDP sub-stage: In the MDP task, the Bert model is trained to predict the data hidden in the input sequence; specifically, for each input sequence, a certain proportion of data points are randomly selected and replaced with a special mask symbol [MASK]; the task of the Bert model is to guess the specific data that is masked; the MDP stage enables the Bert model to learn to understand the contextual information of time series data by training the model's ability to predict different time series data points at different positions.

[0054] NSP sub-stage: In the NSP task, the Bert model is trained to determine whether two sequences are continuous; given an input of a sequence pair, one of which starts with a [CLS] marker and the other ends with a [SEP] marker; the task of the Bert model is to predict whether the two sequences are continuous, outputting 0 if the two sequences are continuous and outputting 1 if the two sequences are discontinuous; through the NSP task, the Bert model learns to capture long-distance dependencies between time series.

[0055] The model training phase outputs the initially trained Bert model.

[0056] During the model fine-tuning stage, the anomaly detection terminal uses dirty data to further fine-tune the output of the initially trained BERT model, so that the initially trained BERT model can more accurately detect system anomalies contained in sensor data.

[0057] Furthermore, the global model training module specifically performs global model training in the following manner:

[0058] Receive the gradient information obtained after training of all anomaly detection terminals and aggregate them; update the parameters of the global model based on the aggregated gradient information. The purpose of the central server updating the global model parameters is to make the global model shared among all anomaly detection terminals and adapt to the data distribution of all anomaly detection terminals.

[0059] Furthermore, the local model update module specifically performs the following steps:

[0060] First, the received global model is verified to ensure the integrity and correctness of the global model. After the verification, the anomaly detection terminal compares the global model with the local model.

[0061] If the version number of the global model is consistent with the local model version number, the anomaly detection terminal uses the global model to overwrite the local model; if the version number of the global model is inconsistent with the local model version number, the anomaly detection terminal merges the local model and the global model to ensure that the local model is consistent with the global model.

[0062] After model synchronization is completed, update the local model so that subsequent training and prediction tasks are based on the latest global model.

[0063] Beneficial effects:

[0064] 1: The federated learning sensor anomaly detection method based on the Bert model provided by the present invention can perform anomaly detection on time series data in an efficient and high-performance manner. At the same time, due to the use of the federated learning framework, all anomaly detection terminals can be trained with low network overhead under the premise of protecting privacy, so that each terminal can obtain information from other terminals to ensure more accurate anomaly detection results.

[0065] 2: In the actual production environment, a large number of terminals that generate time series data are widely physically distributed in a large area. Therefore, deploying a dedicated wired high-performance data network for each device will cause a lot of cost overhead. Only relatively low-cost wired networks or wireless networks can be used. However, the bandwidth and latency of the former are difficult to support real-time anomaly detection of data, and the latter will bring the risk of information leakage. In this environment, large-scale data transmission is bound to be difficult. Therefore, the present invention has significant production significance in this environment, so that the original data on each terminal can be sensed by the anomaly detection algorithm without exchanging data on other terminals. For example, in the environmental application of wind farms, this system can use different sensors, such as temperature sensors, speed sensors, etc., at a lower network construction cost, to timely detect abnormal conditions in wind turbines, such as fan overspeed, fan overtemperature, etc., reduce the labor cost in the operation of wind farms, protect related facilities, and improve efficiency.

[0066] 3: The method and system of the present invention propose a new application mode, which combines the Bert model to model time series data and anomaly detection. At the same time, the federated learning framework can effectively reduce the requirements for network latency and bandwidth in the distributed anomaly detection structure, and achieve a better balance between the accuracy of the detection results and the cost overhead. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Figure 1 is a diagram of the anomaly detection operating environment applied in this embodiment;

[0068] Figure 2 This is a schematic diagram of anomaly detection based on a distributed low-overhead transmission network of federated learning in this implementation mode;

[0069] Figure 3 This is a flow chart of anomaly detection based on a federated learning distributed low-overhead transmission network in this implementation mode;

[0070] Figure 4This is a structural diagram of anomaly detection based on a distributed low-overhead transmission network for federated learning in this embodiment. DETAILED DESCRIPTION

[0071] The present invention is described in detail below with reference to the accompanying drawings and embodiments.

[0072] The core idea of ​​the method of the present invention is: first, the central server discovers the detection terminal in the anomaly detection environment, establishes a link with the anomaly detection terminal and sends down the system parameters of the entire environment, such as the number of training iterations. After the discovery and link establishment are completed, each detection terminal uses local data to train the model and transmits the updated training data back to the central server. After receiving the results returned by the terminal, the central server updates the global model using the federated learning algorithm for the global results and transmits the results back to each anomaly detection terminal. The anomaly terminal then updates the parameters of the local model based on the global model. After completing a round of local and global data updates, iterative training is performed again according to the global parameter requirements, or the anomaly detection task can be performed after the training is completed.

[0073] In this implementation, the anomaly detection requirements running in the existing low-cost network are as follows: Figure 1 As shown; the principle diagram of the method of the present invention proposed a federated learning sensor anomaly detection method based on the Bert model is as follows Figure 2 shown.

[0074] This implementation uses a federated learning sensor anomaly detection method based on the Bert model, such as Figure 3 As shown, at least one anomaly detection terminal and a central server are used to perform anomaly detection, wherein each anomaly detection terminal maintains a local model, and the central server maintains a global model, and both the local model and the global model adopt the Bert model. The anomaly detection method specifically includes the following steps:

[0075] Step 1: Arrange corresponding temperature sensors at key nodes according to the requirements of temperature detection in wind farms, and connect the temperature sensors to abnormality detection terminals; key nodes include wind turbines and transformers.

[0076] Step 2: The central server initializes the following parameters according to the wind farm environment, including the number of rounds of training, learning rate, and detection window, and sends the initialized parameters to each anomaly detection terminal;

[0077] Step 3: The anomaly detection terminal uses local data to train the local model according to the initialization parameters sent by the central server. Specifically:

[0078] The anomaly detection terminal first has two parts of local data built in. One part is the sensor data collected during the normal operation of the system, which is clean data. The other part is the sensor data collected during the normal operation of the system. This part of the data contains anomalies that occur during the operation of the system, including abnormal temperature changes and speed changes, and the anomalies are marked. This part of the data is dirty data. Clean data and dirty data are collectively referred to as training data and are obtained through human supervision.

[0079] Both clean data and dirty data are generated and stored in the anomaly detection terminal. Each anomaly detection terminal uses local training data to train the local anomaly detection algorithm based on the Bert model. The specific data used is selected according to the different training stages of the model.

[0080] In the model training phase, the abnormal terminal uses clean data to learn the patterns and knowledge of time series data through unsupervised training to complete several training tasks of the Bert model. The model training phase of the Bert model is divided into two sub-phases: the MDP sub-phase and the NSP sub-phase, as follows:

[0081] MDP sub-stage: In the MDP task, the Bert model is trained to predict the data hidden in the input sequence; specifically, for each input sequence, a certain proportion of data points are randomly selected and replaced with a special mask symbol [MASK]; the task of the Bert model is to guess the specific data that is masked; the MDP stage enables the Bert model to learn to understand the contextual information of the time series data by training the model's ability to predict different time series data points at different locations;

[0082] NSP sub-stage: In the NSP task, the Bert model is trained to determine whether two sequences are continuous; given an input of a sequence pair, one of which starts with a [CLS] marker and the other ends with a [SEP] marker; the task of the Bert model is to predict whether the two sequences are continuous, outputting 0 if the two sequences are continuous and outputting 1 if the two sequences are discontinuous; through the NSP task, the Bert model learns to capture long-distance dependencies between time series.

[0083] The model training phase outputs the initially trained Bert model;

[0084] During the model fine-tuning stage, the anomaly detection terminal uses dirty data to further fine-tune the output of the initially trained BERT model, so that the initially trained BERT model can more accurately detect system anomalies contained in sensor data.

[0085] Step 4: After completing the local training rounds with parameter settings, the anomaly detection terminal sends the local model gradient adjustment parameters back to the central server, which processes them and updates the global model parameters and sends them back to each anomaly detection terminal for the next training step. The central server executes the following process:

[0086] S4.1. The central server receives the gradient information obtained after training of all anomaly detection terminals;

[0087] S4.2, the central server receives the gradient information of all anomaly detection terminals and aggregates them;

[0088] S4.3. The central server updates the parameters of the global model according to the aggregated gradient information. The purpose of the central server updating the parameters of the global model is to make the global model shared among all anomaly detection terminals and adapt to the data distribution of all anomaly detection terminals.

[0089] S4.4. After the central server updates the global model parameters, it distributes the updated global model to all anomaly detection terminals. After receiving the global model, the anomaly detection terminal updates the parameters of the local model to make the local model consistent with the global model. At the beginning of the next training round, all anomaly detection terminals use the updated global model for training.

[0090] Step 5: After the anomaly detection terminal receives the global model parameters sent back by the central server, the algorithm will update the local model for the next round of training; the specific steps are as follows:

[0091] S5.1. The anomaly detection terminal first verifies the received global model to ensure the integrity and correctness of the global model. After the verification, the anomaly detection terminal compares the global model with the local model.

[0092] S5.2. If the version number of the global model is consistent with the version number of the local model, the anomaly detection terminal uses the global model to overwrite the local model; if the version number of the global model is inconsistent with the version number of the local model, the anomaly detection terminal merges the local model and the global model to ensure that the local model is consistent with the global model;

[0093] S5.3. After completing the model synchronization, the anomaly detection terminal updates the local model so that subsequent training and prediction tasks are based on the latest global model.

[0094] Step 6: The central server and the anomaly detection terminal repeat steps 1-4 in a loop until all local models and the global model achieve the desired detection results, completing the overall training;

[0095] Step 7: After the overall training is completed, each detection terminal performs anomaly detection on the local data and transmits the detection results back to the central server as the results of the anomaly detection of all sensors in the wind farm.

[0096] The second aspect of this implementation method proposes a federated learning time series anomaly detection system and device based on the Bert model, such as Figure 4 As shown, the system includes at least one anomaly detection terminal and a central server; each anomaly detection terminal maintains a local model, and the central server maintains a global model, and both the local model and the global model adopt the Bert model;

[0097] The anomaly detection terminal is used to connect to the corresponding temperature sensors arranged on the key nodes; the local Bert module, the local model training module, the local model parameter transmission module, and the anomaly determination module are set on the anomaly detection terminal; the key nodes in the embodiment of the present invention include fans and transformers.

[0098] The local Bert module contains the structure of the Bert model, which is used to learn the features in the local time series data and make abnormal judgments on subsequent data;

[0099] The local model training module is used to train the local Bert model by combining local data with the global model update parameters sent by the central server;

[0100] A local model parameter transmission module, used to send the local model gradient adjustment parameters back to the global model training module of the central server after completing the local training round;

[0101] The local model update module is used to update the local model after receiving the global model parameters sent back by the central server;

[0102] An initialization module, a global model training module, a global model parameter transmission module, and an anomaly detection result storage module are set on the central server;

[0103] The initialization module initializes the following parameters according to the wind farm environment, including the number of rounds of training, learning rate and detection window, and sends the initialized parameters to the local model training module of each anomaly detection terminal;

[0104] After receiving the local model gradient adjustment parameters sent by each anomaly detection terminal, the global model training module trains the global model and updates the global model parameters;

[0105] A global model parameter transmission module, used to send updated global model parameters back to the local model training modules of each anomaly detection terminal;

[0106] In the local model training module of the anomaly detection terminal, after receiving the global model parameters sent back by the central server, the algorithm will update the local model for the next round of training;

[0107] The global model training module of the central server and the local model training module of the anomaly detection terminal cyclically execute the training process until each local model and the global model achieve the ideal detection results, completing the overall training;

[0108] After the overall training is completed, each anomaly detection terminal performs anomaly detection on the local data and transmits the detection results back to the anomaly detection result storage module of the central server to store the results of the anomaly detection of all sensors in the wind farm.

[0109] In the embodiment of the present invention, the local model training module specifically adopts the following method to perform local model training;

[0110] The anomaly detection terminal first has two parts of local data built in. One part is the sensor data collected during the normal operation of the system, which is clean data. The other part is the sensor data collected during the normal operation of the system. This part of the data contains anomalies that occur during the operation of the system, including abnormal temperature changes and speed changes, and the anomalies are marked. This part of the data is dirty data. Clean data and dirty data are collectively referred to as training data and are obtained through human supervision.

[0111] Both clean data and dirty data are generated and stored in the anomaly detection terminal. Each anomaly detection terminal uses local training data to train the local anomaly detection algorithm based on the Bert model. The specific data used is selected according to the different training stages of the model.

[0112] In the model training phase, the abnormal terminal uses clean data to learn the patterns and knowledge of time series data through unsupervised training to complete several training tasks of the Bert model. The model training phase of the Bert model is divided into two sub-phases: the MDP sub-phase and the NSP sub-phase, as follows:

[0113] MDP sub-stage: In the MDP task, the Bert model is trained to predict the data hidden in the input sequence; specifically, for each input sequence, a certain proportion of data points are randomly selected and replaced with a special mask symbol [MASK]; the task of the Bert model is to guess the specific data that is masked; the MDP stage enables the Bert model to learn to understand the contextual information of the time series data by training the model's ability to predict different time series data points at different locations;

[0114] NSP sub-stage: In the NSP task, the Bert model is trained to determine whether two sequences are continuous; given an input of a sequence pair, one of which starts with a [CLS] marker and the other ends with a [SEP] marker; the task of the Bert model is to predict whether the two sequences are continuous, outputting 0 if the two sequences are continuous and outputting 1 if the two sequences are discontinuous; through the NSP task, the Bert model learns to capture long-distance dependencies between time series.

[0115] The model training phase outputs the initially trained Bert model;

[0116] During the model fine-tuning stage, the anomaly detection terminal uses dirty data to further fine-tune the output of the initially trained BERT model, so that the initially trained BERT model can more accurately detect system anomalies contained in sensor data.

[0117] In an embodiment of the present invention, the global model training module specifically performs global model training in the following manner: receiving and aggregating the gradient information obtained after training of all anomaly detection terminals; updating the parameters of the global model according to the aggregated gradient information. The purpose of the central server updating the global model parameters is to allow the global model to be shared among all anomaly detection terminals and to adapt to the data distribution of all anomaly detection terminals.

[0118] The local model updating module in the embodiment of the present invention specifically performs the following steps:

[0119] First, the received global model is verified to ensure the integrity and correctness of the global model. After the verification is passed, the anomaly detection terminal compares the global model with the local model;

[0120] If the version number of the global model is consistent with the version number of the local model, the anomaly detection terminal uses the global model to overwrite the local model; if the version number of the global model is inconsistent with the version number of the local model, the anomaly detection terminal merges the local model and the global model to ensure that the local model is consistent with the global model;

[0121] After model synchronization is completed, update the local model so that subsequent training and prediction tasks are based on the latest global model.

[0122] In summary, the above are only preferred embodiments of the present invention and are not intended to limit the protection scope of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A federated learning sensor anomaly detection method based on the Bert model, characterized in that: At least one anomaly detection terminal and a central server are used to perform anomaly detection, wherein each anomaly detection terminal maintains a local model, and the central server maintains a global model, and both the local model and the global model adopt the Bert model. The anomaly detection method specifically includes the following steps: Step 1: Arrange corresponding temperature sensors at key nodes according to the requirements of wind farm temperature detection, and connect the temperature sensors to the abnormality detection terminal; Step 2: The central server initializes the following parameters according to the wind farm environment, including the number of rounds of training, learning rate, and detection window, and sends the initialized parameters to each anomaly detection terminal; Step 3: The anomaly detection terminal trains the local model using local data according to the initialization parameters sent by the central server; Step 4: After completing the local training rounds with parameter settings, the anomaly detection terminal sends the local model gradient adjustment parameters back to the central server, which processes them and updates the global model parameters and sends them back to each anomaly detection terminal for the next training step; Step 5: After the anomaly detection terminal receives the global model parameters sent back by the central server, it updates the local model for the next round of training; Step 6: The central server and the anomaly detection terminal repeat steps 1-4 in a loop until all local models and the global model achieve the desired detection results, completing the overall training; Step 7: After the overall training is completed, each detection terminal performs anomaly detection on the local data and transmits the detection results back to the central server as the results of the anomaly detection of all sensors in the wind farm.

2. The method for detecting abnormality of a federated learning sensor based on the Bert model according to claim 1, characterized in that: The key nodes include fans and transformers.

3. The method for detecting abnormality of a federated learning sensor based on the Bert model according to claim 1, characterized in that: The step 3 is specifically: The anomaly detection terminal first has two parts of local data built in. One part is the sensor data collected during the normal operation of the system, which is clean data. The other part is the sensor data collected during the normal operation of the system. This part of the data contains anomalies that occur during the operation of the system, including abnormal temperature changes and speed changes, and the anomalies are marked. This part of the data is dirty data. Clean data and dirty data are collectively referred to as training data and are obtained through human supervision. Both clean data and dirty data are generated and stored at the anomaly detection terminal. Each anomaly detection terminal uses local training data to train a local anomaly detection algorithm based on the Bert model. The specific data used is selected according to the different training stages of the model. In the model training phase, the abnormal terminal uses clean data to learn the patterns and knowledge of time series data through unsupervised training to complete several training tasks of the Bert model. The model training phase of the Bert model is divided into two sub-phases: the MDP sub-phase and the NSP sub-phase, as follows: MDP sub-stage: In the MDP task, the Bert model is trained to predict the data hidden in the input sequence; specifically, for each input sequence, a certain proportion of data points are randomly selected and replaced with a special mask symbol [MASK]; the task of the Bert model is to guess the specific data that is masked; The MDP stage enables the Bert model to understand the contextual information of time series data by training the model's ability to predict different time series data points at different locations; NSP sub-stage: In the NSP task, the Bert model is trained to determine whether two sequences are continuous; given an input of a sequence pair, one of which starts with a [CLS] marker and the other ends with a [SEP] marker; the task of the Bert model is to predict whether the two sequences are continuous, outputting 0 if the two sequences are continuous and outputting 1 if the two sequences are discontinuous; through the NSP task, the Bert model learns to capture long-distance dependencies between time series; The model training phase outputs the initially trained Bert model; During the model fine-tuning stage, the anomaly detection terminal uses dirty data to further fine-tune the output of the initially trained BERT model, so that the initially trained BERT model can more accurately detect system anomalies contained in sensor data.

4. The method for detecting abnormality of a federated learning sensor based on the Bert model according to claim 1, characterized in that: In step 4, the central server executes the following process: S4.

1. The central server receives the gradient information obtained after training of all anomaly detection terminals; S4.2, the central server receives the gradient information of all anomaly detection terminals and aggregates them; S4.3, the central server updates the parameters of the global model according to the aggregated gradient information. The purpose of the central server updating the parameters of the global model is to allow the global model to be shared among all anomaly detection terminals and to adapt to the data distribution of all anomaly detection terminals; S4.

4. After the central server updates the global model parameters, it distributes the updated global model to all anomaly detection terminals; after receiving the global model, the anomaly detection terminal updates the parameters of the local model to make the local model consistent with the global model. At the beginning of the next training round, all anomaly detection terminals use the updated global model for training.

5. The method for detecting abnormality of a federated learning sensor based on the Bert model according to claim 4, characterized in that: The specific steps of step 5 are as follows: S5.

1. The anomaly detection terminal first verifies the received global model to ensure the integrity and correctness of the global model. After the verification, the anomaly detection terminal compares the global model with the local model. S5.

2. If the version number of the global model is consistent with the version number of the local model, the anomaly detection terminal uses the global model to overwrite the local model; if the version number of the global model is inconsistent with the version number of the local model, the anomaly detection terminal merges the local model and the global model to ensure that the local model is consistent with the global model; S5.

3. After completing the model synchronization, the anomaly detection terminal updates the local model so that subsequent training and prediction tasks are based on the latest global model.

6. The federated learning sensor anomaly detection system based on the Bert model is characterized by: It includes at least one anomaly detection terminal and a central server; each anomaly detection terminal maintains a local model, and the central server maintains a global model, and both the local model and the global model adopt the Bert model; The anomaly detection terminal is used to connect to the corresponding temperature sensors arranged on the key nodes; the anomaly detection terminal is provided with a local Bert module, a local model training module, a local model parameter transmission module, and an anomaly determination module; The local Bert module contains the structure of the Bert model, which is used to learn the features in the local time series data and make abnormal judgments on subsequent data; The local model training module is used to train the local Bert model by using local data combined with the global model update parameters sent by the central server; The local model parameter transmission module is used to send the local model gradient adjustment parameters back to the global model training module of the central server after completing the local training round; The local model updating module is used to update the local model after receiving the global model parameters returned by the central server; An initialization module, a global model training module, a global model parameter transmission module and an anomaly detection result storage module are set on the central server; The initialization module initializes the following parameters according to the wind farm environment, including the number of rounds of training, the learning rate and the detection window, and sends the initialized parameters to the local model training module of each anomaly detection terminal; After receiving the local model gradient adjustment parameters sent by each anomaly detection terminal, the global model training module trains the global model and updates the global model parameters; The global model parameter transmission module is used to send the updated global model parameters back to the local model training module of each anomaly detection terminal; The local model training module of the anomaly detection terminal updates the local model after receiving the global model parameters sent back by the central server for the next round of training; The global model training module of the central server and the local model training module of the anomaly detection terminal cyclically execute the training process until each local model and the global model achieve the ideal detection results, completing the overall training; After the overall training is completed, each anomaly detection terminal performs anomaly detection on the local data and transmits the detection results back to the anomaly detection result storage module of the central server to store the results of the anomaly detection of all sensors in the wind farm.

7. The federated learning sensor anomaly detection system based on the Bert model as claimed in claim 6, characterized in that: The key nodes include fans and transformers.

8. The federated learning sensor anomaly detection system based on the Bert model as claimed in claim 6, characterized in that: The local model training module specifically uses the following method to perform local model training; The anomaly detection terminal first has two parts of local data built in. One part is the sensor data collected during the normal operation of the system, which is clean data. The other part is the sensor data collected during the normal operation of the system. This part of the data contains anomalies that occur during the operation of the system, including abnormal temperature changes and speed changes, and the anomalies are marked. This part of the data is dirty data. Clean data and dirty data are collectively referred to as training data and are obtained through human supervision. Both clean data and dirty data are generated and stored at the anomaly detection terminal. Each anomaly detection terminal uses local training data to train a local anomaly detection algorithm based on the Bert model. The specific data used is selected according to the different training stages of the model. In the model training phase, the abnormal terminal uses clean data to learn the patterns and knowledge of time series data through unsupervised training to complete several training tasks of the Bert model. The model training phase of the Bert model is divided into two sub-phases: the MDP sub-phase and the NSP sub-phase, as follows: MDP sub-stage: In the MDP task, the Bert model is trained to predict the data hidden in the input sequence; specifically, for each input sequence, a certain proportion of data points are randomly selected and replaced with a special mask symbol [MASK]; the task of the Bert model is to guess the specific data that is masked; The MDP stage enables the Bert model to understand the contextual information of time series data by training the model's ability to predict different time series data points at different locations; NSP sub-stage: In the NSP task, the Bert model is trained to determine whether two sequences are continuous; given an input of a sequence pair, one of which starts with a [CLS] marker and the other ends with a [SEP] marker; the task of the Bert model is to predict whether the two sequences are continuous, outputting 0 if the two sequences are continuous and outputting 1 if the two sequences are discontinuous; through the NSP task, the Bert model learns to capture long-distance dependencies between time series; The model training phase outputs the initially trained Bert model; During the model fine-tuning stage, the anomaly detection terminal uses dirty data to further fine-tune the output of the initially trained BERT model, so that the initially trained BERT model can more accurately detect system anomalies contained in sensor data.

9. The federated learning sensor anomaly detection system based on the Bert model as claimed in claim 6, characterized in that: The global model training module specifically performs global model training in the following manner: Receive and aggregate the gradient information obtained after training of all anomaly detection terminals; update the parameters of the global model according to the aggregated gradient information. The purpose of the central server updating the global model parameters is to share the global model among all anomaly detection terminals and adapt to the data distribution of all anomaly detection terminals.

10. The federated learning sensor anomaly detection system based on the Bert model according to claim 6, characterized in that: The local model updating module specifically performs the following steps: First, the received global model is verified to ensure the integrity and correctness of the global model. After the verification is passed, the anomaly detection terminal compares the global model with the local model; If the version number of the global model is consistent with the version number of the local model, the anomaly detection terminal uses the global model to overwrite the local model; if the version number of the global model is inconsistent with the version number of the local model, the anomaly detection terminal merges the local model and the global model to ensure that the local model is consistent with the global model; After model synchronization is completed, update the local model so that subsequent training and prediction tasks are based on the latest global model.

Images