Shared data manipulation detection method and system

Abstract

The invention belongs to the technical field of deep learning, and particularly relates to a shared data manipulation detection method and system, and the method comprises the following steps: a data collection module collects shared data in real time from each node of a distributed optimization algorithm; the data preprocessing module performs cleaning, standardization and feature extraction on the acquired data; a convolutional neural network CNN is selected as a basic model, and the model is trained, so that a deep learning module is constructed; and the early warning and response module performs early warning and response according to an output result of the deep learning module. According to the invention, through real-time monitoring and early warning of data manipulation behaviors, the safety and stability of a distributed optimization algorithm are effectively improved. The method comprises the steps of data collection, preprocessing, data manipulation detection, early warning and coping and the like, automatic learning and recognition are carried out in combination with a neural network model, the accuracy and real-time performance of data manipulation detection are improved, and powerful support is provided for safety guarantee of a distributed optimization algorithm.

Description

Technical Field

[0001] The present invention belongs to the field of deep learning technology, and specifically relates to a shared data manipulation detection method and system. Background Art

[0002] In distributed optimization algorithms, data interaction between nodes is key, but it also faces the threat of data manipulation. Traditional data manipulation detection methods are mainly based on rule matching or simple statistical analysis, but these methods often do not work well when dealing with large-scale, high-dimensional and dynamic data. In recent years, machine learning and artificial intelligence technologies have been widely used in the field of data manipulation detection, but due to the explosive growth of data scale and the increase in algorithm complexity, existing machine learning methods are also facing challenges.

[0003] Traditional data manipulation detection methods have the following defects: Limitations of rule matching: Traditional detection methods rely on predefined rules to identify abnormal behaviors. However, these rules may not adapt to the ever-changing data environment, especially when facing complex and dynamic data, and are prone to false positives or false negatives. Insufficient statistical analysis: Traditional statistical analysis methods are usually modeled based on historical data, which makes them perform poorly when faced with new or unseen data patterns. In addition, these methods often lack sensitivity to changes in data distribution, making it difficult to accurately judge abnormal behaviors in practical applications. Challenges in large-scale data processing: With the explosive growth of data volume, traditional detection methods face significant performance bottlenecks when processing large-scale data. For example, in distributed systems, due to the wide distribution and dynamic changes of data, traditional detection methods have difficulty in effectively identifying and processing data flows across nodes. Lack of adaptability and scalability: Many traditional methods perform well in specific scenarios, but lack versatility and scalability across multiple fields or different data sets. This limits their application effects in different environments. Insufficient understanding of the nature of anomalies: Abnormal events have specific properties and characteristics, which are often difficult for traditional methods to capture. For example, some abnormal events may disguise themselves as normal data, thereby bypassing traditional detection mechanisms. Computing resource requirements: Many traditional methods require a lot of computing resources to process data, especially in scenarios with high real-time requirements, such as critical applications such as healthcare. This requirement may lead to delays and misjudgments. Summary of the invention

[0004] In response to the technical problems of defects in the above-mentioned traditional data manipulation detection methods, the present invention provides a shared data manipulation detection method and system. The method combines convolutional neural networks for feature extraction and classification, can automatically learn and identify data manipulation behaviors without preset rules or manual feature engineering, has higher detection accuracy and better real-time performance, and can better adapt to complex and changeable practical application scenarios.

[0005] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0006] A shared data manipulation detection method comprises the following steps:

[0007] S1, the data acquisition module collects shared data from each node of the distributed optimization algorithm in real time;

[0008] S2, the data preprocessing module cleans, standardizes and extracts features from the collected data;

[0009] S3, select the convolutional neural network CNN as the basic model, train the model, and thus build a deep learning module;

[0010] S4. The early warning and response module performs early warning and response according to the output results of the deep learning module.

[0011] The data in S1 includes the iteratively updated solution vector and constraint conditions.

[0012] The method for real-time collection of shared data from each node of the distributed optimization algorithm in S1 is as follows: the data collection module adopts a distributed architecture to communicate securely with each node; the data collection frequency is set to once per second so as to capture any abnormal behavior in time; and the data is encrypted during transmission to ensure data security.

[0013] The method for cleaning the collected data in S2 is as follows: the data cleaning process includes removing outliers, filling missing values ​​and correcting erroneous data; removing outliers: using box plots and Z-score methods to identify outliers in the data, and directly deleting clearly unreasonable outliers; considering adjusting the data that needs to be retained to a reasonable range; determining upper and lower limits by calculating quartiles and interquartile ranges, and values ​​below or above these ranges are replaced with NaN to delete outliers; filling missing values: by calculating the average value of a feature and filling missing values ​​with this average value; correcting erroneous data: using data cleaning tools to handle errors and inconsistencies in the data, and data cleaning tools include removing duplicate data, filling missing values, and standardizing data formats.

[0014] The standardization method in S2 is: the data is standardized to a standard normal distribution through minimum-maximum normalization or Z-score normalization to improve the predictive performance of the model.

[0015] The feature extraction method in S2 is as follows: the features include statistical characteristics of the data, time series patterns, and indicators related to algorithm performance; a feature selection method based on association rules is used to maximize the correlation between the features and data manipulation behaviors.

[0016] The method for training the model in S3 is as follows: first, normalize the data and limit the data values ​​to the range of [-1, +1] to ensure that all features are on the same scale, thereby improving the efficiency and effect of model training; then, divide the preprocessed data set into a training set, a validation set, and a test set. The data set will be allocated in proportion, 80% for training, 10% for validation, and the remaining 10% for testing; next, during the training process, it is necessary to adjust the model's hyperparameters to optimize the model performance. The hyperparameters include the learning rate and the batch size. Through the K-fold cross-validation technique, Effectively evaluate the model performance under different hyperparameter combinations; in each iteration, use a part of the data as the validation set to evaluate the model performance, and adjust the hyperparameters according to the results until the best settings are found; use the training set to train the model, and perform multiple iterations on the validation set to adjust the hyperparameters; through 5-fold cross-validation, ensure that the performance of the model on different subsets is stable and reduce the impact of random errors; after the model converges, use the test set to evaluate the generalization ability of the final model to ensure that the model can effectively predict new data; finally, evaluate the performance of the model by calculating the accuracy, precision, and recall indicators.

[0017] The method of early warning and response based on the output results of the deep learning module in S4 is as follows: when data manipulation behavior is detected, the early warning module will immediately trigger the early warning mechanism and notify the administrator by sound, light or text message; at the same time, the response module will take corresponding measures to limit the information interaction of the manipulated nodes or resynchronize the data, etc.; the early warning and response module will also update the weight of the model to improve the prediction performance and prepare for the next iteration.

[0018] A shared data manipulation detection system includes a data acquisition module, a data preprocessing module, a deep learning module, and an early warning and response module. The data acquisition module is connected to the data preprocessing module, the data preprocessing module is connected to the deep learning module, and the deep learning module is connected to the early warning and response module.

[0019] Compared with the prior art, the present invention has the following beneficial effects:

[0020] The present invention proposes a shared data manipulation detection method and system based on theoretical convergence trajectory analysis, which effectively improves the security and stability of distributed optimization algorithms by real-time monitoring and early warning of data manipulation behaviors. The method includes steps such as data collection, preprocessing, data manipulation detection, early warning and response, and combines the neural network model for automatic learning and recognition, which improves the accuracy and real-time performance of data manipulation detection and provides strong support for the security of distributed optimization algorithms. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the implementation methods of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for the implementation methods or the description of the prior art. Obviously, the drawings in the following description are only exemplary, and for ordinary technicians in this field, other implementation drawings can be derived from the provided drawings without creative work.

[0022] The structures, proportions, sizes, etc. illustrated in this specification are only used to match the contents disclosed in the specification so as to facilitate understanding and reading by persons familiar with the technology. They are not used to limit the conditions under which the present invention can be implemented, and therefore have no substantial technical significance. Any structural modification, change in proportion or adjustment of size shall still fall within the scope of the technical contents disclosed in the present invention without affecting the effects and purposes that can be achieved by the present invention.

[0023] Figure 1 It is a flow chart of the steps of the present invention. DETAILED DESCRIPTION

[0024] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. These descriptions are only to further illustrate the features and advantages of the present invention, rather than to limit the claims of the present invention. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0025] The specific implementation of the present invention is further described in detail below in conjunction with the accompanying drawings and examples. The following examples are used to illustrate the present invention, but are not intended to limit the scope of the present invention.

[0026] A shared data manipulation detection method, such as Figure 1 As shown, the following steps are included:

[0027] Step 1: The data acquisition module collects shared data from each node of the distributed optimization algorithm in real time. The data acquisition module adopts a distributed architecture to communicate securely with each node. The data collection frequency is set to once per second to capture any abnormal behavior in time. The data is encrypted during transmission to ensure data security.

[0028] Step 2: The data preprocessing module cleans, standardizes and extracts features from the collected data.

[0029] The data cleaning process includes removing outliers, filling missing values, and correcting erroneous data. Removing outliers: Use box plots and Z-scores to identify outliers in the data. For clearly unreasonable outliers, delete them directly. For data that needs to be retained, consider adjusting them to a reasonable range. The upper and lower limits are determined by calculating quartiles and interquartile ranges. Values ​​below or above these ranges are replaced with NaN to remove outliers. Filling missing values: Calculate the mean of a feature and use this mean to fill missing values. Correcting erroneous data: Use data cleaning tools to handle errors and inconsistencies in the data. Data cleaning tools include removing duplicate data, filling missing values, and standardizing data formats.

[0030] The data were normalized to a standard normal distribution by min-max normalization or Z-score normalization to improve the predictive performance of the model.

[0031] Features include statistical properties of data, time series patterns, and indicators related to algorithm performance. A feature selection method based on association rules is used to maximize the relevance of features to data manipulation behaviors.

[0032] Step 3. Select the convolutional neural network CNN as the basic model and train the model to build a deep learning module. First, normalize the data and limit the data values ​​to the range of [-1, +1] to ensure that all features are on the same scale, thereby improving the efficiency and effect of model training. Then, divide the preprocessed data set into training set, validation set and test set. The data set will be allocated in proportion, 80% for training, 10% for validation, and the remaining 10% for testing. Next, during the training process, the hyperparameters of the model need to be adjusted to optimize the model performance. The hyperparameters include learning rate and batch size. The K-fold cross-validation technique is used to effectively evaluate the model performance under different hyperparameter combinations. In each iteration, a part of the data is used as a validation set to evaluate the model performance, and the hyperparameters are adjusted according to the results until the best settings are found. The model is trained using the training set, and multiple iterations are performed on the validation set to adjust the hyperparameters. Through 5-fold cross-validation, the performance of the model on different subsets is ensured to be stable and the impact of random errors is reduced. After the model converges, the test set is used to evaluate the generalization ability of the final model to ensure that the model can effectively predict new data. Finally, the performance of the model is evaluated by calculating the accuracy, precision, and recall indicators.

[0033] Step 4: The early warning and response module will issue early warnings and responses based on the output of the deep learning module. When data manipulation is detected, the early warning module will immediately trigger the early warning mechanism and notify the administrator through sound, light or text message. At the same time, the response module will take corresponding measures to limit the information interaction of the manipulated nodes or resynchronize data. The early warning and response module will also update the model weights to improve the prediction performance and prepare for the next iteration.

[0034] A shared data manipulation detection system includes a data acquisition module, a data preprocessing module, a deep learning module, and an early warning and response module. The data acquisition module is connected to the data preprocessing module, and its function is to collect shared data from each node of the distributed optimization algorithm in real time. The data preprocessing module is connected to the deep learning module, and the data preprocessing module is responsible for cleaning, standardizing and feature extraction of the collected data. The deep learning module is connected to the early warning and response module, and the deep learning module is used to learn and identify data manipulation behaviors. The early warning and response module performs early warning and response according to the output results of the deep learning module.

[0035] Embodiment 1

[0036] In this embodiment, a distributed linear optimization problem with 10 nodes is used as an example. Each node is responsible for a subproblem of the optimization problem and interacts with other nodes by sharing data. We use the standard gradient descent method as the optimization algorithm and use random data generation to simulate normal data interaction. At the same time, this embodiment also introduces some abnormal data manipulation behaviors, such as data tampering and injection of malicious solutions.

[0037] The specific parameter settings are as follows: The deep learning model uses a convolutional neural network, which contains two convolutional layers, a pooling layer, and a fully connected layer. The training data set contains 10,000 samples, and the validation set and test set contain 2,000 samples respectively. The Adam optimizer and mean square error loss function are used in the training process.

[0038] The experimental environment is 10 computers with the same configuration, connected through a network for distributed computing. The experimental results are as follows: the detection accuracy rate reaches 97%, the false alarm rate is reduced to 3%, the real-time performance is good, and the data manipulation behavior can be warned within 1 second.

[0039] By comparison, the performance of the present invention is better than that of the prior art. Under the same experimental conditions, the detection accuracy of the prior art is only 85%, and the false alarm rate is as high as 10%. The present invention improves the detection accuracy and reduces the false alarm rate through deep learning technology, and better guarantees the security and stability of the distributed optimization algorithm.

[0040] Embodiment 2

[0041] In this embodiment, a complex distributed nonlinear optimization problem is considered, with 20 nodes participating in the optimization, and normal data interaction and abnormal data manipulation behaviors are simulated.

[0042] The specific parameter settings are as follows: The deep learning model adopts the architecture of combining convolutional neural network and recurrent neural network, including two convolutional layers, one recurrent layer, one pooling layer and one fully connected layer. The training data set contains 20,000 samples, and the validation set and test set contain 4,000 samples respectively. The LSTM optimizer and cross entropy loss function are used in the training process.

[0043] The experimental environment is 20 computers with the same configuration, connected through a network for distributed computing. The experimental results are as follows: the detection accuracy rate reaches 95%, the false alarm rate is reduced to 5%, the real-time performance is good, and the data manipulation behavior can be warned within 500 milliseconds.

[0044] By comparison, the performance of the present invention is also better than the existing technology. Under the same experimental conditions, the detection accuracy of the existing technology is only 80%, and the false alarm rate is as high as 15%. The present invention further improves the detection accuracy and reduces the false alarm rate by improving the deep learning model and parameter settings, and better adapts to more complex distributed optimization problems.

[0045] In summary, through the verification of two embodiments, the present invention demonstrates the effective detection capability of shared data manipulation in distributed optimization algorithms. By introducing deep learning technology, the present invention significantly improves the detection accuracy and reduces the false alarm rate, and has good real-time and adaptability. This provides a strong guarantee for the security and stability of distributed optimization algorithms and has important practical application value.

[0046] Only the preferred embodiments of the present invention are described in detail above, but the present invention is not limited to the above embodiments. Various changes can be made within the knowledge scope of ordinary technicians in this field without departing from the purpose of the present invention, and various changes should be included in the protection scope of the present invention.

Claims

1. A shared data manipulation detection method, characterized in that: The following steps are involved: S1, the data acquisition module collects shared data from each node of the distributed optimization algorithm in real time; S2, the data preprocessing module cleans, standardizes and extracts features from the collected data; S3, select the convolutional neural network CNN as the basic model, train the model, and thus build a deep learning module; S4. The early warning and response module performs early warning and response according to the output results of the deep learning module.

2. A shared data manipulation detection method according to claim 1, characterized in that: The data in S1 includes the iteratively updated solution vector and constraint conditions.

3. A shared data manipulation detection method according to claim 1, characterized in that: The method for collecting shared data from each node of the distributed optimization algorithm in real time in S1 is as follows: the data collection module adopts a distributed architecture to communicate securely with each node; the data collection frequency is set to once per second so as to capture any abnormal behavior in time; The data is encrypted during transmission to ensure data security.

4. A shared data manipulation detection method according to claim 1, characterized in that: The method for cleaning the collected data in S2 is as follows: the data cleaning process includes removing outliers, filling missing values ​​and correcting erroneous data; removing outliers: using box plots and Z-score methods to identify outliers in the data, and directly deleting clearly unreasonable outliers; considering adjusting the data that needs to be retained to a reasonable range; determining upper and lower limits by calculating quartiles and interquartile ranges, and values ​​below or above these ranges are replaced with NaN to delete outliers; filling missing values: by calculating the average value of a feature and filling missing values ​​with this average value; correcting erroneous data: using data cleaning tools to handle errors and inconsistencies in the data, and data cleaning tools include removing duplicate data, filling missing values, and standardizing data formats.

5. A shared data manipulation detection method according to claim 1, characterized in that: The standardization method in S2 is: the data is standardized to a standard normal distribution through minimum-maximum normalization or Z-score normalization to improve the predictive performance of the model.

6. A shared data manipulation detection method according to claim 1, characterized in that: The feature extraction method in S2 is as follows: the features include statistical characteristics of the data, time series patterns, and indicators related to algorithm performance; a feature selection method based on association rules is used to maximize the correlation between the features and data manipulation behaviors.

7. A shared data manipulation detection method according to claim 1, characterized in that: The method for training the model in S3 is as follows: first, normalize the data and limit the data values ​​to the range of [-1, +1] to ensure that all features are on the same scale, thereby improving the efficiency and effect of model training; then, divide the preprocessed data set into a training set, a validation set, and a test set. The data set will be allocated in proportion, 80% for training, 10% for validation, and the remaining 10% for testing; next, during the training process, it is necessary to adjust the model's hyperparameters to optimize the model performance. The hyperparameters include the learning rate and the batch size. Through the K-fold cross-validation technique, Effectively evaluate the model performance under different hyperparameter combinations; in each iteration, use a part of the data as the validation set to evaluate the model performance, and adjust the hyperparameters according to the results until the best settings are found; use the training set to train the model, and perform multiple iterations on the validation set to adjust the hyperparameters; through 5-fold cross-validation, ensure that the performance of the model on different subsets is stable and reduce the impact of random errors; after the model converges, use the test set to evaluate the generalization ability of the final model to ensure that the model can effectively predict new data; finally, evaluate the performance of the model by calculating the accuracy, precision, and recall indicators.

8. A shared data manipulation detection method according to claim 1, characterized in that: The method of early warning and response according to the output result of the deep learning module in S4 is as follows: when data manipulation behavior is detected, the early warning module will immediately trigger the early warning mechanism and notify the administrator by sound, light or text message; at the same time, the response module will take corresponding measures to limit the information interaction of the manipulated node or resynchronize the data, etc.; The early warning and response module also updates the model weights to improve prediction performance in preparation for the next iteration.

9. A shared data manipulation detection system, characterized in that: It includes a data acquisition module, a data preprocessing module, a deep learning module, and an early warning and response module. The data acquisition module is connected to the data preprocessing module, the data preprocessing module is connected to the deep learning module, and the deep learning module is connected to the early warning and response module.

Images