Network data control method and device based on big data and electronic equipment

Abstract

The invention provides a network data control method and device based on big data and electronic equipment, and relates to the field of data processing. The method comprises the following steps: acquiring first network data stored in a first server; second network data stored by a second server is obtained, the second server is a backup server corresponding to the first server, and the second network data is backup data of the first network data; if it is determined that the first server has the network anomaly, deviation data are determined according to the first network data and the second network data, the deviation data are different data generated by the first server relative to the second server within a preset time period, and the preset time period is a time period when the first server has the network anomaly; and sending the deviation data to a second server and a third server for backup, wherein the third server is any one of the plurality of servers except the first server and the second server. By implementing the technical scheme provided by the invention, the network data security can be ensured.

Description

Technical Field

[0001] The present application relates to the technical field of data processing, and in particular to a network data control method, device and electronic equipment based on big data. Background Art

[0002] In today's digital age, with the rapid development of information technology, big data has become a key force in promoting innovation and development in all walks of life. However, with the explosive growth of data volume, data protection and security issues have become increasingly prominent, especially in the context of complex and ever-changing security threats in cyberspace. Traditional data backup methods can no longer meet the increasingly severe data security needs.

[0003] At present, the core of the traditional data backup mechanism is to mirror key business data to another server or dedicated storage device through regular or real-time data replication, as an emergency measure in the event of disaster recovery or data loss. However, this backup strategy may be inadequate in the face of complex attacks and is not conducive to data security.

[0004] Therefore, there is an urgent need for a network data control method, device and electronic equipment based on big data. Summary of the invention

[0005] The present application provides a network data control method, device and electronic equipment based on big data, which are convenient for ensuring network data security.

[0006] In a first aspect of the present application, a network data control method based on big data is provided, the method comprising: obtaining first network data stored in a first server; obtaining second network data stored in a second server, the second server being a backup server corresponding to the first server, and the second network data being backup data of the first network data; if it is determined that a network abnormality occurs in the first server, determining deviation data based on the first network data and the second network data, the deviation data being different data generated by the first server compared with the second server within a preset time period, the preset time period being the time period in which the network abnormality occurs in the first server; sending the deviation data to the second server and a third server for backup, the third server being any one of a plurality of servers except the first server and the second server.

[0007] By adopting the above technical solution, by maintaining a copy of the first network data on the second server, even if the first server has a network anomaly or data loss, the critical business data can be protected. In addition, when a network anomaly is detected and it is determined that there is deviated data, the deviated data is promptly sent to another independent third server for additional backup, further enhancing the security and redundancy of the data. After the network anomaly occurs, by comparing the data difference between the first server and the second server within a preset time period, it is possible to quickly locate and determine which data is the affected deviated data. This precise data recovery mechanism reduces unnecessary data transmission and processing time and improves the efficiency of data recovery. By monitoring the data difference between the first server and the second server and processing the deviated data immediately when a network anomaly is detected, this strategy supports real-time or near real-time protection of data. By sending the deviated data to the third server for backup, a multi-level backup system is constructed. This design enables the system to recover data from the third server even if the main server and the backup server encounter problems at the same time, thereby greatly enhancing the resilience and disaster resistance of the entire system and facilitating the security of network data.

[0008] Optionally, the acquiring of the first network data stored in the first server specifically includes: receiving original network data sent by the first server; preprocessing the original network data to obtain the first network data, wherein the preprocessing includes denoising, filtering, data feature extraction and normalization.

[0009] By adopting the above technical solutions, the original network data often contains noise, redundant information or data formats that do not meet the analysis requirements. Through preprocessing steps such as denoising and filtering, the impact of these adverse factors on subsequent data analysis can be effectively reduced, and the accuracy and reliability of the data can be improved. Data feature extraction helps to extract more valuable information for analysis from the original data, while normalization can ensure that data of different dimensions or ranges are comparable in subsequent processing. The preprocessing process usually includes a series of automated operations that can pre-organize and optimize the data before the data enters the analysis stage. In this way, in the subsequent data analysis, model training and other processes, the preprocessed high-quality data can be directly used, thereby avoiding repeated data cleaning work at each step, and significantly improving the overall efficiency of data processing. The preprocessed data is more in line with the analysis requirements and can better reflect the essential characteristics and laws of the data. This helps to improve the accuracy and depth of data analysis, making the analysis results more reliable and convincing. At the same time, high-quality data is also the basis for building efficient and accurate data models and algorithms. The original network data may contain a large amount of useless or redundant information, which will consume a lot of system resources during data transmission, storage and processing. By removing this information through the preprocessing process, the consumption of system resources can be significantly reduced and the overall performance and response speed of the system can be improved.

[0010] Optionally, obtaining the second network data stored on the second server specifically includes: determining a first storage identifier based on the first server and the first network data; determining the second server based on the first storage identifier; searching for the first storage identifier in the second server, and if it is determined that the second server has a second storage identifier corresponding to the first storage identifier, obtaining backup data corresponding to the second storage identifier to obtain the second network data, the first storage identifier and the second storage identifier are the same storage identifier.

[0011] By adopting the above technical solution, by matching the first storage identifier and the second storage identifier, it can be ensured that the second network data obtained from the second server is indeed an accurate backup of the first network data. This identification-based verification mechanism helps prevent data confusion or erroneous recovery and ensures data consistency and integrity. After determining the first storage identifier, the corresponding backup data can be directly found in the second server based on the identifier. This direct positioning method avoids unnecessary searches and traversals, and significantly improves the efficiency of data recovery. Using a unified storage identifier system enables the system to flexibly expand and manage data backup relationships between multiple servers. Whether adding a new backup server or adjusting an existing backup strategy, it can be achieved by simply updating the storage identifier mapping relationship without large-scale reconstruction or modification of the system. Managing the data backup and recovery process through storage identifiers can greatly simplify the data management process. Administrators or automated scripts can easily query, verify and restore data based on storage identifiers without in-depth understanding of the specific implementation details of the underlying storage system. The storage identifier matching process itself is also a security mechanism. It ensures that only verified requests with correct storage identifiers can access the corresponding backup data. This helps prevent unauthorized access and data leakage and improves data security.

[0012] Optionally, if it is determined that a network anomaly occurs in the first server, deviation data is determined based on the first network data and the second network data, specifically including: receiving real-time network traffic data sent by a first network probe and a second network probe, the first network probe being used to monitor the data entry of the first server, and the second network probe being used to monitor the data exit of the second server; inputting the real-time network traffic data into a preset isolation forest model to obtain an anomaly score corresponding to each data point, the real-time network traffic data including multiple data points; if it is determined that the anomaly score corresponding to the target data point is less than a preset threshold, it is determined that a network anomaly occurs in the target data point, then it is determined that a network anomaly occurs in the first server, the target data point is any one of the multiple data points, and the anomaly score corresponding to the target data point is determined based on the average traversal path length of the target data point in the preset isolation forest model.

[0013] By adopting the above technical solution, network traffic data can be captured in real time by deploying network probes at the data entry of the first server and the data exit of the second server. This real-time performance is crucial for timely detection of network anomalies, which helps to respond quickly and reduce potential losses. The isolation forest model is used to analyze real-time network traffic data, and the anomaly score of each data point is calculated to determine whether a network anomaly occurs. The isolation forest model is an effective unsupervised learning algorithm, which is particularly suitable for anomaly detection because it does not rely on the labels of normal data, can automatically learn the normal patterns of data and identify anomalies that deviate from these patterns. The isolation forest model has good robustness to noise and outliers because it evaluates anomalies based on the degree of isolation of data points rather than directly relying on the statistical distribution of data. This enables the model to maintain a high detection accuracy in a complex and changeable network environment. The entire detection process is highly automated, and no human intervention is required from data collection, processing to anomaly detection. This greatly reduces labor costs and error rates, and improves the stability and reliability of the system.

[0014] Optionally, if it is determined that a network abnormality occurs in the first server, then the deviation data is determined according to the first network data and the second network data, which specifically also includes: determining a benchmark value according to the first network data and the second network data, the benchmark value being the average value between the first network data and the second network data; calculating the difference between the first network data and the benchmark value to obtain a first difference; calculating the difference between the second network data and the benchmark value to obtain a second difference; based on the first difference, the second difference and the benchmark value, calculating the relative deviation of the first network data corresponding to the benchmark value, and the relative deviation of the second network data corresponding to the benchmark value; determining the deviation data according to the relative deviation of the first network data corresponding to the benchmark value, and the relative deviation of the second network data corresponding to the benchmark value.

[0015] By adopting the above technical solution, the difference between the first network data and the second network data and the benchmark value, as well as the ratio of these differences to the benchmark value, can be quantitatively expressed. This quantification method makes the deviation data more intuitive, easy to understand and analyze. Using the benchmark value as a reference point can eliminate the direct comparison error caused by different data scales, units or formats. By calculating the relative deviation, the actual difference between the data can be more accurately reflected, especially when dealing with large-scale or high dynamic range data. The relative deviation not only indicates the size of the difference between the data, but also reveals the ratio of this difference to the overall data level. This makes the deviation data more interpretable in subsequent analysis, reporting or decision-making, and helps relevant personnel better understand the meaning behind the data. By comparing the relative deviation of the first network data and the second network data, it is easier to identify abnormal or inconsistent data points. This is of great significance for timely detection of network anomalies, data errors or potential security threats. After determining the deviation data, these data can be used to guide data recovery or verification work.

[0016] Optionally, before inputting the real-time network traffic data into the preset isolation forest model to obtain the abnormal score corresponding to each data point, the preset isolation forest model is trained; the training of the preset isolation forest model specifically includes: obtaining a training data set, the training data set including normal samples and abnormal samples; setting isolation forest training parameters, the training parameters including the number of isolated trees, the number of samples extracted during training of each tree, and the number of features extracted during training of each tree; randomly extracting samples from the training data set to obtain target samples; randomly selecting features from the target samples to obtain target features; randomly selecting a cutting node between the maximum value and the minimum value of the target feature, and dividing the target sample into two parts and placing them into the left and right child nodes of the cutting node respectively; recursively repeating on the left and right child nodes until a preset stop condition is reached to obtain a target isolated tree; and determining the preset isolation forest model based on multiple target isolated trees.

[0017] By adopting the above technical solution, by obtaining a training data set containing normal samples and abnormal samples, the model can learn the normal mode and abnormal mode of network traffic. This learning method enables the model to more accurately identify abnormal data points in the subsequent detection process, thereby improving the accuracy of detection. During the training process, the diversity and randomness of the model are increased by randomly extracting samples and features, and randomly selecting cutting nodes. This randomness helps the model learn a wider range of network traffic features, thereby enhancing its generalization ability in new environments or new data. The training parameters of the isolation forest model can be adjusted according to actual needs. This flexibility enables the model to adapt to network traffic data of different scales and complexities, improving the adaptability and practicality of the model. The isolation forest model evaluates the abnormality of data points based on the degree of isolation of the data points. This mechanism is particularly efficient when processing large-scale data. Because abnormal data points are usually easier to be isolated than normal data points, the model can quickly identify abnormal data points and reduce computing time and resource consumption.

[0018] Optionally, a backup request sent by a user device is received, wherein the backup request includes a fourth server; and according to the backup request, the deviation data is sent to the fourth server for backup.

[0019] By adopting the above technical solution, redundant storage of data is achieved by backing up the deviation data to the fourth server. In this way, even if the original data is lost or damaged for various reasons, it can be quickly restored through the backup data, thereby enhancing the security of the data. Users can choose which server to back up the deviation data to according to their needs. This flexibility makes data management more convenient, and the best choice can be made according to different business needs, security policies or cost considerations. Backing up data to multiple servers can reduce the risk of single point failure. Even if a server fails, the backup data on other servers is still available, thereby ensuring the continuity and availability of data.

[0020] In a second aspect of the present application, a network data control device based on big data is provided, the network data control device comprising an acquisition module and a processing module, wherein the acquisition module is used to acquire first network data stored in a first server; the acquisition module is also used to acquire second network data stored in a second server, the second server is a backup server corresponding to the first server, and the second network data is backup data of the first network data; the processing module is also used to determine deviation data based on the first network data and the second network data if it is determined that a network anomaly occurs in the first server, the deviation data being different data generated by the first server compared to the second server within a preset time period, and the preset time period being the time period when the network anomaly occurs in the first server; the processing module is also used to send the deviation data to the second server and a third server for backup, the third server being any one of a plurality of servers except the first server and the second server.

[0021] In the third aspect of the present application, an electronic device is provided, which includes a processor, a memory, a user interface and a network interface, the memory is used to store instructions, the user interface and the network interface are both used to communicate with other devices, and the processor is used to execute the instructions stored in the memory so that the electronic device performs the method described above.

[0022] In a fourth aspect of the present application, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores instructions, and when the instructions are executed, the method described above is executed.

[0023] In summary, one or more technical solutions provided in the embodiments of the present application have at least the following technical effects or advantages: By maintaining a copy of the first network data on the second server, critical business data can be protected even if the first server has a network anomaly or data loss. In addition, when a network anomaly is detected and it is determined that there is deviated data, the deviated data is promptly sent to another independent third server for additional backup, further enhancing the security and redundancy of the data. After a network anomaly occurs, by comparing the data differences between the first server and the second server within a preset time period, it is possible to quickly locate and determine which data is the affected deviated data. This precise data recovery mechanism reduces unnecessary data transmission and processing time and improves the efficiency of data recovery. By monitoring the data differences between the first server and the second server and processing the deviated data immediately when a network anomaly is detected, this strategy supports real-time or near real-time protection of data. By sending the deviated data to the third server for backup, a multi-level backup system is constructed. This design enables the system to recover data from the third server even if the main server and the backup server encounter problems at the same time, thereby greatly enhancing the resilience and disaster resistance of the entire system and facilitating the security of network data. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 A flowchart of a network data control method based on big data provided in an embodiment of the present application.

[0025] Figure 2 Another flowchart of a network data control method based on big data provided in an embodiment of the present application.

[0026] Figure 3 A module schematic diagram of a network data control device based on big data provided in an embodiment of the present application.

[0027] Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0028] Explanation of the reference numerals: 31, acquisition module; 32, processing module; 41, processor; 42, communication bus; 43, user interface; 44, network interface; 45, memory. DETAILED DESCRIPTION

[0029] In order to enable technicians in this field to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below in conjunction with the drawings in the embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this application, not all of the embodiments.

[0030] In the description of the embodiments of the present application, words such as "for example" or "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "for example" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "for example" or "for example" is intended to present related concepts in a specific way.

[0031] In the description of the embodiments of the present application, the meaning of the term "multiple" refers to two or more. For example, multiple systems refer to two or more systems, and multiple screen terminals refer to two or more screen terminals. In addition, the terms "first" and "second" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance or implicitly indicating the indicated technical features. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. The terms "include", "comprise", "have" and their variations all mean "including but not limited to", unless otherwise specifically emphasized.

[0032] In today's era of surging digitalization, the rapid development of information technology is reshaping all walks of life at an unprecedented speed. Among them, big data, as the core driving force, is leading a new climax of innovation and development. However, with the explosive growth of data volume, the problem of data security protection has become increasingly severe, especially in the field of cyberspace, which is full of unknowns and challenges. The traditional data backup model is facing unprecedented tests.

[0033] At present, the cornerstone of traditional data backup strategies is to mirror the core business data of enterprises to independent servers or dedicated storage media through preset time intervals or instant response mechanisms, so as to build a line of defense against data loss or system failure. However, in the face of increasingly complex and changing network attacks, this single backup method is increasingly stretched and difficult to fully guarantee the security of data.

[0034] In order to solve the above technical problems, the present application provides a network data control method based on big data, referring to Figure 1 , Figure 1 A flowchart of a network data control method based on big data provided in an embodiment of the present application. The network data control method is applied to a central server and includes steps S110 to S140, which are as follows: S110: Acquire first network data stored in a first server.

[0035] Specifically, the central server is a core system component responsible for managing and coordinating the data flow, task allocation, resource scheduling, etc. of multiple servers in the entire system. The central server has high-performance hardware and complex software systems to support large-scale data processing and high-concurrency requests. The first server is another server node in the system, responsible for storing and processing specific data or tasks. In an embodiment of the present application, the first server stores the first network data, which includes records of network traffic, log information, user behavior data, etc., which are of great significance for system monitoring, analysis or decision-making. The central server will actively read or download the first network data from the first server, which involves the use of network communication protocols (such as HTTP, FTP, etc.), as well as data encryption, compression and transmission optimization technologies to ensure the security, integrity and efficient transmission of data.

[0036] In a possible implementation, obtaining the first network data stored in the first server specifically includes: receiving original network data sent by the first server; preprocessing the original network data to obtain the first network data, wherein the preprocessing includes denoising, filtering, data feature extraction, and normalization.

[0037] Specifically, the central server receives the original network data from the first server through a communication protocol (such as HTTP, FTP, TCP / IP, etc.). The original data contains a large amount of unprocessed, noisy and redundant information. Due to various factors such as network transmission, equipment failure, environmental interference, etc., the original data contains noise, that is, random fluctuations that are irrelevant to the real signal or affect the accuracy of the signal. The denoising step aims to eliminate or reduce these noises to improve the signal-to-noise ratio of the data. Filtering is a commonly used technique in signal processing. It removes unnecessary frequency components or enhances required frequency components by performing specific frequency processing on the signal. In network data, filtering is used to smooth data, remove high-frequency noise, or extract information in a specific frequency band. The original data contains a lot of detailed information, but not all information is useful for subsequent analysis. The data feature extraction step aims to extract key features useful for the analysis target from the original data, such as statistics (mean, variance, standard deviation, etc.), trends, periodicity, outliers, etc. Due to the different dimensions and ranges of the original data, direct comparison or analysis is not appropriate. Normalization is a method of scaling data so that it falls into a small specific interval, such as [0, 1] or [-1, 1]. This can eliminate the impact of different dimensions on data analysis and make the results more accurate and reliable.

[0038] For example, suppose that data from a network traffic monitoring center server is being processed. The center server collects real-time traffic data from each network interface as raw network data. The center server collects traffic data from each network interface once a minute and aggregates the data and sends it to the center server. The center server receives a series of raw data records containing timestamps, interface names, and traffic values. In the raw data, abnormally high or low traffic values ​​due to temporary equipment failures or network congestion are found. These abnormal values ​​can be regarded as noise and can be removed by setting reasonable thresholds or using statistical methods (such as median filtering). To further smooth the data, a low-pass filter can be used to remove high-frequency noise. For example, a moving average filter or an exponential smoothing filter can be used to process traffic data to reduce the impact of short-term fluctuations on the overall trend. Key features are extracted from the processed data, such as the average traffic, maximum traffic, minimum traffic, and traffic change rate for each time period. These features can more intuitively reflect the status of network traffic. Since the traffic value ranges of different interfaces vary greatly, in order to facilitate comparison and analysis, the traffic values ​​of all interfaces can be normalized to the same range (such as [0, 1]). In this way, even if the absolute flow values ​​of different interfaces are different, their relative changes can be compared intuitively.

[0039] S120: Acquire second network data stored in a second server, where the second server is a backup server corresponding to the first server, and the second network data is backup data of the first network data.

[0040] Specifically, suppose there is an online shopping platform whose first server is responsible for processing user orders, storing product information and user data, etc. In order to ensure data security and business continuity, the platform configures a backup server, i.e., the second server, for the main server, and regularly synchronizes the data on the main server to the backup server. One day, the main server suddenly crashed due to a hardware failure, making it impossible to access user data and order information. In order to restore the service as soon as possible, the platform's operation and maintenance team immediately enabled the backup server. The central server established a connection with the backup server and obtained the secondary network data on the backup server. Subsequently, this data was used to restore the main server or temporarily replace the main server to ensure that users can continue to access the shopping platform and process orders.

[0041] In one possible implementation, obtaining second network data stored in a second server specifically includes: determining a first storage identifier based on the first server and the first network data; determining the second server based on the first storage identifier; searching for the first storage identifier in the second server, and if it is determined that the second server has a second storage identifier corresponding to the first storage identifier, obtaining backup data corresponding to the second storage identifier to obtain the second network data, and the first storage identifier and the second storage identifier are the same storage identifier.

[0042] Specifically, the central server needs to determine a unique storage identifier based on certain information of the first server and / or the first network data (such as timestamp, data type, file name, etc.), which is called the first storage identifier. This identifier is used to uniquely identify a certain data set or data file on the first server. With the first storage identifier, the central server can determine which server is responsible for storing the backup of the data set based on this identifier. In this scenario, this server is the second server, which is the backup server of the first server. The central server will then search for this first storage identifier in the second server. This step is to confirm whether the backup data corresponding to the first server is actually stored in the second server. If a second storage identifier matching the first storage identifier is found in the second server, it means that the required backup data is actually stored on the second server. The second storage identifier here is a unique identifier used to identify the backup data on the second server. It is a storage identifier with a similarity that meets a certain threshold with the first storage identifier, but they belong to different servers. Finally, the central server will obtain the corresponding backup data based on the second storage identifier, which is the second network data we are looking for. These data are the same or very close to the first network data because they are backups of the same data set.

[0043] S130. If it is determined that a network anomaly occurs in the first server, deviation data is determined based on the first network data and the second network data. The deviation data is different data generated by the first server compared with the second server within a preset time period. The preset time period is the time period when the network anomaly occurs in the first server.

[0044] Specifically, the central server has detected that the first server has a network anomaly, such as inaccessibility, slow response, or data loss. In this step, the central server compares two data sources: the data of the first server before the anomaly occurs (first network data) and the backup data on the second server (second network data). The purpose of the comparison is to find the difference between the two, that is, the deviation data. Deviation data does not refer to all different data, but specifically refers to different data generated within a preset time period. This preset time period refers to the time period from the occurrence of the network anomaly on the first server to the detection or resolution of the anomaly. It is a time period directly related to the network anomaly of the first server.

[0045] In one possible implementation, if it is determined that a network anomaly occurs in the first server, deviation data is determined based on the first network data and the second network data, specifically including: receiving real-time network traffic data sent by a first network probe and a second network probe, the first network probe being used to monitor the data entry of the first server, and the second network probe being used to monitor the data exit of the second server; inputting the real-time network traffic data into a preset isolation forest model to obtain an anomaly score corresponding to each data point, the real-time network traffic data including multiple data points; if it is determined that the anomaly score corresponding to the target data point is less than a preset threshold, it is determined that a network anomaly occurs in the target data point, then it is determined that a network anomaly occurs in the first server, the target data point is any one of the multiple data points, and the anomaly score corresponding to the target data point is determined based on the average traversal path length of the target data point in the preset isolation forest model.

[0046] Specifically, the first network probe is installed at the data entry of the first server to monitor the real-time network traffic data entering the first server. These data reflect all network requests and data transmission received by the first server. The second network probe is installed at the data exit of the second server (backup server) to monitor the real-time network traffic data sent from the second server. These data include backup data transmission, synchronization requests, etc. The two network probes collect the real-time network traffic data of their respective servers and send these data to the central server. These data contain multiple data points, each of which represents a network request, data packet or traffic statistics.

[0047] The central server inputs the collected real-time network traffic data into the preset isolation forest model. Isolation forest is a machine learning algorithm for anomaly detection that isolates data points by constructing multiple isolation trees (iTrees). For each data point, the isolation forest model calculates its average traversal path length, that is, the average path length from the root node to the leaf node where the data point is located. The shorter the path, the easier it is for the data point to be isolated, that is, the more abnormal it is. The model assigns an anomaly score to each data point, which is determined based on the average traversal path length of the data point in the isolation forest. The shorter the average path length, the lower the anomaly score, indicating that the data point is more abnormal.

[0048] Once it is determined that the first server has a network anomaly, the central server will further compare the data on the first server (the partial period of time when the server can work normally before or during the anomaly) and the data on the second server. These data include user data, transaction records, system logs, etc. By comparing these data, the central server can determine the difference between the two during the preset time period (i.e., the time period when the first server has a network anomaly), i.e., the deviation data. These deviation data represent the data that is lost, erroneous, or unprocessed due to the network anomaly.

[0049] In a possible implementation, if it is determined that a network abnormality occurs in the first server, deviation data is determined based on the first network data and the second network data, and specifically also includes: determining a benchmark value based on the first network data and the second network data, the benchmark value being the average value between the first network data and the second network data; calculating the difference between the first network data and the benchmark value to obtain a first difference; calculating the difference between the second network data and the benchmark value to obtain a second difference; based on the first difference, the second difference and the benchmark value, calculating the relative deviation of the first network data corresponding to the benchmark value, and the relative deviation of the second network data corresponding to the benchmark value; determining the deviation data based on the relative deviation of the first network data corresponding to the benchmark value, and the relative deviation of the second network data corresponding to the benchmark value.

[0050] Specifically, the benchmark value is a reference point between the first network data and the second network data, which is used for subsequent calculation of relative deviation. In an embodiment of the present application, the benchmark value is defined as the average value between the two data sets. This means that the benchmark value attempts to reflect the common characteristics of the two data sets in a "normal" or "average" state. The first difference is the difference between the first network data and the benchmark value. This difference indicates the degree of deviation of the first network data from the benchmark value. If the first network data is higher than the benchmark value, the difference is positive; if it is lower than the benchmark value, the difference is negative. Similar to the first difference, the second difference is the difference between the second network data and the benchmark value. This difference indicates the degree of deviation of the second network data from the benchmark value.

[0051] Among them, the relative deviation is the ratio of the difference value to the reference value, which is used to measure the degree of deviation of the data point from the reference value. Here, the relative deviation of the first network data and the second network data corresponding to the reference value is calculated respectively. The relative deviation is a dimensionless quantity, which allows data of different magnitudes to be compared with each other. Finally, based on the relative deviation corresponding to the first network data and the second network data, the deviation data can be determined. This involves identifying which data points (or data subsets) have a relative deviation exceeding a certain preset threshold, or which data points have the largest relative deviation. Deviation data represents significant differences due to network anomalies, data errors or inconsistencies, etc.

[0052] For example, suppose there is a monitoring system that monitors the CPU usage of two servers (as a representative of network data). The first server is the primary server and the second server is the backup server. During a certain period of time, the central server collects the CPU usage of the first server (the first network data) and the CPU usage of the second server (the second network data). Assume that the CPU usage of the first server is 70% and the CPU usage of the second server is 60%. The benchmark value is the average value between the two, that is, (70%+60%) / 2=65%. The first difference is 70%-65%=5%, indicating that the CPU usage of the first server is 5% higher than the benchmark value. The second difference is 60%-65%=-5%, indicating that the CPU usage of the second server is 5% lower than the benchmark value. The relative deviation corresponding to the first network data is 5% / 65%≈7.7%. The relative deviation corresponding to the second network data is -5% / 65%≈-7.7%.

[0053] In this example, although the CPU utilization of the two servers is different, their relative deviation is within the acceptable range (assuming that the preset threshold is greater than 7.7%). However, if the CPU utilization of the first server suddenly soars to 90%, while the second server remains unchanged, the relative deviation of the first server will increase significantly and exceed the preset threshold. At this time, we can consider that the CPU utilization of the first server is abnormal and treat it as deviation data. Deviation data indicates that the first server is experiencing overload, attack, or other performance problems.

[0054] S140: Send the deviation data to a second server and a third server for backup, where the third server is any server among the multiple servers except the first server and the second server.

[0055] Specifically, the deviation data is first sent to the second server for backup. The second server exists as a backup or redundant server, and it has a certain data synchronization or backup mechanism with the first server. Sending the deviation data to the second server can ensure that the data can still be accessed and restored when the first server fails or data is lost. In addition to the second server, the deviation data is also sent to the third server for backup. The third server is any server other than the first and second servers among multiple servers, which means that the system contains multiple additional servers to further enhance the reliability and fault tolerance of the data. Sending the deviation data to the third server (or more servers) can further disperse the data risk and ensure the persistence and availability of the data. The purpose of backup is to be able to recover the data when the original data is lost, damaged or unavailable. By storing multiple copies of data on different servers, the system can improve its resistance to data loss or damage. In addition, backup can also help quickly restore services in the event of system failure and reduce downtime.

[0056] Therefore, by maintaining a copy of the first network data on the second server, even if the first server has a network anomaly or data loss, the critical business data can be protected. In addition, when a network anomaly is detected and it is determined that there is deviated data, the deviated data is promptly sent to another independent third server for additional backup, further enhancing the security and redundancy of the data. After the network anomaly occurs, by comparing the data differences between the first server and the second server within a preset time period, it is possible to quickly locate and determine which data is the affected deviated data. This precise data recovery mechanism reduces unnecessary data transmission and processing time and improves the efficiency of data recovery. By monitoring the data differences between the first server and the second server and processing the deviated data immediately when a network anomaly is detected, this strategy supports real-time or near real-time protection of data. By sending the deviated data to the third server for backup, a multi-level backup system is constructed. This design enables the system to recover data from the third server even if the main server and the backup server encounter problems at the same time, thereby greatly enhancing the resilience and disaster resistance of the entire system and facilitating the security of network data.

[0057] In a possible implementation, before inputting real-time network traffic data into a preset isolation forest model to obtain an abnormal score corresponding to each data point, the preset isolation forest model is trained; the training of the preset isolation forest model specifically includes: obtaining a training data set, the training data set includes normal samples and abnormal samples; setting training parameters of the isolation forest, the training parameters include the number of isolated trees, the number of samples extracted during training of each tree, and the number of features extracted during training of each tree; randomly extracting samples from the training data set to obtain target samples; randomly selecting features from the target samples to obtain target features; randomly selecting a cutting node between the maximum value and the minimum value of the target feature, and dividing the target sample into two parts and placing them into the left and right child nodes of the cutting node respectively; recursively repeating on the left and right child nodes until a preset stop condition is reached to obtain a target isolated tree; and determining the preset isolation forest model based on multiple target isolated trees.

[0058] Specifically, the training dataset is the basis for training the isolation forest model, and it should contain a sufficient number of normal and abnormal samples. These samples should be able to represent various situations that occur in the actual data so that the model can learn the characteristics of normal and abnormal behaviors. The training parameters of the isolation forest include the number of isolated trees, the number of samples extracted during training for each tree, and the number of features extracted during training for each tree. The selection of these parameters will affect the performance and efficiency of the model. For example, more isolated trees can improve the accuracy of the model, but will also increase the computational cost; while the number of samples and features extracted from each tree needs to balance coverage and computational complexity. During the training process, the algorithm randomly extracts a certain number of samples from the training dataset as target samples, and randomly selects a certain number of features from these samples as target features. This randomness helps to increase the generalization ability of the model and prevent overfitting. For each target feature, the algorithm randomly selects a cutting node between its maximum and minimum values, and divides the target sample into two parts according to this node, and puts them into the left and right child nodes of the cutting node respectively. The algorithm then recursively repeats this process on the left and right child nodes until a preset stopping condition is reached, such as the number of samples in the child node reaches a certain threshold, or the depth of the tree reaches a preset maximum value. This process generates an isolation tree in which each data point follows a path from the root node to a leaf node.

[0059] By repeating the above process multiple times (according to the preset number of isolated trees), multiple isolated trees can be obtained. These isolated trees together constitute the preset isolation forest model. In the detection phase, when a new data point is input into this model, it will be processed by each tree separately and a corresponding path length, i.e., anomaly score, will be obtained. Finally, the anomaly scores of these data points will be summarized to determine whether they are abnormal.

[0060] In one possible implementation, refer to Figure 2 , Figure 2 Another flow chart of a network data control method based on big data provided for an embodiment of the present application includes steps S210 to S220, and the above steps are as follows: S210, receiving a backup request sent by a user device, the backup request includes a fourth server; S220, according to the backup request, sending the deviation data to the fourth server for backup.

[0061] Specifically, a user device (such as a personal computer, mobile phone, tablet computer, etc.) sends a backup request to the central server through a communication method (such as a network request). This request contains the specific operation that the user wants to perform, including the type of data to be backed up and the target location. In this particular backup request, the user explicitly specifies that the data should be backed up to the fourth server. This means that the central server needs to know the address, permission information, and any necessary configuration of the fourth server in order to successfully send the data to the server. After receiving the backup request, the central server will parse the information in the request to determine the data to be backed up (here, the deviation data) and the target server (the fourth server). The central server will then retrieve the deviation data from its storage location (which is a local hard disk, database, or other storage medium) and send this data to the fourth server through an appropriate network protocol (such as HTTP, FTP, etc.). During the transmission process, the central server will encrypt the data to protect its security, and verify the data after it successfully reaches the fourth server to ensure the integrity and accuracy of the data.

[0062] The present application also provides a network data control device based on big data, referring to Figure 3 , Figure 3 A module schematic diagram of a network data control device based on big data provided in an embodiment of the present application. The network data control device is a central server, and the central server includes an acquisition module 31 and a processing module 32, wherein the acquisition module 31 acquires the first network data stored in the first server; the acquisition module 31 acquires the second network data stored in the second server, the second server is a backup server corresponding to the first server, and the second network data is the backup data of the first network data; if the processing module 32 determines that a network anomaly occurs in the first server, then the deviation data is determined according to the first network data and the second network data, and the deviation data is the different data generated by the first server compared with the second server in a preset time period, and the preset time period is the time period when the network anomaly occurs in the first server; the processing module 32 sends the deviation data to the second server and the third server for backup, and the third server is any one of the multiple servers except the first server and the second server.

[0063] In a possible implementation, the acquisition module 31 acquires the first network data stored in the first server, specifically including: the acquisition module 31 receives the original network data sent by the first server; the processing module 32 preprocesses the original network data to obtain the first network data, and the preprocessing includes denoising, filtering, data feature extraction and normalization.

[0064] In a possible implementation, the acquisition module 31 acquires the second network data stored in the second server, specifically including: the processing module 32 determines the first storage identifier according to the first server and the first network data; the processing module 32 determines the second server based on the first storage identifier; the processing module 32 searches for the first storage identifier in the second server, and if it is determined that the second server has a second storage identifier corresponding to the first storage identifier, then the backup data corresponding to the second storage identifier is acquired to obtain the second network data, and the first storage identifier and the second storage identifier are the same storage identifier.

[0065] In one possible implementation, if the processing module 32 determines that a network anomaly occurs in the first server, the deviation data is determined based on the first network data and the second network data, specifically including: the acquisition module 31 receives real-time network traffic data sent by the first network probe and the second network probe, the first network probe is used to monitor the data entry of the first server, and the second network probe is used to monitor the data exit of the second server; the processing module 32 inputs the real-time network traffic data into a preset isolation forest model to obtain an anomaly score corresponding to each data point, and the real-time network traffic data includes multiple data points; if the processing module 32 determines that the anomaly score corresponding to the target data point is less than a preset threshold, it is determined that a network anomaly occurs in the target data point, and then it is determined that a network anomaly occurs in the first server, the target data point is any one of the multiple data points, and the anomaly score corresponding to the target data point is determined based on the average traversal path length of the target data point in the preset isolation forest model.

[0066] In a possible implementation, if the processing module 32 determines that a network abnormality occurs in the first server, the deviation data is determined based on the first network data and the second network data, and specifically includes: the processing module 32 determines a benchmark value based on the first network data and the second network data, and the benchmark value is the average value between the first network data and the second network data; the processing module 32 calculates the difference between the first network data and the benchmark value to obtain a first difference; the processing module 32 calculates the difference between the second network data and the benchmark value to obtain a second difference; the processing module 32 calculates the relative deviation of the first network data corresponding to the benchmark value, and the relative deviation of the second network data corresponding to the benchmark value based on the first difference, the second difference and the benchmark value; the processing module 32 determines the deviation data based on the relative deviation of the first network data corresponding to the benchmark value, and the relative deviation of the second network data corresponding to the benchmark value.

[0067] In a possible implementation, the processing module 32 inputs the real-time network traffic data into the preset isolation forest model to obtain the abnormal score corresponding to each data point, and then trains the preset isolation forest model; the training of the preset isolation forest model specifically includes: the acquisition module 31 acquires the training data set, and the training data set includes normal samples and abnormal samples; the processing module 32 sets the training parameters of the isolation forest, and the training parameters include the number of isolated trees, the number of samples extracted during the training of each tree, and the number of features extracted during the training of each tree; the processing module 32 randomly extracts samples from the training data set to obtain target samples; the processing module 32 randomly selects features from the target samples to obtain target features; the processing module 32 randomly selects a cutting node between the maximum value and the minimum value of the target feature, and divides the target sample into two parts and respectively puts them into the left and right child nodes of the cutting node; the processing module 32 recursively repeats on the left and right child nodes until the preset stop condition is reached to obtain the target isolated tree; the processing module 32 determines the preset isolation forest model based on multiple target isolated trees.

[0068] In a possible implementation, the acquisition module 31 receives a backup request sent by the user device, where the backup request includes a fourth server; and the processing module 32 sends the deviation data to the fourth server for backup according to the backup request.

[0069] It should be noted that: when the device provided in the above embodiment realizes its function, only the division of the above functional modules is used as an example. In actual application, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the device and method embodiments provided in the above embodiment belong to the same concept, and the specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0070] The present application also provides an electronic device, referring to Figure 4 , Figure 4 The electronic device may include: at least one processor 41 , at least one network interface 44 , a user interface 43 , a memory 45 , and at least one communication bus 42 .

[0071] The communication bus 42 is used to realize the connection and communication between these components.

[0072] The user interface 43 may include a display screen (Display) and a camera (Camera). Optionally, the user interface 43 may also include a standard wired interface and a wireless interface.

[0073] The network interface 44 may optionally include a standard wired interface or a wireless interface (such as a WI-FI interface).

[0074] Among them, the processor 41 may include one or more processing cores. The processor 41 uses various interfaces and lines to connect various parts in the entire server, and executes various functions of the server and processes data by running or executing instructions, programs, code sets or instruction sets stored in the memory 45, and calling data stored in the memory 45. Optionally, the processor 41 can be implemented in at least one hardware form of digital signal processing (Digital Signal Processing, DSP), field programmable gate array (Field-Programmable Gate Array, FPGA), and programmable logic array (Programmable Logic Array, PLA). The processor 41 can integrate one or a combination of a central processing unit (Central Processing Unit, CPU), a graphics processing unit (Graphics Processing Unit, GPU) and a modem. Among them, the CPU mainly processes the operating system, user interface and application programs; the GPU is responsible for rendering and drawing the content to be displayed on the display screen; the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor 41, and it can be implemented separately through a chip.

[0075] Among them, the memory 45 may include a random access memory (RAM) or a read-only memory (Read-Only Memory). Optionally, the memory 45 includes a non-transitory computer-readable storage medium. The memory 45 can be used to store instructions, programs, codes, code sets or instruction sets. The memory 45 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the above-mentioned method embodiments, etc.; the data storage area may store data involved in the above-mentioned method embodiments, etc. The memory 45 may also be optionally at least one storage device located away from the aforementioned processor 41. As Figure 4 As shown, the memory 45 as a computer storage medium may include an operating system, a network communication module, a user interface module, and an application program of a network data control method based on big data.

[0076] exist Figure 4In the electronic device shown, the user interface 43 is mainly used to provide an input interface for the user and obtain data input by the user; and the processor 41 can be used to call an application program stored in the memory 45 for a network data control method based on big data. When executed by one or more processors, the electronic device executes one or more methods in the above-mentioned embodiments.

[0077] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the order of the actions described, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required for the present application.

[0078] The present application also provides a computer-readable storage medium, which stores instructions. When executed by one or more processors, the electronic device executes one or more of the methods described in the above embodiments.

[0079] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0080] In the several embodiments provided in this application, it should be understood that the disclosed devices can be implemented in other ways. For example, the device embodiments described above are only schematic, such as the division of units, which is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some service interfaces, and the indirect coupling or communication connection of devices or units can be electrical or other forms.

[0081] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0082] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0083] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a memory and includes several instructions for a computer device (which can be a personal computer, server or network device, etc.) to execute all or part of the steps of the various embodiments of the present application. The aforementioned memory includes: various media that can store program codes, such as USB flash drives, mobile hard drives, magnetic disks or optical disks.

[0084] The above is only an exemplary embodiment of the present disclosure, and the scope of the present disclosure cannot be limited thereto. That is, any equivalent changes and modifications made according to the teachings of the present disclosure are still within the scope of the present disclosure. After considering the disclosure of the specification and the truth of practice, it will be easy for those skilled in the art to think of other embodiments of the present disclosure. This application is intended to cover any modification, use or adaptation of the present disclosure, which follows the general principles of the present disclosure and includes common knowledge or customary technical means in the technical field that are not recorded in the present disclosure. The description and examples are regarded as exemplary only, and the scope and spirit of the present disclosure are defined by the claims.

Claims

1. A network data control method based on big data, characterized in that: The method comprises: Acquire first network data stored in the first server; Acquire second network data stored in a second server, where the second server is a backup server corresponding to the first server, and the second network data is backup data of the first network data; If it is determined that a network anomaly occurs on the first server, then determining deviation data according to the first network data and the second network data, the deviation data being different data generated by the first server compared with the second server within a preset time period, and the preset time period being a time period in which the network anomaly occurs on the first server; The deviation data is sent to the second server and a third server for backup, where the third server is any one of the multiple servers except the first server and the second server.

2. The network data control method based on big data according to claim 1 is characterized in that: The obtaining of the first network data stored in the first server specifically includes: Receiving original network data sent by the first server; The original network data is preprocessed to obtain the first network data, wherein the preprocessing includes denoising, filtering, data feature extraction and normalization processing.

3. The network data control method based on big data according to claim 1 is characterized in that: The obtaining of the second network data stored in the second server specifically includes: Determine a first storage identifier according to the first server and the first network data; Determining the second server based on the first storage identifier; Search for the first storage identifier in the second server. If it is determined that the second server has a second storage identifier corresponding to the first storage identifier, obtain the backup data corresponding to the second storage identifier to obtain the second network data. The first storage identifier and the second storage identifier are the same storage identifier.

4. The network data control method based on big data according to claim 1 is characterized in that: If it is determined that a network anomaly occurs on the first server, determining deviation data according to the first network data and the second network data specifically includes: receiving real-time network traffic data sent by a first network probe and a second network probe, wherein the first network probe is used to monitor a data inlet of the first server, and the second network probe is used to monitor a data outlet of the second server; Inputting the real-time network traffic data into a preset isolation forest model to obtain an anomaly score corresponding to each data point, wherein the real-time network traffic data includes multiple data points; If it is determined that the anomaly score corresponding to the target data point is less than a preset threshold, it is determined that a network anomaly occurs at the target data point, and it is determined that a network anomaly occurs at the first server. The target data point is any one of the multiple data points, and the anomaly score corresponding to the target data point is determined based on the average traversal path length of the target data point in the preset isolation forest model.

5. The network data control method based on big data according to claim 4 is characterized in that: If it is determined that a network anomaly occurs on the first server, determining deviation data according to the first network data and the second network data specifically includes: Determine a reference value according to the first network data and the second network data, wherein the reference value is an average value between the first network data and the second network data; Calculating a difference between the first network data and the reference value to obtain a first difference; Calculating a difference between the second network data and the reference value to obtain a second difference; Based on the first difference, the second difference and the reference value, calculate a relative deviation of the first network data corresponding to the reference value and a relative deviation of the second network data corresponding to the reference value; The deviation data is determined according to a relative deviation of the first network data corresponding to the reference value and a relative deviation of the second network data corresponding to the reference value.

6. The network data control method based on big data according to claim 4 is characterized in that: Before inputting the real-time network traffic data into the preset isolation forest model to obtain an abnormality score corresponding to each data point, training the preset isolation forest model; The training of the preset isolation forest model specifically includes: Acquire a training data set, where the training data set includes normal samples and abnormal samples; Setting the training parameters of the isolation forest, wherein the training parameters include the number of isolated trees, the number of samples extracted during the training of each tree, and the number of features extracted during the training of each tree; Randomly extracting samples from the training data set to obtain target samples; Randomly select features from the target sample to obtain target features; A cutting node is randomly selected between the maximum value and the minimum value of the target feature, and the target sample is divided into two parts and respectively placed in the left and right child nodes of the cutting node; Recursively repeat on the left and right child nodes until the preset stop condition is reached to obtain the target isolated tree; The preset isolation forest model is determined according to the multiple target isolated trees.

7. The network data control method based on big data according to claim 1 is characterized in that: The method further comprises: receiving a backup request sent by a user device, wherein the backup request includes a fourth server; According to the backup request, the deviation data is sent to the fourth server for backup.

8. A network data control device based on big data, characterized in that: The network data control device comprises an acquisition module (31) and a processing module (32), wherein: The acquisition module (31) is used to acquire the first network data stored in the first server; The acquisition module (31) is further used to acquire second network data stored in a second server, the second server being a backup server corresponding to the first server, and the second network data being backup data of the first network data; The processing module (32) is further configured to determine deviation data based on the first network data and the second network data if it is determined that a network anomaly occurs on the first server, wherein the deviation data is different data generated by the first server compared with the second server within a preset time period, and the preset time period is a time period in which the network anomaly occurs on the first server; The processing module (32) is further used to send the deviation data to the second server and the third server for backup, and the third server is any one of the multiple servers except the first server and the second server.

9. An electronic device, characterized in that: The electronic device comprises a processor (41), a memory (45), a user interface (43) and a network interface (44), wherein the memory (45) is used to store instructions, the user interface (43) and the network interface (44) are both used to communicate with other devices, and the processor (41) is used to execute the instructions stored in the memory (45) so that the electronic device executes the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores instructions, and when the instructions are executed, the method according to any one of claims 1 to 7 is performed.

Images