Monitoring data storage method and system based on time series database

By collecting, preprocessing and sharding monitoring data in a time series database, and building Meta and Data nodes in the server cluster, combining exception detection and encryption protocol matching, the single storage point threat of monitoring data storage is solved, and data efficiency is achieved , secure and reliable storage.

CN118838900BActive Publication Date: 2024-12-03JIANGSU HUAKU DATA TECH CO LTD +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411313855.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-09-20
Publication Date
2024-12-03
Estimated Expiration
2044-09-20

AI Technical Summary

Technical Problem

The metric data storage of the monitoring system faces the threat of a single storage point, and it is difficult to ensure the efficiency, security and reliability of the data.

Method used

Using the monitoring data storage method based on the timing database, by collecting and preprocessing the monitoring data, building Meta nodes and Data nodes in the server cluster, performing data sharding and abnormal detection, and matching the data transmission encryption protocol based on the abnormal detection results, and Storage data in the storage cluster.

Benefits of technology

Improve the confidentiality and integrity of data during transmission, reduce the risk of data stolen or tampered, realize the scalability and reliability of the storage system, timely identify and process abnormal data, improve data processing speed and system performance .

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118838900B_ABST
    Figure CN118838900B_ABST
Patent Text Reader

Abstract

The present invention relates to the field of data processing technology, and discloses a monitoring data storage method and system based on a time series database. The method includes: collecting the monitoring data to be stored in the time series database and preprocessing it to obtain the processed data; building multiple Meta nodes and multiple Data nodes in the server cluster to obtain a storage cluster; slicing the processed data to obtain multiple fragment data; inputting the fragment data into a preset anomaly detection model to obtain anomaly detection results; based on the anomaly detection results, matching the data transmission encryption protocol of the fragment data to obtain a data encryption protocol set for each fragment data; based on the data encryption protocol of each fragment data, storing multiple fragment data to the storage cluster. The present invention can select personalized encryption schemes for different types of monitoring data by analyzing data attributes and matching encryption protocols, thereby improving the pertinence and effectiveness of data security.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a monitoring data storage method and system based on a time series database. Background Art

[0002] In the context of the rapid development of information technology, the importance of business data and indicators has become increasingly prominent, and monitoring systems have emerged as a result. The monitoring system provides timely monitoring and alerts for system developers and operation and maintenance personnel by displaying the core indicators and performance data of the business system in real time, helping business personnel to quickly discover and solve potential problems and ensure the stable operation of the business. In this context, the storage of indicator data of the monitoring system faces the threat of a single storage point. Summary of the invention

[0003] In view of this, an embodiment of the present invention provides a monitoring data storage method and system based on a time series database, which are used to improve the efficiency and security of monitoring data storage based on a time series database.

[0004] The present invention provides a monitoring data storage method based on a time series database, comprising: collecting monitoring data to be stored in a time series database, and preprocessing the monitoring data to be stored to obtain processed data; building multiple Meta nodes and multiple Data nodes in a server cluster to obtain a storage cluster; slicing the processed data to obtain multiple fragment data; inputting the multiple fragment data into a preset anomaly detection model to perform data anomaly detection to obtain anomaly detection results; based on the anomaly detection results, matching the data transmission encryption protocol of the multiple fragment data to obtain a data encryption protocol set for each fragment data; based on the data encryption protocol of each fragment data, storing the multiple fragment data in the storage cluster.

[0005] In the present invention, the step of collecting the monitoring data to be stored in the time series database and preprocessing the monitoring data to be stored to obtain the processed data includes: collecting the monitoring data to be stored in the time series database, performing outlier processing on the monitoring data to be stored to obtain the first candidate data; filling the missing values ​​of the first candidate data to obtain the second candidate data; performing data cleaning on the second candidate data to obtain the third candidate data; extracting the time series information of the third candidate data to obtain the target time series information; performing time synchronization processing on the third candidate data based on the target time series information to obtain the processed data.

[0006] In the present invention, the step of building multiple Meta nodes and multiple Data nodes in a server cluster to obtain a storage cluster includes: building multiple Meta nodes on multiple first servers in the server cluster; building multiple Data nodes on multiple second servers in the server cluster; and configuring node information for multiple Meta nodes and multiple Data nodes respectively to obtain the storage cluster.

[0007] In the present invention, the step of slicing the processed data to obtain a plurality of fragment data includes: matching the processed data with a data slicing strategy to obtain a target slicing strategy; dividing the processed data into time periods to obtain a plurality of sub-time periods; slicing the processed data based on the plurality of sub-time periods to obtain a plurality of sub-data sets; and slicing the plurality of sub-data sets through the target slicing strategy to obtain a plurality of fragment data.

[0008] In the present invention, the step of inputting the plurality of the fragment data into a preset anomaly detection model to perform data anomaly detection and obtain anomaly detection results includes: inputting the plurality of the fragment data into the anomaly detection model to construct a data box plot to obtain a target box plot; extracting image trend data from the target box plot to obtain target trend data; performing image segmentation on the target box plot based on the target trend data to obtain multiple sub-box plots; performing abnormal drift point identification on each sub-box plot to obtain an abnormal drift point set for each sub-box plot; performing anomaly value analysis on the abnormal drift point set for each sub-box plot to obtain a data anomaly value for each sub-box plot; performing data anomaly detection on the plurality of the fragment data based on the data anomaly value for each sub-box plot to obtain anomaly detection results.

[0009] In the present invention, the step of matching the data transmission encryption protocol for the multiple data fragments based on the anomaly detection result to obtain a data encryption protocol set for each data fragment includes: extracting the data to be encrypted for the multiple data fragments based on the anomaly detection result to obtain multiple data fragments to be encrypted; performing data segmentation on the multiple data fragments based on the multiple data fragments to be encrypted to obtain multiple first data fragments and multiple second data fragments; matching the data transmission encryption protocol for the multiple first data fragments to obtain a first data encryption protocol; matching the data transmission encryption protocol for the multiple second data fragments to obtain a second data encryption protocol; and merging the first data encryption protocol and the second data encryption protocol into the data encryption protocol set.

[0010] In the present invention, the step of matching the data transmission encryption protocol for the multiple first fragment data to obtain the first data encryption protocol includes: performing data attribute analysis on the multiple first fragment data to obtain data attribute data; and matching the data transmission encryption protocol for the multiple first fragment data based on the data attribute data to obtain the first data encryption protocol.

[0011] The present invention also provides a monitoring data storage system based on a time series database, comprising:

[0012] The acquisition module is used to acquire the monitoring data to be stored in the time series database, and pre-process the monitoring data to be stored to obtain processed data;

[0013] The building module is used to build multiple Meta nodes and multiple Data nodes in the server cluster to obtain a storage cluster;

[0014] A slicing module, used for slicing the processed data to obtain a plurality of fragment data;

[0015] A detection module, used for inputting the plurality of fragment data into a preset anomaly detection model to perform data anomaly detection and obtain an anomaly detection result;

[0016] A matching module, configured to match the data transmission encryption protocol of the plurality of data segments based on the anomaly detection result, to obtain a data encryption protocol set for each data segment;

[0017] The storage module is used to store the plurality of fragment data in the storage cluster based on a data encryption protocol of each of the fragment data.

[0018] In the technical solution provided by the present invention, by matching the data transmission encryption protocol, a suitable encryption scheme is selected for each fragment of data, which effectively ensures the confidentiality and integrity of the data during transmission and reduces the risk of data theft or tampering. The use of the anomaly detection model, combined with the analysis of data outliers, can timely identify abnormal situations in monitoring data, making the processing of abnormal data more accurate, and helping to timely discover and solve potential problems. By building Meta nodes and Data nodes in a server cluster and configuring node information, the construction and management of a storage cluster are realized. Such an architecture can improve the scalability and reliability of the storage system and effectively manage large-scale monitoring data. The matching and implementation of data sharding strategies make the management of processed data more flexible and efficient. By segmenting and distributing data, the data processing speed and the overall performance of the system can be improved. By analyzing data attributes and matching encryption protocols, personalized encryption schemes can be selected for different types of monitoring data, improving the pertinence and effectiveness of data security. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0020] Figure 1 The present invention is a flowchart of a monitoring data storage method based on a time series database in an embodiment of the present invention.

[0021] Figure 2 The flowchart is for building multiple Meta nodes and multiple Data nodes in a server cluster in an embodiment of the present invention.

[0022] Figure 3 Schematic diagram of a monitoring data storage system based on a time series database in an embodiment of the present invention.

[0023] Reference numerals:

[0024] 301, acquisition module; 302, building module; 303, segmentation module; 304, detection module; 305, matching module; 306, storage module. DETAILED DESCRIPTION

[0025] The technical solution of the present invention will be described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0026] In addition, the technical features involved in the different embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0027] For ease of understanding, the specific process of the embodiment of the present invention is described below. Figure 1 , Figure 1 is a flow chart of a monitoring data storage method based on a time series database according to an embodiment of the present invention. Figure 1 As shown, the following steps are included:

[0028] S101, collecting monitoring data to be stored in a time series database, and preprocessing the monitoring data to be stored to obtain processed data;

[0029] S102, building multiple Meta nodes and multiple Data nodes in the server cluster to obtain a storage cluster;

[0030] S103, fragmenting the processed data to obtain multiple fragment data;

[0031] S104, inputting the plurality of fragment data into a preset anomaly detection model to perform data anomaly detection to obtain an anomaly detection result;

[0032] S105, based on the anomaly detection result, performing data transmission encryption protocol matching on the multiple fragment data to obtain a data encryption protocol set for each fragment data;

[0033] S106. Based on the data encryption protocol of each fragment data, store the plurality of fragment data in the storage cluster.

[0034] It should be noted that the monitoring data to be stored is first collected from the time series database. These monitoring data may include the CPU usage, network bandwidth, disk space usage, etc. of the server. Then, the monitoring data to be stored is preprocessed to ensure the quality and accuracy of the data. For example, preprocessing operations such as outlier processing, missing value filling, and data cleaning can be performed on the monitoring data. For example, for the CPU usage data of the server, if there are abnormal fluctuations or missing values, the smoothing algorithm can be used to process the outliers, the interpolation method can be used to fill the missing values, and the duplicate data can be removed to ensure the consistency and reliability of the data. After preprocessing the monitoring data to be stored, it is necessary to build a server cluster and deploy multiple Meta nodes and Data nodes in it to build a storage cluster. These Meta nodes are responsible for managing metadata, while the Data nodes are responsible for storing actual data. For example, two Meta nodes and three Data nodes can be built on three servers respectively to ensure the high availability and fault tolerance of the cluster.

[0035] Subsequently, the processed monitoring data is sharded into multiple fragments. This allows for horizontal expansion and load balancing of data, ensuring that data is evenly distributed in the storage cluster. For example, the monitoring data for a day is sharded by hour or minute, and then the data for each time period is stored on different Data nodes. Next, multiple fragments of data are input into a preset anomaly detection model for data anomaly detection. The anomaly detection model can determine whether the current data is abnormal based on the historical information of the monitoring data. For example, machine learning algorithms such as support vector machines (SVMs) can be used for anomaly detection to identify anomalies in monitoring data, such as abnormal fluctuations or abnormal trends. In this way, the anomaly detection results for each fragment of data can be obtained, indicating which data fragments have anomalies.

[0036] Based on the anomaly detection results, multiple fragment data are matched with data transmission encryption protocols to obtain a data encryption protocol set for each fragment data. For example, for data fragments identified as abnormal, a more secure SSL / TLS encryption protocol can be selected for data transmission encryption to protect the security of the data. For normal data fragments, a lighter encryption protocol can be selected to ensure transmission efficiency. Finally, based on the data encryption protocol of each fragment data, multiple fragment data are stored in the storage cluster. This can ensure the security and reliability of data during storage and transmission. For example, for data fragments using SSL / TLS encryption protocols, they can be stored on Data nodes with SSL / TLS support to ensure that the data is encrypted and protected during storage. For other data fragments, appropriate storage nodes can be selected for storage to achieve distributed storage and load balancing of data. Through the implementation of the above solutions, monitoring data can be effectively collected, preprocessed, stored and protected, providing reliable data support for the stable operation of the system.

[0037] By executing the above steps, matching the data transmission encryption protocol, and selecting a suitable encryption scheme for each fragment of data, the confidentiality and integrity of the data during transmission are effectively guaranteed, and the risk of data theft or tampering is reduced. The use of the anomaly detection model, combined with the analysis of data outliers, can timely identify anomalies in the monitoring data, making the processing of abnormal data more accurate, and helping to timely discover and solve potential problems. By building Meta nodes and Data nodes in the server cluster and configuring node information, the construction and management of the storage cluster are realized. Such an architecture can improve the scalability and reliability of the storage system and effectively manage large-scale monitoring data. The matching and implementation of the data sharding strategy makes the management of processed data more flexible and efficient. By segmenting and distributing data, the data processing speed and the overall performance of the system can be improved. By analyzing data attributes and matching encryption protocols, personalized encryption schemes can be selected for different types of monitoring data, improving the pertinence and effectiveness of data security.

[0038] In a specific embodiment, the process of executing step S101 may specifically include the following steps:

[0039] (1) Collect the monitoring data to be stored in the time series database, process the abnormal values ​​of the monitoring data to be stored, and obtain the first candidate data;

[0040] (2) Fill the missing values ​​of the first candidate data to obtain the second candidate data;

[0041] (3) Perform data cleaning on the second candidate data to obtain third candidate data;

[0042] (4) Extracting the timing information of the third candidate data to obtain the target timing information;

[0043] (5) Perform time synchronization processing on the third candidate data based on the target timing information to obtain processed data.

[0044] Specifically, first, the monitoring data to be stored in the time series database is collected. These data include the CPU usage rate, network bandwidth, disk space usage, etc. of the server. The collected data may contain abnormal values, such as sudden data fluctuations or extreme values. In order to ensure the accuracy and reliability of the data, the monitoring data to be stored is processed for abnormal values. For example, assuming that the CPU usage rate of the server at a certain moment is abnormally high and exceeds the normal range, this abnormal value can be corrected to a reasonable value, such as using a smoothing algorithm to process the abnormal value so that it is consistent with the surrounding data. After processing the abnormal value, the first candidate data is then filled with missing values. There are often missing values ​​in the monitoring data, such as data cannot be collected due to sensor failure or network delay. In order to ensure the integrity of the data, the interpolation method can be used to fill the missing values. For example, for the CPU usage data that has not been collected in a certain period of time, the linear interpolation method can be used to interpolate the data based on the data of the previous and next time points to fill the missing values ​​so that the data sequence is continuous.

[0045] After filling the missing values, the second candidate data is cleaned. Data cleaning includes operations such as removing duplicate data and processing erroneous data to ensure data consistency and reliability. For example, if there are duplicate records or data formats that do not conform to the specifications in the monitoring data, these data can be cleared or corrected. After cleaning the data, time series information can be extracted from the third candidate data. Time series information extraction refers to extracting time-related information from the original monitoring data, such as timestamps, time intervals, etc. These time series information can help better understand the time series characteristics of the data and prepare for subsequent time synchronization processing.

[0046] Based on the extracted target time series information, the third candidate data can be time synchronized. Time synchronization refers to aligning data in different time periods to the same time axis to facilitate subsequent data analysis and application. For example, monitoring data in different time periods can be aligned according to timestamps so that they are consistent in time.

[0047] In a specific embodiment, if Figure 2 As shown, the process of executing step S102 may specifically include the following steps:

[0048] S201, building multiple Meta nodes on multiple first servers in a server cluster;

[0049] S202, building multiple Data nodes on multiple second servers in the server cluster;

[0050] S203: Perform node information configuration on multiple Meta nodes and multiple Data nodes respectively to obtain a storage cluster.

[0051] It should be noted that a server cluster is established to build a storage environment. First, multiple Meta nodes are built on multiple first servers in the server cluster. The Meta node is responsible for managing the metadata of the cluster, including information about data shards, node status, and data replication. By building multiple Meta nodes, the fault tolerance and availability of the storage cluster can be improved. For example, suppose there are three servers, namely Server1, Server2, and Server3. Two Meta nodes are deployed on Server1 and Server2, namely Meta1, Meta2 and Meta3, Meta4. In this way, even if one of the servers fails, the storage cluster can continue to operate normally to ensure the security and reliability of data.

[0052] Next, multiple Data nodes are built on multiple second servers in the server cluster. Data nodes are responsible for storing actual monitoring data and reading and writing data. By building multiple Data nodes, horizontal expansion and load balancing of data can be achieved, and the performance and scalability of the storage cluster can be improved. For example, three Data nodes can be deployed on Server2 and Server3, namely Data1, Data2, Data3 and Data4, Data5, Data6. This ensures that data is evenly distributed in the storage cluster to avoid the situation where some nodes are overloaded or data is unbalanced. For the multiple Meta nodes and multiple Data nodes built, node information configuration is required to ensure the normal operation and management of the cluster. Node information configuration includes specifying the host name or IP address of the node, setting the communication port between nodes, configuring the role and permissions of the node, etc. Through reasonable configuration, it can be ensured that the nodes can communicate and collaborate with each other to achieve efficient operation of the storage cluster. For example, the replication strategy of metadata can be configured in the Meta node to ensure the consistency and availability of metadata. At the same time, the sharding storage strategy of data is configured in the Data node to ensure the even distribution and high availability of data in the cluster.

[0053] In a specific embodiment, the process of executing step S103 may specifically include the following steps:

[0054] (1) Match the data sharding strategy to the processed data to obtain the target sharding strategy;

[0055] (2) Divide the processed data into time periods to obtain multiple sub-time periods;

[0056] (3) Segment the processed data based on multiple sub-time periods to obtain multiple sub-data sets;

[0057] (4) Through the target sharding strategy, multiple sub-data sets are sharded to obtain multiple fragment data.

[0058] Specifically, in order to achieve effective management and storage of processed data, it is necessary to design a suitable data sharding strategy. First, match the data sharding strategy for the processed data to determine the target sharding strategy that suits the current environment and needs. This target sharding strategy can be determined based on factors such as the characteristics of the data, the scale of the storage cluster, and performance requirements. For example, a time-based sharding strategy can be adopted to shard the monitoring data according to time periods to achieve time series storage and management of the data. Next, the processed data is divided into time periods to divide the entire data set into multiple sub-time periods. These sub-time periods can be determined based on the time span and analysis requirements of the monitoring data. For example, the monitoring data within a day can be divided into a sub-time period per hour, which can better manage and analyze the data.

[0059] Based on multiple sub-time periods, the processed data is further segmented to obtain multiple sub-data sets. These sub-data sets can be determined based on the characteristics of the monitoring data and the sharding strategy of the storage cluster. For example, for the monitoring data in a certain sub-time period, it can be segmented according to the server or monitoring indicator to obtain multiple sub-data sets, each of which corresponds to a specific server or monitoring indicator. Finally, through the target sharding strategy, multiple sub-data sets are segmented to obtain multiple fragmented data. This can achieve horizontal expansion and load balancing of data, ensure that data is evenly distributed in the storage cluster, and improve data storage efficiency and access speed. For example, for sub-data sets divided by time period, each sub-data set can be segmented according to the timestamp according to the target sharding strategy to obtain fragmented data in multiple time periods for subsequent data storage and management.

[0060] In a specific embodiment, the process of executing step S104 may specifically include the following steps:

[0061] (1) Input multiple fragment data into the anomaly detection model to construct the data box plot and obtain the target box plot;

[0062] (2) Extracting image trend data from the target box plot to obtain target trend data;

[0063] (3) Perform image segmentation on the target box plot based on the target trend data to obtain multiple sub-box plots;

[0064] (4) Identify the abnormal drift points of each sub-box plot respectively to obtain the abnormal drift point set of each sub-box plot;

[0065] (5) Perform outlier analysis on the abnormal drift point set of each sub-box plot to obtain the data outlier value of each sub-box plot;

[0066] (6) Based on the data outliers of each sub-box plot, data anomaly detection is performed on multiple fragments of data to obtain anomaly detection results.

[0067] Specifically, in the process of realizing anomaly detection of monitoring data, multiple fragment data are first input into the anomaly detection model. In order to identify anomalies more accurately, an anomaly detection method based on a data box plot is selected. By constructing a data box plot for each fragment data, a target box plot can be obtained, which shows the distribution of the monitoring data and the location of the outliers. Based on the target box plot, image trend data extraction is performed, and the target trend data is extracted from the graph. These trend data reflect the changing trend and periodicity of the monitoring data. For example, for the CPU usage data of a group of servers, the average value or change range of the CPU usage in each time period, as well as its trend change, can be extracted from the box plot.

[0068] Next, the target box plot is segmented based on the target trend data and divided into multiple sub-box plots. In this way, the monitoring data can be divided according to time periods or other specific rules, which is convenient for subsequent anomaly detection and analysis. For example, the monitoring data within a day can be divided into multiple sub-time periods, and anomaly detection and analysis can be performed for each time period. For each sub-box plot, abnormal drift point identification is performed separately to identify the set of abnormal drift points in each sub-box plot. An abnormal drift point refers to a point where a sudden and drastic change occurs in the data distribution, which may indicate an abnormality or failure in the system. By identifying abnormal drift points, abnormal conditions in the monitoring data can be discovered in a timely manner, improving the reliability and stability of the system. For example, if the monitoring data in a certain time period has a sudden fluctuation or abnormal value, these abnormal values ​​can be identified as abnormal drift points for subsequent abnormal analysis.

[0069] Next, perform outlier analysis on the abnormal drift point set of each sub-box plot to obtain the data outlier of each sub-box plot. Outlier analysis can help further understand the abnormal situation of monitoring data and take corresponding measures to deal with it. For example, for monitoring data identified as abnormal, its causes and impacts can be further analyzed, and corresponding adjustment or repair measures can be taken. Finally, based on the data outlier of each sub-box plot, data anomaly detection is performed on multiple fragments of data to obtain anomaly detection results. In this way, the abnormal situation of monitoring data can be comprehensively evaluated, and corresponding measures can be taken in time to deal with it. For example, if the monitoring data in multiple sub-time periods are abnormal, the entire system can be investigated and repaired to ensure the stable operation and performance optimization of the system. Through the above steps, abnormal situations in monitoring data can be effectively identified and handled to ensure the reliability and stability of the system.

[0070] In a specific embodiment, the process of executing step S105 may specifically include the following steps:

[0071] (1) Extracting data to be encrypted from multiple fragments of data based on the anomaly detection result to obtain multiple fragments of data to be encrypted;

[0072] (2) based on the plurality of fragment data to be encrypted, performing data segmentation on the plurality of fragment data to obtain a plurality of first fragment data and a plurality of second fragment data;

[0073] (3) performing data transmission encryption protocol matching on the plurality of first fragment data to obtain a first data encryption protocol;

[0074] (4) performing data transmission encryption protocol matching on the plurality of second fragment data to obtain a second data encryption protocol;

[0075] (5) The first data encryption protocol and the second data encryption protocol are combined into a data encryption protocol set.

[0076] In the process of monitoring data storage, it is crucial to protect data security. Based on the anomaly detection results, it is necessary to extract the data to be encrypted from multiple fragments of data to ensure that the data is not stolen or tampered with during transmission. These fragments of data to be encrypted may contain important business indicators and performance data, which need to be effectively protected. For example, assuming that there are server monitoring data in multiple sub-time periods, it is necessary to extract the data fragments to be encrypted from them for subsequent data transmission encryption processing. Based on the multiple fragments of data to be encrypted, these data are segmented to obtain multiple first fragments of data and multiple second fragments of data. Such segmentation operations can be performed according to the characteristics of the data and encryption requirements to achieve effective management and protection of the data. For example, for the monitoring data in a certain time period, it can be segmented into first fragments of data and second fragments of data, each containing different monitoring indicators or business data. For multiple first fragments of data, data transmission encryption protocols are matched to obtain a first data encryption protocol. This encryption protocol can be determined according to the sensitivity of the data and the transmission method to ensure the security of the data during transmission. For example, appropriate encryption algorithms and protocols, such as TLS / SSL, can be selected according to the nature and transmission method of the monitoring data to protect the confidentiality and integrity of the data.

[0077] Similarly, for multiple second fragments of data, data transmission encryption protocols are also matched to obtain a second data encryption protocol. This ensures that different types of data are effectively protected and encrypted during transmission. For example, for log information and user identity data in a set of monitoring data, different encryption protocols and algorithms can be selected to meet the security requirements of different data types.

[0078] Finally, the first data encryption protocol and the second data encryption protocol are combined into a data encryption protocol set. In this way, different types of data encryption protocols can be integrated and managed to facilitate subsequent data transmission and encryption operations. For example, the encryption protocols of all monitoring data can be centrally managed to ensure that the data is effectively protected and managed during transmission and storage.

[0079] In a specific embodiment, the process of performing the step of matching the data transmission encryption protocol for the plurality of first fragment data may specifically include the following steps:

[0080] (1) performing data attribute analysis on the plurality of first fragment data to obtain data attribute data;

[0081] (2) Matching the data transmission encryption protocol of the plurality of first fragment data based on the data attribute data to obtain a first data encryption protocol.

[0082] It should be noted that in the process of monitoring data storage and transmission, the attributes of the data are crucial to the selection of a suitable encryption protocol. First, data attribute analysis is performed on multiple first fragments of data to obtain detailed information about the characteristics and sensitivity of the data. These data attributes include data type, data format, data size, data source, etc. For example, for a set of server monitoring data, data attributes may include the numerical range, change trend, and abnormal conditions of indicators such as CPU usage, memory usage, and network traffic.

[0083] Based on the acquired data attribute data, multiple first fragment data are matched with data transmission encryption protocols to select the first data encryption protocol suitable for the current data attribute. This process needs to be carried out according to the sensitivity of the data, the transmission environment and the security requirements. For example, for numerical monitoring data, the AES encryption algorithm and the TLS transmission protocol can be selected to protect the confidentiality and integrity of the data. For text-based monitoring data, the RSA encryption algorithm and the HTTPS transmission protocol can be selected to protect the security and reliability of the data.

[0084] By analyzing the data attributes and matching the encryption protocol, we can ensure that the data is effectively protected and encrypted during the data transmission process. For example, for a set of server monitoring data, we can determine to use the AES encryption algorithm and TLS transmission protocol for encryption and transmission based on its data attributes to ensure the confidentiality and integrity of the monitoring data. For another set of network log data, we can choose the RSA encryption algorithm and HTTPS transmission protocol to protect the security and reliability of the data.

[0085] The embodiment of the present invention also provides a monitoring data storage system based on a time series database, such as Figure 3 As shown, the monitoring data storage system based on the time series database specifically includes:

[0086] The acquisition module 301 is used to acquire the monitoring data to be stored in the time series database, and pre-process the monitoring data to be stored to obtain processed data;

[0087] A building module 302 is used to build multiple Meta nodes and multiple Data nodes in the server cluster to obtain a storage cluster;

[0088] A fragmentation module 303 is used to fragment the processed data to obtain multiple fragment data;

[0089] A detection module 304 is used to input the plurality of fragment data into a preset anomaly detection model to perform data anomaly detection and obtain an anomaly detection result;

[0090] A matching module 305 is used to match the data transmission encryption protocol of the plurality of data segments based on the anomaly detection result to obtain a data encryption protocol set for each data segment;

[0091] The storage module 306 is configured to store the plurality of fragment data in the storage cluster based on a data encryption protocol for each of the fragment data.

[0092] Through the collaborative work of the above modules and the matching of data transmission encryption protocols, the appropriate encryption scheme is selected for each fragment of data, which effectively ensures the confidentiality and integrity of the data during transmission and reduces the risk of data theft or tampering. The use of anomaly detection models, combined with the analysis of data outliers, can timely identify anomalies in monitoring data, making the processing of abnormal data more accurate, and helping to timely discover and solve potential problems. By building Meta nodes and Data nodes in the server cluster and configuring node information, the construction and management of storage clusters are realized. Such an architecture can improve the scalability and reliability of the storage system and effectively manage large-scale monitoring data. The matching and implementation of data sharding strategies make the management of processed data more flexible and efficient. By segmenting and distributing data, the data processing speed and the overall performance of the system can be improved. By analyzing data attributes and matching encryption protocols, personalized encryption schemes can be selected for different types of monitoring data, improving the pertinence and effectiveness of data security.

[0093] The above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit the same. Although the present invention has been described in detail with reference to the embodiments, a person skilled in the art should understand that the specific implementation modes of the present invention can still be modified or replaced by equivalents, and any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be included in the scope of the claims of the present invention.

Claims

1. A monitoring data storage method based on a time series database, characterized in that: include: Collecting monitoring data to be stored in a time series database, and preprocessing the monitoring data to be stored to obtain processed data; Build multiple Meta nodes and multiple Data nodes in the server cluster to obtain a storage cluster; Slicing the processed data to obtain a plurality of fragment data; Inputting the plurality of fragment data into a preset anomaly detection model to perform data anomaly detection to obtain an anomaly detection result; comprising: Inputting a plurality of the fragment data into the anomaly detection model to construct a data box plot, and obtaining a target box plot; Extracting image trend data from the target box plot to obtain target trend data; Performing image segmentation on the target box plot based on the target trend data to obtain multiple sub-box plots; Identify abnormal drift points for each section of the sub-box plot respectively to obtain an abnormal drift point set for each section of the sub-box plot; Performing outlier value analysis on the abnormal drift point set of each section of the sub-box plot respectively to obtain the data outlier value of each section of the sub-box plot; Based on the data outlier value of each segment of the sub-box plot, data anomaly detection is performed on the plurality of segment data to obtain anomaly detection results; Based on the anomaly detection result, performing data transmission encryption protocol matching on the plurality of fragment data to obtain a data encryption protocol set for each fragment data; Based on a data encryption protocol for each of the segment data, a plurality of the segment data are stored in the storage cluster.

2. The monitoring data storage method based on the time series database according to claim 1 is characterized in that: The step of collecting the monitoring data to be stored in the time series database and preprocessing the monitoring data to be stored to obtain processed data includes: Collecting monitoring data to be stored in a time series database, performing outlier processing on the monitoring data to be stored, and obtaining first candidate data; Filling missing values ​​of the first candidate data to obtain second candidate data; Performing data cleaning on the second candidate data to obtain third candidate data; Extracting timing information from the third candidate data to obtain target timing information; The third candidate data is subjected to time synchronization processing based on the target timing information to obtain the processed data.

3. The monitoring data storage method based on the time series database according to claim 1 is characterized in that: The step of building multiple Meta nodes and multiple Data nodes in the server cluster to obtain a storage cluster includes: Building a plurality of Meta nodes on a plurality of first servers in the server cluster; Building a plurality of the Data nodes on a plurality of second servers in the server cluster; Node information is configured for the plurality of Meta nodes and the plurality of Data nodes respectively to obtain the storage cluster.

4. The monitoring data storage method based on a time series database according to claim 1 is characterized in that: The step of slicing the processed data to obtain a plurality of fragment data includes: Performing data sharding strategy matching on the processed data to obtain a target sharding strategy; Dividing the processed data into time periods to obtain a plurality of sub-time periods; Segmenting the processed data based on the multiple sub-time periods to obtain multiple sub-data sets; The target sharding strategy is used to perform data sharding on the plurality of sub-data sets to obtain a plurality of the fragment data.

5. The monitoring data storage method based on time series database according to claim 1 is characterized in that: The step of performing data transmission encryption protocol matching on the plurality of data segments based on the anomaly detection result to obtain a data encryption protocol set for each data segment comprises: Based on the abnormality detection result, extracting data to be encrypted from the plurality of fragment data to obtain a plurality of fragment data to be encrypted; Based on the plurality of fragment data to be encrypted, data segmentation is performed on the plurality of fragment data to obtain a plurality of first fragment data and a plurality of second fragment data; Perform data transmission encryption protocol matching on the plurality of first fragment data to obtain a first data encryption protocol; Perform data transmission encryption protocol matching on the plurality of the second fragment data to obtain a second data encryption protocol; The first data encryption protocol and the second data encryption protocol are combined into the data encryption protocol set.

6. The monitoring data storage method based on the time series database according to claim 5 is characterized in that: The step of performing data transmission encryption protocol matching on the plurality of first fragment data to obtain a first data encryption protocol comprises: Performing data attribute analysis on the plurality of first fragment data to obtain data attribute data; Based on the data attribute data, data transmission encryption protocols are matched on the plurality of first fragment data to obtain a first data encryption protocol.

7. A monitoring data storage system based on a time series database, used to execute the monitoring data storage method based on a time series database according to any one of claims 1 to 6, characterized in that: include: The acquisition module is used to acquire the monitoring data to be stored in the time series database, and pre-process the monitoring data to be stored to obtain processed data; The building module is used to build multiple Meta nodes and multiple Data nodes in the server cluster to obtain a storage cluster; A slicing module, used for slicing the processed data to obtain a plurality of fragment data; A detection module, used for inputting the plurality of said fragment data into a preset anomaly detection model to perform data anomaly detection and obtain an anomaly detection result; comprising: Inputting a plurality of the fragment data into the anomaly detection model to construct a data box plot, and obtaining a target box plot; Extracting image trend data from the target box plot to obtain target trend data; Performing image segmentation on the target box plot based on the target trend data to obtain multiple sub-box plots; Identify abnormal drift points for each section of the sub-box plot respectively to obtain an abnormal drift point set for each section of the sub-box plot; Performing outlier value analysis on the abnormal drift point set of each section of the sub-box plot respectively to obtain the data outlier value of each section of the sub-box plot; Based on the data outlier value of each segment of the sub-box plot, data anomaly detection is performed on the plurality of segment data to obtain anomaly detection results; A matching module, configured to match the data transmission encryption protocol of the plurality of data segments based on the anomaly detection result, to obtain a data encryption protocol set for each data segment; The storage module is used to store the plurality of fragment data in the storage cluster based on a data encryption protocol of each of the fragment data.