Data processing method, system and device of time series database and storage medium
By processing time-series data using custom calculation rules and calculation chains, the problem of huge resource consumption in time-series database clusters is solved, and efficient data storage processing is achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- PING AN TECH (SHENZHEN) CO LTD
- Filing Date
- 2022-09-06
- Publication Date
- 2026-05-29
Smart Images
Figure CN115422259B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of big data processing technology, and in particular to a data processing method, system, device and storage medium for time-series databases. Background Technology
[0002] Time series databases are primarily used to process time-stamped (time-series) data. They feature high performance, data compression, automatic cleanup of expired data, and support for various aggregation functions, and are gradually becoming the mainstream database in fields such as monitoring and the Internet of Things.
[0003] Time series data primarily originates from industries such as power, chemical, meteorology, and geographic information systems, which require real-time monitoring, inspection, and analysis equipment to collect and generate industrial data. Conventional real-time monitoring systems in these industries typically have tens of thousands of monitoring points, generating data every second. Each monitoring point can produce multiple data points per second, and each data point must correspond to a unique time. A single real-time monitoring system can generate tens of gigabytes of data daily during actual production applications. Therefore, this industrial data is characterized by high generation frequency, heavy reliance on acquisition time, numerous monitoring points, and a large volume of information.
[0004] Time-series databases can be combined with techniques such as stream computing and downsampling to perform further computational processing before the time-series data is stored in the database. This computational processing can optimize the storage structure of the time-series database, but it often requires a lot of resources, especially in scenarios where multiple time-series databases form a cluster, where the demand for resources is particularly huge.
[0005] Therefore, there is an urgent need for a solution that can effectively support time-series data processing in large-scale time-series database cluster systems. Summary of the Invention
[0006] This invention provides a data processing method, system, device, and storage medium for time-series libraries. Its main purpose is to solve the problem that time-series data processing requires huge resources in the prior art.
[0007] In a first aspect, to achieve the above objectives, the present invention provides a data processing method for a time-series database, the method comprising:
[0008] The time-series data reported to the time-series database is aggregated and processed.
[0009] According to the data processing rules loaded from the preset rule component, the aggregated time-series data is processed through the first and second computing chains; and the processed time-series data is then transmitted to the data forwarding module.
[0010] The processed time-series data is forwarded to the time-series database via the data forwarding module.
[0011] Secondly, in order to solve the above problems, the present invention also provides a data processing system for a time-series database, comprising:
[0012] The time-series data aggregation unit is used to aggregate and process the time-series data reported to the time-series database.
[0013] The data processing unit is used to load data processing rules from the preset rule component, process the aggregated time-series data through the first calculation chain and the second calculation chain, and transmit the processed time-series data to the data forwarding module.
[0014] The data forwarding unit is used to forward the processed time-series data to the time-series database through the data forwarding module.
[0015] Thirdly, in order to solve the above problems, the present invention also provides an electronic device, the electronic device comprising:
[0016] At least one processor; and,
[0017] A memory communicatively connected to the at least one processor; wherein,
[0018] The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the steps of the time-series database data processing method as described above.
[0019] Fourthly, in order to solve the above problems, the present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the data processing method for a time-series database as described above.
[0020] The data processing method, system, device, and storage medium for time-series databases proposed in this invention provide flexible and diverse computational functions for the data reporting to the database process through customized calculation rules and calculation chains, data auditing, and calculation recovery, which can effectively support the database storage processing of large-scale time-series database clusters. Attached Figure Description
[0021] Figure 1 This is a flowchart illustrating a data processing method for a time-series database provided in an embodiment of the present invention;
[0022] Figure 2 This is a data processing example of a time-series database provided in an embodiment of the present invention;
[0023] Figure 3This is an example of performing data calculations on aggregated time-series data according to an embodiment of the present invention;
[0024] Figure 4 This is an example of a computation chain provided in an embodiment of the present invention;
[0025] Figure 5 This is a custom example of two calculation rules provided in one embodiment of the present invention;
[0026] Figure 6 This is a schematic diagram of the logical framework of a data processing system for a time-series database provided in an embodiment of the present invention;
[0027] Figure 7 A schematic diagram of the internal structure of an electronic device that implements a data processing method for a time-series database according to an embodiment of the present invention.
[0028] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation
[0029] To address the issue of excessive resource consumption in existing time-series data processing technologies, this invention proposes a data processing scheme that uses custom calculation rules and chains to process and store reported time-series data in a database, thereby supporting large-scale time-series database cluster storage processing.
[0030] The specific implementation scheme of the present invention will be described in detail below with reference to specific embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the present invention and are not intended to limit the present invention.
[0031] The embodiments of this application can acquire and process relevant time-series data based on artificial intelligence technology. Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.
[0032] Foundational technologies for artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.
[0033] Figure 1 This is a flowchart illustrating a data processing method for a time-series database according to an embodiment of the present invention. The method can be executed by a device, which can be implemented in software and / or hardware.
[0034] like Figure 1 As shown, the data processing method for time-series databases provided in this embodiment includes:
[0035] Step S110: Aggregate the time-series data reported to the time-series database;
[0036] Step S120: According to the data processing rules loaded from the preset rule component, the aggregated time-series data is processed through the first calculation chain and the second calculation chain; and the processed time-series data is transmitted to the data forwarding module.
[0037] Step S130: The processed time-series data is forwarded to the time-series database through the data forwarding module.
[0038] The following will provide a more detailed explanation of each step.
[0039] Generally, aggregation refers to data from different sources arriving at the same location, which can be a message queue, cache, forwarding program, etc. Aggregating reported time-series data means summarizing the time-series data reported according to a preset format into a preset location. In step S110, the reporting methods for time-series data include direct reporting, pull reporting, and proxy reporting.
[0040] Specifically, as an example, the data reporting party can send the data to be reported from the data reporting terminal to a preset address according to the agreed-upon preset format; or a dedicated time-series data aggregation module can periodically pull the time-series data to be aggregated from the data reporting terminal; or the data reporting terminal / data reporting party can send the data to be reported to a proxy server according to the agreed-upon preset format, and the proxy server will forward it to the preset address. The aggregation of all reported time-series data (preset address, preset server, etc.) completes the time-series data aggregation process.
[0041] Based on the industry-specific and time-series characteristics of time-series data, the format of reported time-series data is generally: table + label + timestamp + value.
[0042] Specifically, as an example, the time-series data of a record representing water level is shown below:
[0043]
[0044]
[0045] In the above example of time-series data representing water levels, "name:h2o_feet" is the "table", "time" is the "timestamp", "level description" and "water_level" are the "values", and "location" is the "label". In a time-series data record, there can be 0, 1 or more labels, and there can be one or more values. "Table + label" can be understood as representing a certain type of data.
[0046] This time-series data example has only one label, "location," representing water level records for location = coyote_creek. The water_level values for three different times—2015-08-18T00:00:00Z, 2015-08-18T00:06:00Z, and 2015-08-18T00:12:00Z—are 8.12, 8.005, and 7.887, respectively. The corresponding "level description" value is between 6 and 9 feet. In this example, the "level description" is a range, while the "water_level" is a specific value. This specific value can be used to calculate parameters such as averages and rates of change.
[0047] After the time series data has been aggregated, the process proceeds to step S120, where the aggregated time series data is processed according to the data processing rules loaded from the preset rule component.
[0048] Specifically, after aggregation, the aggregated time-series data can be distributed and processed. Figure 2 This illustrates a specific example of data processing for a time-series database. Figure 3 An example of performing data computation on aggregated time-series data is shown. For example... Figure 2 and Figure 3 As shown, the data reporting party reports time-series data, which is then aggregated. After aggregation, the aggregated time-series data is distributed to three data processing units for parallel processing, and then stored in the corresponding time-series database. The distribution of time-series data can generally be based on table names. For example, tables named A, B, and F can be distributed to the first data processing unit, tables named C and X to the second data processing unit, and so on. This allows multiple pre-defined data processing units to simultaneously process the aggregated data according to their respective loaded data processing rules, thereby improving the processing efficiency of the aggregated data.
[0049] Distributing data by table name is primarily to facilitate cross-table calculations of time-series data. For example, time-series data representing total memory usage and time-series data representing memory utilization can be distributed to the same data calculation, allowing memory usage (memory usage = total memory * memory utilization) to be calculated using preset data processing rules.
[0050] The preset rule component may include various custom calculation rules, as well as a first calculation chain and a second calculation chain. Figure 3 In the embodiment shown, after the original time-series data comes in, it first reaches the first computing chain. The processing principle of the first computing chain is to minimize the impact on the original data, so the computing rules can define a few simple calculations. The second computing chain mainly performs deep processing of the data, so the computing rules in the second computing chain will almost always generate new time-series data.
[0051] Although Figure 3 The embodiment shown has only two computing chains, but since each computing chain can be configured with multiple computing rules, and the computing rules can be freely combined, the effect achieved after data processing by these two computing chains is actually equivalent to multiple links.
[0052] Since the calculation rules can be freely combined, the calculation chain is not necessarily a straight line. Figure 4 This is a specific example of a computation chain. For example... Figure 4 As shown, in this computation chain, the aggregated data is first processed by computation rule A. The result of computation rule A (tentatively referred to as A') is then input into two sub-routes. One sub-routes process A' sequentially using computation rules B, c, and D to obtain the result D' of that sub-routes. The other sub-routes first process A' using computation rule E, and the result E' then enters two sub-routes: computation rule G and computation rule F. The result G' of computation rule G is simultaneously input into computation rules F and H. Computation rule F then performs calculations based on results E' and G' to obtain result F'. D', F', and G' are simultaneously input into computation rule H for processing to obtain the final result H' of the computation chain. This result H' is then used for data forwarding or data caching.
[0053] Because time-series data comes from a wide range of sources and involves various industries, the required data processing methods also vary greatly. Therefore, this method, which allows for the independent definition of calculation rules based on specific data processing requirements, can generate a diverse range of calculation rules. Furthermore, the ability to freely combine calculation rules within the calculation chain enables the time-series database data processing method of this invention to be widely adaptable to the data processing needs of different applications.
[0054] Figure 5These are custom examples of two calculation rules provided for embodiments of the present invention. For example... Figure 5 As shown, a computation rule can include entries such as name, type, rule scope, trigger condition, trigger effect, expression, and data flow direction. The name defines the identity / ID of the computation rule for easy referencing in the computation chain; the type indicates the computation category of the rule; and the rule scope limits the specific fields of the time-series data in the preset format involved in the computation rule. A computation rule can limit multiple rule scopes, for example... Figure 5 The calculation rule 0001 has 3 rules, while the calculation rule 2345 has only one rule. The triggering condition and triggering effect are fixed and related to the type of calculation rule. The expression refers to some special parameters required by the calculation rule. The data flow direction indicates which direction the result of the calculation rule will flow (to another or several calculation rules).
[0055] The following example of processing network interface card (NIC) traffic time-series data will be used to provide a more detailed explanation of the data processing method for the time-series database of the present invention.
[0056] Assuming network interface card (NIC) traffic is reported every ten seconds and stored in a table named "NIC Traffic," the same NIC on the same machine will generate 6 data points per minute, in bytes. To save the machine's network speed in kb / s (the total needs to be calculated for multiple NICs on the same machine), aggregation is needed once per minute (i.e., 1 data point per minute). In this example, NIC traffic has two labels: machine ID and NIC name.
[0057] In this scenario, traffic can be routed using traffic splitting rules to separate data with the table name "network card traffic". Then, using unit conversion rules, the data value is divided by 1024 to obtain a value in KB. Next, using preset aggregation rules, the data is aggregated by machine ID to obtain the machine's network traffic. Finally, using the rate of change rule, the machine traffic in the next minute is subtracted from the machine traffic in the previous minute at 1-minute intervals, and then divided by 60 to obtain the machine's network speed within 1 minute.
[0058] Similarly, as shown in the example above, different branches can be made in a computation chain by combining different computation rules to calculate other desired values.
[0059] In another specific embodiment of the present invention, the aggregated time-series data is processed through a first computation chain and a second computation chain according to data processing rules loaded from a preset rule component, including:
[0060] S121. Synchronize the calculation rules and calculation chains from the preset rule components, and update the calculation rules of the first calculation chain and the second calculation chain according to the synchronized calculation rules and calculation chains.
[0061] S122: Based on the first computation chain, perform computational processing on the aggregated time-series data (data specification verification, empty data verification, timestamp verification, tag change, etc.); then, pass the data that has completed the first computation chain to the data forwarding module and the data caching module, and pass the filtered data to the auditing module. The verification rules involved in this filtering include, but are not limited to, data specification verification, empty data verification, timestamp verification, etc.
[0062] S123: Based on the second calculation chain, perform calculation processing (aggregation calculation, rate of change calculation, unit conversion, cross-table calculation, traffic splitting, etc.) on the data cached in the data caching module after processing by the first calculation chain, and then pass the processing result to the data forwarding module; wherein, the composition of the calculation rules in the second calculation chain is customizable, and the type, quantity and order are not limited. The calculation rules of the second calculation chain are mainly data processing type and will generate new data.
[0063] In step S121, the calculation rules for the first and second calculation chains can be customized and freely combined. The scope of the calculation rules includes global calculation rules and table-level calculation rules. Global calculation rules refer to calculation rules that apply to all data, while table-level calculation rules refer to calculation rules that apply to a specified table (a portion of the table in the time-series data format).
[0064] For example, based on a series of possible processing requirements for storing time-series data in the database, the first calculation chain may include calculation rules such as data specification verification, empty data verification, timestamp verification, and tag change; the second calculation chain may include calculation rules such as aggregation calculation, rate of change calculation, unit conversion, cross-table calculation, and traffic splitting.
[0065] More specifically, the specific processing of the aggregated time-series data according to the above calculation rules may include, but is not limited to, the following:
[0066] Data standardization verification: Verify whether the table names, label keys, and field keys of the aggregated time-series data conform to the standards. The standardization requirements for table names, label keys, and field keys of the time-series data can be tailored to the specific time-series data.
[0067] Empty data verification: This function verifies whether the aggregated time-series data is empty, filtering out any empty data. Specifically, it determines whether the time-series data is empty by checking if any field value is empty; if a field value is empty, the entire time-series data is filtered out.
[0068] Timestamp verification: Set the verification range for timestamps to filter out time series data that are outside the specified timestamp range. For example, setting the timestamp verification range to [-20, 5] means that if the timestamp of the aggregated time series data is 20 minutes before or 5 minutes after the current time, it will be filtered out.
[0069] Tag changes: Add tags, delete tags, or modify specified tags; where a tag is a key=value combination, for example: vmId=xxx, disk=sda. A time series data can have multiple tags, which are usually used to describe the characteristics of the data and facilitate retrieval and aggregation;
[0070] Aggregation calculation: Perform aggregation calculations on time series data within a specific range, such as specifying the time series data as tables, time windows, labels, etc.
[0071] Change rate calculation: Calculate the change rate of time series data; where change rate calculation means subtracting the previous point from the next point and then dividing by time.
[0072] Unit conversion: Converting units for time series data; for example, if the unit of a certain data is g, and you want the data in kg, you need to convert the unit according to the unit conversion rules.
[0073] Cross-table calculation: Perform calculations on specified tables based on the same labels and calculation methods;
[0074] Data splitting: Data that meets the specified table name, label, and other conditions is split off from the link.
[0075] More specifically, step S121 may further include:
[0076] S1211: Load calculation rules and calculation chains from the preset rule component, wherein the calculation chains include a first calculation chain and a second calculation chain;
[0077] S1212: Generate a computation link based on the loaded computation rules and computation chain; since the configuration information of the rules and computation chain is stored in the rule component, the data computation component needs to load this configuration information when generating the computation link;
[0078] S1213: Periodically query the preset rule component according to the preset timed query period to determine whether the calculation rules and calculation chains in the preset rule component have been updated, and then update the rules of the calculation chain according to the updates of the calculation rules and calculation chains.
[0079] In step S122, since the first calculation chain needs to minimize its impact on the original data, its calculation rules mainly consist of verification and adjustment rules, not limited to the four types mentioned: data specification verification, empty data verification, timestamp verification, and tag change. Of course, the first calculation chain doesn't necessarily include these four types of calculation rules; it can also customize calculation rules according to specific data processing needs, with no limit on the type, quantity, or order. For example, performing data specification verification of type A on some types of data and data specification verification of type B on other types of data, etc.
[0080] After the time-series data collected in step S122 is processed by the complete first computing chain, the data is passed to the data forwarding module to forward the data from the first computing chain to the time-series database. The data is also passed to the data caching module to cache the data from the first computing chain for later use by the second computing chain. The filtered data is passed to the auditing module to collect the data filtered out by the first computing chain for statistical analysis and recording.
[0081] After the filtered data is passed to the audit module, the audit module performs statistics and records the data. Specifically, as an example, the specific steps for the audit module to collect, statistically analyze, and record the data filtered out by the first computation chain may include:
[0082] S1221: Determine the audit period; wherein, the determination of the audit period is similar to the data retrieval period of the second calculation chain mentioned above. Let the current time be CT, the delay time be LT, the time interval be R, and the last end time be ET. When CT>=ET+R+LT, then let ST=ET, ET=ST+R, and determine the audit period as the time window [ST,ET).
[0083] S1222: Aggregate the data filtered out by the first calculation chain periodically according to the audit cycle. That is, aggregate the data within the time window [ST, ET).
[0084] Specifically, the aggregation process may include: aggregating and counting the data sequences and filtering reasons filtered out by the first computation chain according to the time window, and adding a field to record the filtering reasons. Since the data is continuously reported, in order to avoid generating a large amount of garbage data, the amount of data can be greatly reduced by aggregating the data filtered out by the first computation chain.
[0085] The data sequence here refers to a combination of tables and labels used to represent a certain type of time-series data; the table is the table in the time-series database. For example, if disk write speed is used as a table, and the labels include machine ID and disk, then the disk write speed with machine ID = AA and disk = sda can represent this type of time-series data.
[0086] In step S123, the second calculation chain periodically processes the data in the data cache module. Specifically, as an example, a control module can periodically notify the second calculation chain to perform calculations: Let the current time be CT, the delay time be LT, the time interval be R, and the last end time (the last end time of the calculation performed by the second calculation chain) be ET. When CT >= ET + R + LT, then let the current start time ST = the last end time ET. After the time interval R, the current end time ET = the current start time ST + R. Data within the time window [ST, ET) is retrieved from the data cache and calculated by the second calculation chain; the cache with times less than ET is cleared.
[0087] Furthermore, in the aggregation calculation, for aggregation calculations performed by time windows, the window size B of the time window can be specified as an integer multiple of the time interval R, B = nR, where n is an integer; during runtime, for aggregation calculations performed by time windows, this type of calculation can first calculate intermediate results at time interval R and increment a counter. If the current counter value is less than n, the intermediate results are cached; if the current counter value is equal to n, the counter is cleared, and the intermediate results are aggregated to obtain the calculation result data of this type of rule.
[0088] Specifically, as an example, suppose some time-series data is reported every 10 seconds. If you need to calculate the average value over 1 minute, theoretically you could simply sum the data points within the same minute and then divide by the number of data points. The intermediate result would then be the sum of the data points and the number of data points. Similarly, this method can also be used to calculate the maximum value over 5 minutes or other similar values.
[0089] After the time-series data has been processed as described above, it is transmitted to the data forwarding module, and then proceeds to step S130. The data forwarding module forwards the data from the first computing chain and the second computing chain to the time-series database, thereby completing the final storage of the time-series data in the database.
[0090] As can be seen from the above embodiments, the data processing method for time-series databases provided by this invention offers flexible and diverse computational functions during the data reporting and data storage process, including custom computational rules and chains, data auditing, and computational recovery, effectively supporting the storage processing of large-scale time-series database clusters. Corresponding to the above-described data processing method for time-series databases, this invention also provides a data processing system for time-series databases.
[0091] Figure 6 This is a schematic diagram of the logical framework of a time-series database data processing system according to an embodiment of the present invention.
[0092] like Figure 6As shown, the data processing system 600 for the time-series database of the present invention can be installed in an electronic device. Depending on the functions implemented, the data processing system for the time-series database may include a time-series data aggregation unit 610, a data processing unit 620, and a database storage unit 630. The unit described in this invention can also be called a module, which refers to a series of computer program segments that can be executed by the processor of an electronic device and can perform a fixed function, and which are stored in the memory of the electronic device.
[0093] In this embodiment, the functions of each module / unit are as follows:
[0094] The time series data aggregation unit 610 is used to aggregate the time series data reported to the time series database.
[0095] The data processing unit 620 is used to load data processing rules from the preset rule component, process the aggregated time-series data through the first calculation chain and the second calculation chain, and transmit the processed time-series data to the data forwarding module.
[0096] The data transfer unit 630 is used to forward the processed time-series data to the time-series database through the data forwarding module.
[0097] Specifically, the time-series data aggregation unit 610 will aggregate the time-series data reported according to the preset format to a preset location (such as a message queue, cache, forwarding program, etc.). The time-series data reporting methods include direct reporting, pull reporting, and proxy reporting. The format of the reported time-series data is generally: table + tag + timestamp + value.
[0098] The data processing unit 620 performs multi-channel distribution processing on the aggregated time-series data. During each distribution process, the aggregated time-series data is processed according to data processing rules loaded from a preset rule component, and then stored in the corresponding time-series database. The distribution of time-series data can generally be based on table names. For example, data with table names A, B, and F can be distributed to the first data processing unit, data with table names C and X can be distributed to the second data processing unit, and so on. This allows multiple preset data processing units to simultaneously process the aggregated data according to their respective loaded data processing rules, thereby improving the processing efficiency of the aggregated data.
[0099] Distributing data by table name facilitates cross-table calculations on time-series data. For example, time-series data representing total memory usage and time-series data representing memory utilization can be distributed to the same data calculation, so that memory usage (memory usage = total memory * memory utilization) can be calculated using preset data processing rules.
[0100] The preset rule component can include various custom rules, as well as a first calculation chain and a second calculation chain. Since each calculation chain can be configured with multiple calculation rules, and these rules can be freely combined, the effect achieved after data processing through these two calculation chains is essentially equivalent to multiple links. Generally, when the original time-series data arrives, it first reaches the first calculation chain. The processing principle of the first calculation chain is to minimize the impact on the original data, so the calculation rules can define a few simple calculations. The second calculation chain mainly performs deep data processing; therefore, the calculation rules in the second calculation chain almost always generate new time-series data.
[0101] Because time-series data comes from a wide range of sources and involves various industries, the required data processing methods also vary greatly. Therefore, this method, which allows for the independent definition of calculation rules based on specific data processing requirements, can generate a diverse range of calculation rules. Furthermore, the ability to freely combine calculation rules within the calculation chain enables the time-series database data processing method of this invention to be widely adaptable to the data processing needs of different applications.
[0102] Specifically, as an example, the data processing unit 620 may include:
[0103] The rule synchronization unit 621 is used to synchronize the calculation rules and calculation chains from the rule components, and update the calculation rules of the first calculation chain and the second calculation chain according to the synchronized calculation rules and calculation chains.
[0104] The calculation rules for the first and second calculation chains can be customized and freely combined. The scope of the calculation rules includes global calculation rules and table-level calculation rules. Global calculation rules are calculation rules that apply to all data, while table-level calculation rules are calculation rules that apply to a specified table (the table portion in a time-series data format).
[0105] For example, based on a series of possible processing requirements for storing time-series data in the database, the first calculation chain may include calculation rules such as data specification verification, empty data verification, timestamp verification, and tag change; the second calculation chain may include calculation rules such as aggregation calculation, rate of change calculation, unit conversion, cross-table calculation, and traffic splitting.
[0106] The first computational processing unit 622 is used to perform computational processing on the aggregated time-series data based on the first computational chain (data specification verification, empty data verification, timestamp verification, tag change, etc.); then, the data that has completed the first computational chain is passed to the data forwarding module and the data caching module, and the filtered data is passed to the auditing module. The verification rules involved in this filtering include, but are not limited to, data specification verification, empty data verification, timestamp verification, etc.
[0107] The second calculation processing unit 623 is used to perform calculation processing (aggregation calculation, rate of change calculation, unit conversion, cross-table calculation, traffic splitting, etc.) on the data cached in the data caching module based on the second calculation chain, and then pass the processing result to the data forwarding module; wherein, the composition of the calculation rules in the second calculation chain is customizable, and the type, quantity and order are not limited. The calculation rules of the second calculation chain are mainly data processing type and will generate new data.
[0108] As can be seen, the specific processing of the aggregated time-series data according to the above calculation rules may include, but is not limited to, the following:
[0109] Data standardization verification: Verify whether the table names, label keys, and field keys of the aggregated time-series data conform to the standards. The standardization requirements for table names, label keys, and field keys of the time-series data can be tailored to the specific time-series data.
[0110] Empty data verification: This function verifies whether the aggregated time-series data is empty, filtering out any empty data. Specifically, it determines whether the time-series data is empty by checking if any field value is empty; if a field value is empty, the entire time-series data is filtered out.
[0111] Timestamp verification: Set the verification range for timestamps to filter out time series data that are outside the specified timestamp range. For example, setting the timestamp verification range to [-20, 5] means that if the timestamp of the aggregated time series data is 20 minutes before or 5 minutes after the current time, it will be filtered out.
[0112] Tag changes: Add tags, delete tags, or modify specified tags; where a tag is a key=value combination, for example: vmId=xxx, disk=sda. A time series data can have multiple tags, which are usually used to describe the characteristics of the data and facilitate retrieval and aggregation;
[0113] Aggregation calculation: Perform aggregation calculations on time series data within a specific range, such as specifying the time series data as tables, time windows, labels, etc.
[0114] Change rate calculation: Calculate the change rate of time series data; where change rate calculation means subtracting the previous point from the next point and then dividing by time.
[0115] Unit conversion: Converting units for time series data; for example, if the unit of a certain data is g, and you want the data in kg, you need to convert the unit according to the unit conversion rules.
[0116] Cross-table calculation: Perform calculations on specified tables based on the same labels and calculation methods;
[0117] Data splitting: Data that meets the specified table name, label, and other conditions is split off from the link.
[0118] Furthermore, the rule synchronization unit 621 may further include:
[0119] A rule loading unit is used to load computation rules and computation chains from the rule component, wherein the computation chains include a first computation chain and a second computation chain;
[0120] The computation link generation unit is used to generate computation links based on the loaded computation rules and computation links. Since the configuration information of the rules and computation links is stored in the rule component, the data computation component needs to load this configuration information when generating computation links.
[0121] The periodic update unit is used to periodically query the rule component according to a preset timed query cycle, so as to update the rules of the computing link according to the computing rules and the update of the computing link.
[0122] Since the first computation chain needs to minimize its impact on the original data, its calculation rules mainly consist of verification and adjustment rules, not limited to the four types mentioned: data specification verification, empty data verification, timestamp verification, and tag change. Of course, the first computation chain doesn't necessarily include these four types of calculation rules; custom calculation rules can be defined according to specific data processing needs, with no restrictions on type, number, or order. For example, performing data specification verification (method A) on some types of data and data specification verification (method B) on other types of data.
[0123] After the first computing chain completes the computation and processing of the aggregated time-series data, the data processing unit 620 transmits the data to the data forwarding module so that the data forwarding module can forward the data from the first computing chain to the time-series database. At the same time, the data is transmitted to the data caching module to cache the data from the first computing chain, so as to facilitate the subsequent call of the second computing chain. The filtered data is transmitted to the auditing module to collect the data filtered out by the first computing chain, and to perform statistics and recording.
[0124] After the filtered data is passed to the audit module, the audit module performs statistical analysis and records the data. Specifically, as an example, the audit module further includes:
[0125] The audit period determination module is used to determine the audit period; the determination of the audit period is similar to the data retrieval period of the second calculation chain mentioned above. Let the current time be CT, the delay time be LT, the time interval be R, and the last end time be ET. When CT>=ET+R+LT, then let ST=ET, ET=ST+R, and determine the audit period as the time window [ST,ET).
[0126] The data aggregation module is used to periodically aggregate the data filtered out by the first calculation chain according to the audit period. That is, it aggregates the data within the time window [ST, ET).
[0127] Specifically, the aggregation process may include: aggregating and counting the data sequences and filtering reasons filtered out by the first computation chain according to the time window, and adding a field to record the filtering reasons. Since the data is continuously reported, in order to avoid generating a large amount of garbage data, the amount of data can be greatly reduced by aggregating the data filtered out by the first computation chain.
[0128] After the time-series data has been processed as described above, it is transmitted to the data forwarding module. The data forwarding module can then forward the data from the first computing chain and the second computing chain to the time-series database, thereby completing the final storage of the time-series data into the database.
[0129] like Figure 7 The diagram shown is a structural schematic of an electronic device that implements a data processing method for a time-series database according to an embodiment of the present invention.
[0130] The electronic device 7 may include a processor 70, a memory 71 and a bus, and may also include a computer program stored in the memory 71 and executable on the processor 70, such as a data processing program 72 for a time-series database.
[0131] The memory 71 includes at least one type of readable storage medium, including flash memory, portable hard drive, multimedia card, card-type memory (e.g., SD or DX memory), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 71 can be an internal storage unit of the electronic device 7, such as a portable hard drive. In other embodiments, the memory 71 can be an external storage device of the electronic device 7, such as a plug-in portable hard drive, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the electronic device 7. Furthermore, the memory 71 can include both internal and external storage units of the electronic device 7. The memory 71 can be used not only to store application software and various types of data installed on the electronic device 7, such as the code of a time-series database data processing program, but also to temporarily store data that has been output or will be output.
[0132] In some embodiments, the processor 70 may be composed of integrated circuits, such as a single packaged integrated circuit or multiple integrated circuits with the same or different functions, including combinations of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The processor 70 is the control unit of the electronic device, connecting various components of the entire electronic device through various interfaces and lines. It executes programs or modules (such as time-series database data processing programs) stored in the memory 71, and calls data stored in the memory 71 to perform various functions of the electronic device 7 and process data.
[0133] The bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This bus can be divided into an address bus, a data bus, a control bus, etc. The bus is configured to enable communication between the memory 71 and at least one processor 70, etc.
[0134] Figure 7 Only electronic devices with components are shown; those skilled in the art will understand that... Figure 7 The structure shown does not constitute a limitation on the electronic device 7, and may include fewer or more components than shown, or combine certain components, or have different component arrangements.
[0135] For example, although not shown, the electronic device 7 may also include a power supply (such as a battery) to power the various components. Preferably, the power supply can be logically connected to the at least one processor 70 through a power management device, thereby enabling functions such as charging management, discharging management, and power consumption management. The power supply may also include one or more DC or AC power supplies, recharging devices, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components. The electronic device 7 may also include various sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be described in detail here.
[0136] Furthermore, the electronic device 7 may also include a network interface. Optionally, the network interface may include a wired interface and / or a wireless interface (such as a Wi-Fi interface, a Bluetooth interface, etc.), which is typically used to establish communication connections between the electronic device 7 and other electronic devices.
[0137] Optionally, the electronic device 7 may further include a user interface, which may be a display, an input unit (such as a keyboard), or a standard wired or wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen. The display may also be appropriately referred to as a screen or display unit, used to display information processed in the electronic device 7 and to display a visual user interface.
[0138] It should be understood that the embodiments described are for illustrative purposes only and are not limited to this structure in the scope of the patent application.
[0139] The data processing program 72 of the time-series database stored in the memory 71 of the electronic device 7 is a combination of multiple instructions. When run in the processor 70, it can achieve the following:
[0140] The time-series data reported to the time-series database is aggregated and processed.
[0141] According to the data processing rules loaded from the preset rule component, the aggregated time-series data is processed through the first and second computing chains; and the processed time-series data is then transmitted to the data forwarding module.
[0142] The processed time-series data is forwarded to the time-series database via the data forwarding module.
[0143] Specifically, the processor 70's implementation method for the above instructions can be found in [reference needed]. Figures 1-5 The descriptions of the relevant steps in the corresponding embodiments are not repeated here. It should be emphasized that, to further ensure the privacy and security of the above-mentioned stored files, the stored files can also be stored in a blockchain node.
[0144] Furthermore, if the modules / units integrated in the electronic device 7 are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. The computer-readable medium may include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, or a read-only memory (ROM).
[0145] In the embodiments provided by this invention, it should be understood that the disclosed devices, systems, and methods can be implemented in other ways. For example, the system examples described above are merely illustrative; for instance, the division of modules is only a logical functional division, and other division methods may be used in actual implementation.
[0146] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.
[0147] Furthermore, the functional modules in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional modules.
[0148] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.
[0149] Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be embraced within the invention. No appended diagram markings in the claims should be construed as limiting the scope of the claims.
[0150] The blockchain referred to in this invention is a novel application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and encryption algorithms. Essentially, a blockchain is a decentralized database, a chain of data blocks linked together using cryptographic methods. Each data block contains information about a batch of network transactions, used to verify the validity of the information (anti-counterfeiting) and generate the next block. A blockchain can include an underlying blockchain platform, a platform product service layer, and an application service layer.
[0151] Furthermore, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices recited in a system claim may also be implemented by a single unit or device through software or hardware. The term "second class" is used to indicate names and does not indicate any specific order.
[0152] Finally, it should be noted that the above embodiments are only used to illustrate the technical invention of the present invention and not to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical invention of the present invention without departing from the spirit and scope of the present invention.
Claims
1. A data processing method for a time-series database, applied to an electronic device, characterized in that, The method includes: The time-series data reported to the time-series database is aggregated and processed. According to the data processing rules loaded from the preset rule component, the aggregated time-series data is processed through the first calculation chain and the second calculation chain; and the processed time-series data is transmitted to the data forwarding module; wherein, the preset rule component includes custom calculation rules, as well as the first calculation chain and the second calculation chain; each of the first calculation chain and the second calculation chain is configured with at least one calculation rule, and the calculation rules can be freely combined in the first calculation chain and the second calculation chain; The processed time-series data is forwarded to the time-series database via the data forwarding module. Specifically, based on the data processing rules loaded from the preset rule component, the aggregated time-series data is processed through the first and second computation chains, including: Synchronize the calculation rules and calculation chains from the preset rule components, and update the calculation rules of the first calculation chain and the second calculation chain according to the synchronized calculation rules and calculation chains; Based on the first computing chain, the aggregated time-series data is processed, and the processed data is transmitted to the data forwarding module and the data caching module. Based on the second computing chain, the data cached in the data caching module that has been processed by the first computing chain is processed, and the processing result is transmitted to the data forwarding module. Furthermore, after processing the aggregated time-series data based on the first computing chain, the method further includes: filtering the data processed by the first computing chain and passing the filtered data to the audit module, wherein the audit module is used to collect the data filtered out by the first computing chain for statistical analysis and recording.
2. The data processing method for time-series databases according to claim 1, characterized in that, After aggregating the time-series data reported to the time-series database, the aggregated time-series data is distributed and processed in parallel through multiple channels; among them, In each processing step, the aggregated time-series data is processed through the first and second computing chains according to the data processing rules loaded from the preset rule components.
3. The data processing method for time-series databases according to claim 2, characterized in that, The format of the reported time-series data is: table + label + timestamp + value; During the process of multi-path distribution and parallel processing of the aggregated time-series data, the data is distributed according to the table name of the time-series data.
4. The data processing method for a time-series database according to claim 1, characterized in that, The entries for the calculation rule include name, type, rule scope, triggering condition, triggering effect, expression, and data flow direction; among which, The name is used to define the identity / ID of the calculation rule; The type is used to represent the calculation category of the calculation rule; The rule scope is used to limit the specific fields of the time-series data in the preset format involved in the calculation rule; The triggering conditions and triggering effects are associated with the type of the calculation rule; The expression contains the specific parameters required by the calculation rule; The data flow direction indicates the result flow direction of the calculation rule.
5. The data processing method for a time-series database according to claim 1, characterized in that, Based on the first computing chain, the computational processing performed on the aggregated time-series data includes data specification verification, empty data verification, timestamp verification, and label change. Based on the second calculation chain, the calculation processing performed on the data cached in the data caching module that has been processed by the first calculation chain includes aggregation calculation, rate of change calculation, unit conversion, cross-table calculation, and traffic splitting.
6. The data processing method for a time-series database according to claim 5, characterized in that, The rules for the validation classes involved in the filtering include data specification validation, empty data validation, and timestamp validation.
7. The data processing method for a time-series database according to claim 6, characterized in that, The steps of the audit module in collecting, statistically analyzing, and recording the data filtered out by the first computation chain include: Determine the audit period; where the current time is CT, the delay time is LT, the time interval is R, and the last end time is ET. When CT>=ET+R+LT, then let ST=ET and ET=ST+R, and determine the audit period as the time window [ST, ET). The data filtered out by the first calculation chain is aggregated periodically according to the audit cycle.
8. The data processing method for a time-series database according to claim 7, characterized in that, The aggregation process includes: according to the time window, aggregating and counting the data sequences and filtering reasons filtered out by the first calculation chain, and adding a field to record the filtering reasons.
9. The data processing method for a time-series database according to claim 1, characterized in that, Synchronizing calculation rules and calculation chains from the preset rule component, and updating the calculation rules of the first and second calculation chains according to the synchronized calculation rules and calculation chains, including: The calculation rules and calculation chains are loaded from the preset rule component, wherein the calculation chains include a first calculation chain and a second calculation chain; A computation link is generated based on the loaded computation rules and computation link; The preset rule component is queried periodically according to a preset timed query cycle to determine whether the calculation rules and calculation chains in the preset rule component have been updated, and then the rules of the calculation chain are updated according to the updates of the calculation rules and calculation chains.
10. A data processing system for a time-series database, characterized in that, include: The time-series data aggregation unit is used to aggregate and process the time-series data reported to the time-series database. The data processing unit is used to load data processing rules from the preset rule component and process the aggregated time-series data through the first calculation chain and the second calculation chain. The processed time-series data is then transmitted to the data forwarding module; wherein, the preset rule component includes custom calculation rules, as well as a first calculation chain and a second calculation chain; each of the first and second calculation chains is configured with at least one calculation rule, and the calculation rules can be freely combined in the first and second calculation chains; The data forwarding unit is used to forward the processed time-series data to the time-series database through the data forwarding module. The data processing unit includes: The rule synchronization unit is used to synchronize the calculation rules and calculation chains from the preset rule components, and update the calculation rules of the first calculation chain and the second calculation chain according to the synchronized calculation rules and calculation chains. The first computing processing unit is used to perform computing processing on the aggregated time-series data based on the first computing chain; then, the data that has completed the first computing chain is passed to the data forwarding module and the data caching module, and the data filtered out by the first computing chain is passed to the auditing module, which is used to collect the data filtered out by the first computing chain for statistics and recording. The second computing processing unit is used to perform computing processing on the data cached in the data caching module that has been processed by the first computing chain, based on the second computing chain, and then pass the processing result to the data forwarding module.
11. An electronic device, characterized in that, The electronic device includes: At least one processor; and, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the steps of the data processing method for a time-series database as described in any one of claims 1 to 9.
12. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps in the data processing method for a time-series database as described in any one of claims 1 to 9.