Data downsampling method and device

Abstract

The invention provides a data down-sampling method, and relates to the field of databases, the method comprises the following steps: receiving a user-defined down-sampling strategy of multiple levels, the down-sampling strategy of each level indicating that after original data is stored in multiple time partitions and a defined time interval is passed, the down-sampling strategy of each level is stored in the multiple time partitions; carrying out downsampling on the data in the plurality of time partitions according to a defined sampling period; according to a first time interval after first original data is stored in a first time partition, a first downsampling strategy is determined from the multiple levels of downsampling strategies, and the longer the first time interval is, the longer the sampling period of the first downsampling strategy is; obtaining first data currently stored in the first time partition; and down-sampling the first data according to the first down-sampling strategy to obtain second data. And a user can set a downsampling strategy according to own needs, so that the applicability of downsampling is improved.

Description

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on July 8, 2023, with application number 202310833432.X and application name “A data downsampling method, device and computing device cluster”, the entire contents of which are incorporated by reference in this application. Technical Field

[0002] The present application relates to the field of databases, and more specifically, to a data downsampling method and device. Background Art

[0003] In scenarios such as development operation (DevOps) or the Internet of Things (IoT), users are more sensitive to recent data and often only perform complete data queries on recent data. For older data, users often only perform data trend queries, that is, queries on sampled data. Therefore, for older data, down sampling can be used to process it. Down sampling can aggregate data points within a certain time interval into one or a group of values ​​based on certain rules. Down sampling can reduce the overall amount of stored data, thereby reducing the computing pressure of storage and query.

[0004] Storing data in a database is also called "data storage". Currently, downsampling is often performed before data storage. The data is stored in memory and then stored on disk after a certain period of time, which results in high memory costs.

[0005] Therefore, how to reduce the memory cost during downsampling is an urgent problem to be solved. Summary of the invention

[0006] The present application provides a data downsampling method, which can reduce memory costs during downsampling.

[0007] In a first aspect, a data downsampling method is provided, which is applied to a database, wherein the database includes multiple time partitions, and the multiple time partitions are used to store data in different time periods. The method includes: receiving multiple levels of downsampling strategies defined by a user, wherein each level of the downsampling strategy indicates that after the original data is stored in the multiple time partitions and a defined time interval has passed, the data in the multiple time partitions is downsampled according to a defined sampling period; determining a first downsampling strategy from the multiple levels of downsampling strategies according to a first time interval that has passed after the first original data is stored in the first time partition, wherein the longer the first time interval is, the longer the sampling period of the first downsampling strategy is, and the first time partition is any one of the multiple time partitions; obtaining first data currently stored in the first time partition, wherein the first data is the first original data, or data that has been downsampled at least once to the first original data; and downsampling the first data according to the first downsampling strategy to obtain second data.

[0008] In the embodiment of the present application, the user can set the downsampling strategy according to their own needs, thereby improving the applicability of downsampling. Moreover, the user only needs one instruction, and the background automatically performs maintenance and scheduling, thereby reducing the user's operating and maintenance costs. In this way, when it is necessary to set more levels of downsampling strategies, the data downsampling method provided in the embodiment of the present application can conveniently build a database storage service. On the other hand, the above scheme performs downsampling after the data is dropped into the database, thereby saving memory storage space. Moreover, the above scheme can perform multi-level downsampling, that is, further downsampling is performed based on the downsampled data.

[0009] In combination with the first aspect, in some implementations of the first aspect, the method further includes: storing the second data in the first time partition, and deleting the first data.

[0010] The data downsampling method provided in the embodiment of the present application can perform multi-level downsampling, and replace the data before downsampling with the data after downsampling. The data that has undergone different degrees of downsampling can be stored in the same database. On the basis of ensuring that the data after downsampling can restore the change trend of the original data, the disk storage space is greatly saved, thereby improving the processing speed of the database.

[0011] In combination with the first aspect, in certain implementations of the first aspect, before receiving the user-defined multiple levels of downsampling strategies, the method also includes: receiving the first original data and generating a timestamp of the first original data; based on the timestamp, storing the first original data in the first time partition.

[0012] The data downsampling method provided in the embodiment of the present application can perform downsampling after the data is stored in the disk to the database, thereby saving memory storage space.

[0013] In combination with the first aspect, in certain implementations of the first aspect, the first data is downsampled according to the first downsampling strategy to obtain second data, including: dividing the first data into multiple groups according to the sampling period of the first sampling strategy; and aggregating the data of each group in the multiple groups according to the aggregation rule of the first downsampling strategy to obtain the second data.

[0014] Based on the above solution, the first data is grouped and aggregated using the first downsampling strategy, thereby achieving downsampling of the first data and releasing the storage space of the disk, thereby improving the processing speed of the database.

[0015] In combination with the first aspect, in certain implementations of the first aspect, the second data includes: at least one of the minimum value, maximum value, sum, number of data points, average value, value of the earliest input data point, and value of the latest input data point of each group in the first data.

[0016] Based on the above scheme, the first downsampling strategy is adopted to downsample the first data to obtain the second data. The second data includes at least one aggregate value, so the second data can reflect the change trend of the original data, thereby saving disk storage space and restoring the change trend of the original data, thereby improving the accuracy of downsampling.

[0017] In combination with the first aspect, in certain implementations of the first aspect, the method also includes: receiving a query instruction, the query instruction being used to query a first aggregation result after aggregating the second original data of a second time partition based on a first aggregation rule, wherein the second time partition is any one of the multiple time partitions, and the data currently stored in the second time partition is obtained by aggregating the second original data based on at least one aggregation rule; determining a second aggregation rule based on the first aggregation rule and the at least one aggregation rule; and aggregating the data currently stored in the second time partition based on the second aggregation rule to obtain the first aggregation result.

[0018] The solution for rewriting query statements provided in the present application can perform simpler processing on the downsampled data, thereby saving computing resources.

[0019] In a second aspect, an embodiment of the present application provides a data downsampling device, which includes a module for implementing the first aspect or any possible implementation method of the first aspect.

[0020] In a third aspect, a computing device cluster is provided, comprising at least one computing device, each computing device comprising a processor and a memory; the processor of the at least one computing device is used to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster performs the data downsampling method of the first aspect or any possible implementation of the first aspect.

[0021] In a fourth aspect, a computer program product comprising instructions is provided. When the instructions are executed by a computing device cluster, the computing device cluster executes the data downsampling method of the first aspect or any possible implementation of the first aspect.

[0022] In a fifth aspect, a computer-readable storage medium is provided, comprising computer program instructions. When the computer program instructions are executed by a computing device cluster, the computing device cluster executes the data downsampling method of the first aspect or any possible implementation of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 It is a system diagram based on a cloud server system according to an embodiment of the present application.

[0024] Figure 2 It is a schematic flow chart of a data downsampling method provided in an embodiment of the present application.

[0025] Figure 3 It is a schematic diagram of data aggregation provided in an embodiment of the present application.

[0026] Figure 4 It is a schematic block diagram of a database provided in an embodiment of the present application.

[0027] Figure 5 It is a schematic block diagram of a data downsampling device provided in an embodiment of the present application.

[0028] Figure 6 It is a schematic diagram of a data downsampling method provided in an embodiment of the present application.

[0029] Figure 7 It is a schematic diagram of the first data and the second data provided in the embodiment of the present application.

[0030] Figure 8 It is a schematic diagram of data downsampling and query provided in an embodiment of the present application.

[0031] Fig. 9 An architectural diagram of a computing device provided in an embodiment of the application.

[0032] Fig.10 It is a schematic diagram of the architecture of a computing device cluster provided in an embodiment of the present application.

[0033] Fig.11 It is a schematic diagram of a connection between two computing devices via a network provided in an embodiment of the present application. DETAILED DESCRIPTION

[0034] The technical solution in this application will be described below in conjunction with the accompanying drawings.

[0035] The present application will present various aspects, embodiments or features around a system that may include multiple devices, components, modules, etc. It should be understood and appreciated that each system may include additional devices, components, modules, etc., and / or may not include all of the devices, components, modules, etc. discussed in conjunction with the figures. In addition, combinations of these schemes may also be used.

[0036] In addition, in the embodiments of the present application, words such as "exemplary" and "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" in the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of the word "exemplary" is intended to present concepts in a concrete way.

[0037] The business scenarios described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Ordinary technicians in this field can know that with the emergence of new business scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0038] References to "one embodiment" or "some embodiments" etc. described in this specification mean that a particular feature, structure or characteristic described in conjunction with the embodiment is included in one or more embodiments of the present application. Thus, the phrases "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. appearing in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0039] For ease of understanding, some terms that may be involved in this application are introduced below.

[0040] Time series: This refers to the change of a metric over time. The combination of metrics, labels, and measurements forms a time series.

[0041] Measurement: Measurement is similar to the concept of table. Measurement represents a collection of similar data. For example, a measurement can be established for environmental sensors to store the monitoring data of all environmental sensors.

[0042] Tag: A tag can describe the characteristics of a data point, and the tag does not change over time. For example, the identification of an environmental sensor, the region where the environmental sensor is located, and other information can be used as a tag.

[0043] Field: Similar to the concept of column. Fields can describe the measurement indicators of data, and they usually change over time. For example, data such as temperature, humidity, and pressure detected by environmental sensors can be called field.

[0044] Data point: A data point is similar to a row or record.

[0045] Timestamp: Timestamp can be used to mark the time of a data point. This application does not limit the generation method of timestamp. For example, the timestamp can be the time when the data is collected, the time when it is stored in the database, or the time when the database receives the data point.

[0046] Figure 1 It is a system diagram based on a cloud server system according to an embodiment of the present application.

[0047] like Figure 1 As shown, the cloud management platform 110 is used to manage the infrastructure that provides multiple cloud services, the infrastructure includes multiple cloud data centers, each cloud data center includes multiple cloud servers, each cloud server includes cloud service resources, and provides corresponding cloud services for tenants. For example, the cloud server resources can be cloud databases.

[0048] The cloud management platform 110 provides an access interface (such as an interface or an application program interface (API)). The tenant can operate the client remote access access interface to register a cloud account and password on the cloud management platform and log in to the cloud management platform. After the cloud management platform successfully authenticates the cloud account and password, the tenant can further pay to select and purchase a virtual machine with specific specifications (processor, memory, disk) on the cloud management platform. After the payment is successful, the cloud management platform provides the remote login account and password of the purchased virtual machine. The client can remotely log in to the virtual machine and install and run the tenant's application in the virtual machine. The tenant of the cloud service can be an individual, enterprise, school, hospital, administrative agency, etc.

[0049] The functions of the cloud management platform 110 include, but are not limited to, user console, computing management service, network management service, storage management service, authentication service, and image management service. The user console provides an interface or API to interact with tenants, the computing management service is used to manage servers running virtual machines and containers and bare metal servers, the network management service is used to manage network services (such as gateways, firewalls, etc.), the storage management service is used to manage storage services (such as data bucket services), the authentication service is used to manage tenant accounts and passwords, and the image management service is used to manage virtual machine images. Tenants can use the client 130 to log in to the cloud management platform 110 through the Internet 120 to manage the rented cloud services.

[0050] The tenant can store data in the cloud data center through the cloud management platform 110 to implement storage management services. For example, the tenant can store monitoring data (such as temperature, pressure, etc.) of the sensor in the cloud data center.

[0051] In scenarios such as development operation (DevOps) or the Internet of Things (IoT), users are more sensitive to recent data and often only perform complete data queries on recent data. For older data, users often only perform data trend queries, that is, queries on sampled data. Therefore, for older data, down sampling can be used to process it. Down sampling can aggregate data points within a certain time interval into one or a group of values ​​based on certain rules. Down sampling can reduce the overall amount of stored data, thereby reducing the computing pressure of storage and query.

[0052] Storing data in a database is also called "data storage". Currently, downsampling is often performed before data storage. The data is stored in memory and then stored on disk after a certain period of time, which results in high memory costs.

[0053] Some technical solutions use multiple clusters to store the original data and the downsampled data separately. However, this solution requires storing more data than just storing the original data, which increases maintenance costs and data storage costs.

[0054] Figure 2 1 is a schematic flow chart of a data downsampling method 200 provided in an embodiment of the present application. The method 200 can reduce the memory cost in downsampling. The method 200 is applied to a database, which includes multiple time partitions (shards), and the multiple time partitions are used to store data in different time periods. The present application does not limit the specific execution subject of the method 200, and the method 200 can be executed by any device with computing function.

[0055] In some databases, different time partitions can be divided according to the time range. This application does not limit the division method of time partitions. For example, a time partition can be data for one day or one hour. For example, time partition A stores data for September 1, and time partition B stores data for September 22. In other words, the original data written on September 1 is stored in time partition A, and the original data written on September 22 is stored in time partition B.

[0056] S210, receiving multiple levels of downsampling strategies defined by a user, wherein each level of downsampling strategy indicates that after the original data is stored in the multiple time partitions and a defined time interval has passed, the data in the multiple time partitions is downsampled according to a defined sampling period.

[0057] In some embodiments, the database may be a cloud database, which runs in a cloud data center as a cloud service, and S210 may be performed by the cloud management platform 110. In this way, the cloud management platform 110 receives the downsampling task creation instruction and sends it to the cloud database.

[0058] For example, a client may send an instruction to the foreground of the execution subject of the embodiment of the present application, and then the background of the execution subject determines multiple levels of downsampling strategies according to the instruction, so that the user can customize the downsampling strategy.

[0059] This application does not limit the specific content of the instruction, as long as the instruction can be used to indicate the downsampling strategy, it is within the scope of this application. As an example, the instruction may include multiple levels of downsampling strategies. In this way, the execution subject can parse the instruction and extract multiple levels of downsampling strategies. For example, the instruction includes a time interval and a sampling period. As another example, the instruction may instruct the execution subject to select multiple levels of pre-set downsampling strategies. For example, the instruction includes an identifier of the downsampling strategy, the background of the execution subject stores the pre-set downsampling strategy, and the execution subject determines that the downsampling strategy corresponding to the identifier is effective.

[0060] For the sake of clear description, the following takes a user-defined two-level downsampling strategy as an example, assuming that the multiple-level downsampling strategy is: for any time partition, starting from the time when the original data is stored in the time partition, after 7 days, the first-level downsampling with a sampling period of 5 minutes is performed, that is, every 5 minutes of data in the original data is sampled into 1 data point; starting from the time when the original data is stored in the time partition, after 28 days, the second-level downsampling with a sampling period of 60 minutes is performed, that is, every 60 minutes of data in the original data is sampled into 1 data point. The above is just an example, and those skilled in the art can understand that the multiple-level downsampling strategy can also be in other forms.

[0061] S220, determining a first downsampling strategy from the multiple levels of downsampling strategies according to a first time interval that has passed after the first original data is stored in the first time partition, wherein the longer the first time interval is, the longer the sampling period of the first downsampling strategy is, and the first time partition is any one of the multiple time partitions.

[0062] For a time partition, the longer the time interval from the time when the original data is stored in the time partition, the longer the sampling period of the data in the time partition. For example, the data written on September 1st is stored in time partition A; after 7 days, on September 8th, the data in time partition A is downsampled at the first level with a sampling period of 5 minutes; after 28 days, on September 29th, the data in time partition A is downsampled at the second level with a sampling period of 60 minutes. It should be noted that the data downsampled on September 29th is further downsampled after the downsampled data on September 8th; after 365 days, on September 1st of the following year, the data in time partition A will be deleted.

[0063] In a possible embodiment, based on the resource pool scheduling strategy, the storage node will perform downsampling on all time partitions that need downsampling according to the corresponding downsampling strategy at a certain moment when the business is idle. Relative to the time point of downsampling execution, the longer the time a time partition stores the original data, the longer the sampling period of the data in the time partition. In other words, for different time partitions, downsampling can be performed according to different sampling periods. For example, assuming that according to the resource pool scheduling policy, a downsampling task is executed every 3 days, and a downsampling task includes multiple levels of downsampling. The time points of the execution of two adjacent downsampling tasks fall on September 26 and September 29, respectively. Time partition B and time partition C store data on September 22 and September 21, respectively, and time partition A and time partition D store data on September 1 and August 31, respectively. Then, when downsampling is performed on September 29, since 7 days have passed and less than 28 days have passed since the original data was written to time partition B and time partition C, the data in time partition B and time partition C will be downsampled at the first level according to a sampling period of 5 minutes, and the data in time partition A and time partition D will be downsampled at the second level according to a sampling period of 60 minutes.

[0064] It can be seen that the user can generate multiple levels of downsampling strategies using a simple instruction, wherein each of the multiple levels of downsampling strategies corresponds to one or more time partitions.

[0065] The original data written to the first time partition is called the first original data. The first time interval is the time interval between the time when the original data of the first time partition is written and the time point when the downsampling is performed. The first downsampling strategy is the downsampling strategy corresponding to the first time interval. For example, assuming that the current time is September 29, and the first time partition is time partition A (the original data is written on September 1), then the first time interval is 28 days, and the corresponding first downsampling strategy is to downsample according to a sampling period of 60 minutes.

[0066] S230, obtaining first data currently stored in the first time partition, where the first data is the first original data, or data obtained by downsampling the first original data at least once.

[0067] The first data is any data in the first time partition. It should be understood that obtaining the first data currently stored in the first time partition indicates that the first data has been "downloaded" in the first time partition. In other words, the first data is not stored in the memory, but in the first time partition of the database. The first data can be original data, data that has been downsampled once, or data that has been downsampled multiple times. In other words, method 200 can further downsample the downsampled data.

[0068] The method 200 provided in the embodiment of the present application can be applied to any database compatible with time partitioning, for example, the method 200 can be applied to a time series database (TSDB), a relational database, a non-relational database, etc. For another example, the method 200 can be applied to a log database.

[0069] S240: Downsample the first data according to the first downsampling strategy to obtain second data.

[0070] The second data is the data after the first data is downsampled. For example, assuming that the data written in time partition A on September 1 is the first data, the first data needs to be downsampled once on September 8. That is, according to the sampling period of 5 minutes, the first data is downsampled once every 5 minutes. For example, assuming that one data point is written every 1 minute, there are 5 data points in 5 minutes. The first downsampling can aggregate every 5 data points into 1 data point, and multiple aggregated data points constitute the second data.

[0071] In some embodiments, the first data is downsampled according to the first downsampling strategy to obtain the second data, including: dividing the first data into multiple groups according to the sampling period of the first sampling strategy; and aggregating the data of each group in the multiple groups according to the aggregation rule of the first downsampling strategy to obtain the second data.

[0072] For example, data with timestamps within a certain sampling period may be grouped according to the timestamps of the first data, and multiple data points (belonging to the first data) in each group may be aggregated into one data point (belonging to the second data).

[0073] Based on the above solution, the first downsampling strategy is adopted to perform grouping, aggregation and compression (aggregate compaction) on the first data, thereby achieving downsampling of the first data and releasing the storage space of the disk, thereby improving the processing speed of the database.

[0074] In some embodiments, the second data includes: at least one of the minimum value (min), the maximum value (max), the sum (sum), the number of data points (count), the average value (mean), the value of the earliest input data point (first), and the value of the latest input data point (last) of each group in the first data.

[0075] Based on the above scheme, the first downsampling strategy is adopted to downsample the first data to obtain the second data. The second data includes at least one aggregate value, so the second data can reflect the change trend of the original data, thereby saving disk storage space and restoring the change trend of the original data, thereby improving the accuracy of downsampling.

[0076] In some embodiments, each group in the first data includes multiple timelines, each timeline includes multiple measurement values, wherein the data of each group in the multiple groups is aggregated, including: sequentially traversing each measurement value in the multiple measurement values ​​in each timeline, and for each measurement value, aggregating the data corresponding to each group.

[0077] Figure 3 This is a schematic diagram of data aggregation provided by the embodiment of the present application. Figure 3 Describe the above scenario.

[0078] A time series 310 may refer to the change of a certain indicator over time. A combination of a measurement (MST) 320, a tag 330, and a field 340 forms a time series.

[0079] The MST 320 is similar to a table in concept, and represents a collection of a series of similar data. For example, an MST 320 may be established for an environmental sensor, and the MST 320 is used to store monitoring data of all environmental sensors.

[0080] The tag 330 can describe the characteristics of the data. The tag 330 is relatively stable and generally does not change over time. For example, information such as the identification of the environmental sensor and the area where the environmental sensor is located can be used as the tag 330 .

[0081] The concept of measurement value 340 is similar to that of column. Measurement value 340 can describe the measurement index of data, and measurement value 340 usually changes over time. For example, data such as temperature, humidity, and pressure detected by an environmental sensor can be called measurement value 340.

[0082] In addition to the label 330 and the measurement value 340 , a data point may also have a timestamp (not shown in the figure). Data with a timestamp in a time interval may be grouped into a group 350 .

[0083] Thus, each group 350 includes multiple timelines 310 , and each timeline 310 includes multiple measurement values ​​340 .

[0084] In some embodiments, a timeline identifier can be assigned to each timeline 310, and there is a size relationship between the timeline identifiers. When aggregating the data in each group 350, the first measurement value 340 of the timeline 350 with the smallest timeline identifier can be aggregated first. It should be noted that the "aggregation" here is aggregation within the group 350, and each group 350 includes multiple data points before aggregation and one data point after aggregation. Subsequently, the second measurement value 340 is aggregated until all measurement values ​​340 of the timeline 350 with the smallest timeline identifier are aggregated. Then, the above-mentioned traversal operation is performed on the timeline 350 with the second smallest timeline identifier, thereby realizing sequential traversal of each measurement value 340 of the multiple measurement values ​​340 in each timeline 350, and for each measurement value 340, the data corresponding to each group 350 is aggregated.

[0085] Based on the above solution, the first data is aggregated in a sequential traversal manner. Compared with random reading, the method provided in the embodiment of the present application saves memory overhead. Figure 2In the embodiment of the present application, the user can set the downsampling strategy according to their own needs, thereby improving the applicability of downsampling. Moreover, the user only needs one instruction, and the background automatically performs maintenance and scheduling, thereby reducing the user's operating and maintenance costs. In this way, when it is necessary to set more levels of downsampling strategies, the data downsampling method provided in the embodiment of the present application can conveniently build a database storage service. On the other hand, the above scheme performs downsampling after the data is dropped into the database, thereby saving memory storage space. Moreover, the above scheme can perform multi-level downsampling, that is, further downsampling is performed based on the downsampled data.

[0086] In some embodiments, the method 200 includes: storing the second data to the first time partition, and deleting the first data.

[0087] The data downsampling method provided in the embodiment of the present application can also perform multi-level downsampling, and replace the data before downsampling with the data after downsampling. The same database can store data that has undergone different degrees of downsampling, and on the basis of ensuring that the data after downsampling can restore the change trend of the original data, it greatly saves the storage space of the disk, thereby improving the processing speed of the database.

[0088] In some embodiments, before S210, method 200 further includes: receiving first original data and generating a timestamp of the first original data; and storing the first original data in the first time partition based on the timestamp.

[0089] The above solution can be understood as the process of storing the original data in the database.

[0090] In some embodiments, there is an integer multiple relationship between the multiple sampling periods corresponding to the multiple levels of downsampling strategies.

[0091] The sampling periods of each downsampling strategy have an integer multiple relationship, which can improve the accuracy of downsampling. "Accuracy" can be understood as that although the amount of data after downsampling is reduced, this does not affect the restoration of the data trend before downsampling.

[0092] For example, according to the multi-level downsampling strategy, the data in the time partition where the original data is written within 7 days is downsampled once with a sampling period of 15 minutes, and the data in the time partition where the original data is written within 28 days but more than 7 days is downsampled once with a sampling period of every 60 minutes, where 60 is an integer multiple of 15.

[0093] Thus, as time goes by, when the time partition in which the original data is written within the 7 days becomes a time partition in which the original data is written within 28 days and exceeds 7 days, it is only necessary to aggregate the four 15-minute data points into one 60-minute data point. For example, the four 15-minute data points are subjected to the minimum value, the maximum value, the sum, the number of statistical points, the average value, the value of the earliest input data point, and the value of the latest input data point, and the obtained aggregated values ​​are used as one 60-minute data point.

[0094] Based on the above scheme, there is an integer multiple relationship between multiple sampling periods corresponding to multiple levels of downsampling strategies, and when further downsampling the downsampled data, aggregation can be performed according to the integer multiple relationship. The processing process of the method provided in the embodiment of the present application is simple and can improve the efficiency of downsampling.

[0095] It should be noted that the present application does not limit the multiple sampling periods corresponding to the multiple downsampling strategies to have an integer multiple relationship. For example, the multiple downsampling strategies may also be to sample the data in the time partition for writing the original data within 7 days every 3 minutes, and sample the data in the time partition for writing the original data within 28 days every 5 minutes. In this way, when the time partition for writing the original data within 7 days becomes the time partition for writing the original data within 28 days, 5 3-minute data points can be aggregated into 3 5-minute data points. Obviously, if there is no integer multiple relationship between the multiple sampling periods corresponding to the multiple downsampling strategies, the data downsampling method provided in the embodiment of the present application can also be implemented. However, in the process of aggregation, the above scheme may need to average or weighted average between multiple data points that have been downsampled, or consider more 3-minute data points for trend fitting, so that the data accuracy after further downsampling is low.

[0096] It should be noted that the above only takes two levels of downsampling strategies as an example, and those skilled in the art can understand that there can be more levels of downsampling strategies. For example, the first-level downsampling strategy can be to downsample the data in the time partition 3 days after the original data is written and within 30 days with a sampling period of 5 minutes, the second-level downsampling strategy can be to downsample the data in the time partition 30 days after the original data is written and within 90 days with a sampling period of 15 minutes, and the third-level downsampling strategy can be to downsample the data in the time partition 90 days after the original data is written and have a sampling period of 60 minutes.

[0097] The method provided in the embodiment of the present application realizes "multi-level" downsampling. Compared with "single-level" downsampling, multi-level downsampling further releases the storage space of the disk, thereby improving the processing speed of the database.

[0098] In some embodiments, for the above-mentioned solution of "replacing the data before downsampling with the data after downsampling", a transaction may be first opened for the first time partition to avoid reading and writing operations on the first time partition. The event of the file corresponding to the first data is written to the log, and the file is renamed so that the file corresponding to the first data is not visible to the user. Then, the event of the file corresponding to the second data is written to the log, and the file is renamed so that the file corresponding to the second data is visible to the user. Finally, the file corresponding to the first data is deleted, and the transaction is ended to restore the reading and writing of the first time partition.

[0099] In this way, if an exception occurs during the execution of the above solution, the current log can be replayed after the database is restarted. The database can execute the remaining operations in the above solution based on the status of the current end event.

[0100] The downsampled data can also be stored in multiple clusters according to different downsampling strategies. However, in this solution, each new level of downsampling strategy requires more storage space, and users need to rewrite the code related to database storage. In addition, each cluster needs to interact with each other (for example, cluster B obtains cluster A's data for downsampling), which reduces the efficiency of downsampling.

[0101] The data downsampling method provided in the embodiment of the present application can perform downsampling after the data is dropped to the disk in the database, saving the storage space of the memory. In addition, the data downsampling method provided in the embodiment of the present application can also perform multi-level downsampling, replacing the data before downsampling with the data after downsampling, which can save the storage space of the disk and improve the processing speed of the database compared to the solution of saving all data. In addition, when it is necessary to set more levels of downsampling strategies, the data downsampling method provided in the embodiment of the present application can easily build a database storage service and further reduce the storage space of the disk.

[0102] The following further describes the method embodiment of the present application in conjunction with the exemplary architecture of the database.

[0103] Figure 4 4 is a schematic block diagram of a database 400 provided in an embodiment of the present application. It should be noted that the architecture of the database 400 is only schematic and does not constitute a limitation to the present application.

[0104] The database 400 may include a query unit 410, a storage unit 420 and a metadata management cluster 430. The query unit 410 and the storage unit 420 may be distributed in multiple nodes. The metadata management cluster 430 may manage the status and downsampling strategy of each node.

[0105] The query unit 410 may include a structured query language (SQL) parser, a semantic analyzer, an optimizer, a runtime instruction library, and the like.

[0106] Among them, the SQL parser and semantic analyzer can parse and compile the SQL statements written by the user into executable instructions. The optimizer can be used for query optimization, such as logical optimization or physical optimization. The runtime instruction library is used to provide some codes when the computer program is running.

[0107] The storage unit 420 may include a parser, a directed acyclic graph (DAG) generator, a chunk reader, a data interface, a data storage area, and the like.

[0108] A time partition has a separate block reader, a data interface, and a data storage area. The block reader can read data files in the data storage area through the data interface.

[0109] The metadata management cluster 430 can manage the downsampling policy, and the user can maintain the downsampling policy through instructions. The level of the downsampling policy corresponds to the level of the retention policy (RP).

[0110] There can be multiple data points in an MST, one RP can correspond to multiple MSTs, and one database can have multiple RPs. In other words, the downsampling strategy can correspond to multiple MSTs.

[0111] The downsampling strategy can be implemented based on the original RP information in the database. The original RP information includes the names of the MST and the measurement value, as well as the types of the MST and the measurement value. The original RP information also includes the retention time (duration), which specifies the expiration time of the data. If a data point exceeds the retention time, the data point will be deleted. Downsampling data can add an aggregation operator set (calls) based on the original RP information. The aggregation operator set includes the type of the measurement value and the name of the aggregation operation taken on the measurement value.

[0112] The storage unit 420 may periodically send a request to the metadata management cluster 430 to determine the time slices that need to perform downsampling, obtain an updated downsampling strategy, and then generate a DAG task according to the downsampling strategy. According to the DAG task, the data file may be downsampled, and then the data file before downsampling may be replaced with the data file after downsampling.

[0113] Specifically, before sending a request to the metadata management cluster 430 , the node timer of the storage unit 420 may set the state of the downsampling strategy stored in the engine of the storage unit 420 to be inactive. The engine may be used to execute a DAG task in the storage unit 420 .

[0114] After the storage unit 420 obtains the downsampling strategy from the metadata management cluster 430, if the downsampling strategy already exists in the engine of the storage unit 420, the state of the downsampling strategy can be set to activated. If the downsampling strategy is not in the engine of the storage unit 420, the downsampling strategy can be first input into the DAG generator, and the DAG generator can create an MST-level query mode (schema) and DAG tasks based on the MST and measurement value information of the downsampling strategy. After that, the downsampling strategy is added to the engine's map, and the state of the downsampling strategy is set to activated.

[0115] After the storage unit 420 acquires all downsampling strategies, the execution of the DAG task begins.

[0116] Figure 5 is a schematic block diagram of a data downsampling device 500 provided in an embodiment of the present application. The data downsampling device 500 includes a receiving module 510, a processing module 520 and a downsampling module 530. The embodiments of the above three modules are described later and are not described here. The downsampling module 530 includes a reading module 610, an execution module 620 and a writing module 630.

[0117] The downsampling module 530 can be used to execute a DAG task, which includes three operations, namely, obtaining data, downsampling data, and writing a new file. Among them, the reading module 610 can execute the operation of obtaining data, the execution module 620 can execute the operation of downsampling data, and the writing module 630 can execute the operation of writing a new file.

[0118] Figure 6 2 is a schematic diagram of a data downsampling method 200 provided in an embodiment of the present application.

[0119] See also Figure 6, a DAG task can correspond to a time slice, and each DAG task can operate concurrently, that is, when one DAG is acquiring the first data, another DAG can downsample the third data. Therefore, the reading module 610, the execution module 620, and the writing module 630 can also operate in parallel. In some embodiments, when the reading module 610 reads the first data, the execution module 620 can downsample the data of other time slices. In other embodiments, the reading module 610 can read the first data and the data of other time slices at the same time.

[0120] It should be noted that Figure 6 The storage unit 420 in the example may be any node of the storage unit 420. The first time slice and the second time slice may be in a node of the storage unit 420 or in different nodes of the storage unit 420. The reading module 610, the executing module 620, and the writing module 630 may be in a node of the storage unit 420 or in different nodes of the storage unit 420.

[0121] Figure 7 It is a schematic diagram of the first data and the second data provided in the embodiment of the present application. Figure 7 This is merely an example and does not constitute a limitation to the present application.

[0122] Figure 7 The above figure in is a schematic diagram of the first data. The first data may include three files, namely file 1, file 2 and file 3. Each file includes multiple timelines, wherein timeline identifier 1 is an identifier of timeline 1, and timeline 1 includes measurement value 1 and measurement value 2. Timeline identifier 2 is an identifier of timeline 2, and timeline 2 includes measurement value 1 and measurement value 2.

[0123] Figure 7 The following figure is a schematic diagram of the second data. The second data includes three files, namely file 1, file 2 and file 3. The three files in the second data are obtained by downsampling the three files of the first data. According to the sampling period of the downsampling strategy, each measurement value can be aggregated.

[0124] Exemplarily, the aggregation performed on the measurement value 1 may be summation; and the aggregation performed on the measurement value 2 may be maximum value.

[0125] It is understandable that more types of aggregation operations can be performed on each measurement value. For example, the aggregation of multiple data in measurement value 1 can also be to take the minimum value, take the maximum value, calculate the number of data points, calculate the average value, take the value of the earliest input data point and / or take the value of the latest input data point. In this case, Figure 7 In the "Measurement value 1 sum" position in the figure below, store more aggregate values.

[0126] The following describes the syntax of the data downsampling method provided by the present application in the implementation. The following syntax is merely exemplary and does not constitute a limitation to the present application.

[0127] As a possible embodiment, the above method can be implemented using RP. For example, when a user creates a downsampling policy, it can be defined which RP of which database the downsampling policy is applied to, for example, the database name and the RP name can be entered. A retention time (duration) can be defined, for example, a retention time of 7 days means that the data in each metric under the downsampling policy expires after 7 days.

[0128] When the user creates a downsampling strategy, the sampling level interval (sample interval) can also be defined. For example, if the sampling level interval is (1 day, 2 days), it means that the first level of downsampling will be applied after 1 day. The second level of downsampling will be applied after 2 days. The sampling period (time interval) can be defined. For example, if the sampling period is (1 minute, 3 minutes), it means that the sampling period of the first level of downsampling is 1 minute, and the sampling period of the second level of downsampling is 3 minutes.

[0129] The sampling level interval corresponds to the sampling period. The sampling level interval includes a number of downsampling strategies. It is necessary to specify the sampling period of each downsampling strategy in these downsampling levels.

[0130] In some embodiments, there is an integer multiple relationship between multiple sampling periods corresponding to multiple downsampling strategies. In the above grammar, it can be understood that the values ​​in the brackets defining the sampling period are in an integer multiple relationship. For example, the sampling period can be defined as (2 minutes, 8 minutes, 40 minutes).

[0131] When users create a downsampling strategy, they can also define the measurement value to be aggregated, the data type, and the corresponding aggregation operator.

[0132] In some embodiments, the user can display the downsampling strategy through instructions.

[0133] In some embodiments, the user can delete (drop) the downsampling strategy through instructions.

[0134] Some embodiments of database storage are described above, and some embodiments of database query are described in detail below.

[0135] Assuming that the downsampling strategy has two levels, combined with the above syntax, the retention time is defined as 365 days, the sampling level interval is (7 days, 28 days), and the sampling period is (15 minutes, 60 minutes). In this way, for a time partition, starting from the time when the original data is written to the time partition, when the time partition has passed 7 days but not more than 28 days, the data in the time partition can be downsampled with a sampling period of 15 minutes; when the time partition has passed 28 days but not more than 365 days, the data in the time partition can be downsampled with a sampling period of 60 minutes; when the time partition has passed 365 days, the data in the time partition can be deleted.

[0136] In some embodiments, the method 200 also includes: receiving a query instruction, the query instruction is used to query a first aggregation result after aggregating the second original data of the second time partition based on the first aggregation rule, wherein the second time partition is any one of the multiple time partitions, and the data currently stored in the second time partition is obtained by aggregating the second original data based on at least one aggregation rule; determining a second aggregation rule according to the first aggregation rule and the at least one aggregation rule; and aggregating the data currently stored in the second time partition according to the second aggregation rule to obtain the first aggregation result.

[0137] The second time partition may be the same as the first time partition or different from the first time partition. The second original data is the original data written into the second time partition. The first aggregation rule is one of the above aggregation rules. For example, the first aggregation rule may be count. The following description is made by taking count as an example.

[0138] When a user queries, for the time slices that have completed downsampling, the storage unit 420 can rewrite the query statement according to the time partitions to feed back the correct query result to the user.

[0139] For example, suppose that a user wants to query data within the past 90 days, and the data is divided into 1 group every 60 minutes, and the number of data points (count) in each group is returned to the user.

[0140] The embodiment of the present application can rewrite the instruction sent by the user and perform different aggregation operations on different time partitions. Specifically, for the time partitions where the original data is written within 7 days, since downsampling has not been performed, the original data in these time partitions can be aggregated, divided into 1 group every 60 minutes, and the number of data points in each group is counted to obtain the count aggregation value. For the data in the second time partition where the original data is written for more than 7 days and no more than 28 days, since these data have been downsampled to the first level, that is, the original data has been aggregated according to at least one aggregation rule (such as count, sum, means, etc.), each data point contains a first count aggregation value of 15 minutes (for example, there is a data point of original data every 1 minute, and after aggregation according to at least one aggregation rule, the first count aggregation value of each data point is 15). Therefore, when at least one of the above aggregation rules includes count, and the first aggregation rule is count, the second aggregation rule can be determined to be sum, that is, every 4 data points can be divided into 1 group, and the first count aggregation values ​​contained in the 4 data points in each group can be added (summed) to obtain the second count aggregation value (for example, there is a data point of original data every 1 minute, and the second count aggregation value is 60 when the 4 first count aggregation values ​​are added). For data written in time partitions that are more than 28 days but not more than 90 days, since these data have been downsampled twice, each data point contains exactly 60 minutes of data points, so the count aggregation value contained in each data point can be extracted. Then, the count aggregation values ​​obtained by the above three time partitions are returned to the user.

[0141] If full query is used, data in time partitions where the original data is written for more than 7 days but not more than 28 days and data in time partitions where the original data is written for more than 28 days but not more than 90 days still need to be aggregated. The solution of rewriting query statements provided in this application can perform simpler processing on the downsampled data, thereby saving computing resources.

[0142] The downsampled data can also be stored in multiple clusters according to different downsampling strategies. However, in this solution, the user needs to rewrite the query code for each additional downsampling strategy. In addition, a materialized view needs to be added for each additional downsampling strategy, which further occupies storage space.

[0143] Based on the above solution for rewriting query statements, users do not need to modify query statements, which facilitates users to build database storage and query services. Users will not perceive these processes when querying, which improves user experience. In addition, the above solution for rewriting query statements reduces the occupancy of materialized views, thereby further saving storage space.

[0144] According to the method 200 and the query rewriting solution provided in the embodiment of the present application, more than 90% of computing resources and storage resources can be saved.

[0145] Figure 8 It is a schematic diagram of data downsampling and query provided in an embodiment of the present application.

[0146] See also Figure 8 If the user wishes to perform query 810, the user can obtain the target file when the lock is open.

[0147] Request 820 may be sent by storage unit 420 to a certain RP. When RP receives request 820, it may open a transaction for the MST corresponding to the RP. At this time, the lock is closed, and the user cannot query the MST corresponding to the RP. Subsequently, the MST corresponding to the RP is processed, and the first data is downsampled to the second data. The file of the first data is renamed to make the first data invisible. The file of the second data is renamed to make the second data visible. The file processing is completed, the transaction is ended, and the lock is unlocked. After that, if the user wants to perform query 810, the file of the second data can be queried.

[0148] The process of processing the file will be saved in the log, and if a failure occurs, the operation of restarting 830 can be triggered. For example, the unprocessed files can be deleted according to the log and reprocessed.

[0149] The method embodiments of the present application are described in detail above. Figure 5 , Figures 9 to 11 The device embodiment of the embodiment of the present application is described. The device embodiment corresponds to the method embodiment, so the parts that are not described in detail can refer to the previous method embodiments. The device can implement any possible implementation method of the above method.

[0150] Continue to see Figure 5The data downsampling device 500 includes: a receiving module 510, which is used to receive multiple levels of downsampling strategies defined by a user, wherein each level of downsampling strategy indicates that after the original data is stored in the multiple time partitions and a defined time interval has passed, the data in the multiple time partitions are downsampled according to a defined sampling period; a processing module 520, which is used to determine a first downsampling strategy from the multiple levels of downsampling strategies according to a first time interval that has passed after the first original data is stored in the first time partition, wherein the longer the first time interval is, the longer the sampling period of the first downsampling strategy is, and the first time partition is any one of the multiple time partitions; a downsampling module 530, which is used to obtain the first data currently stored in the first time partition, wherein the first data is the first original data, or data that has been downsampled at least once for the first original data, and is used to downsample the first data according to the first downsampling strategy to obtain the second data.

[0151] In a possible embodiment, the downsampling module 530 is further configured to store the second data in the first time partition and delete the first data.

[0152] In a possible embodiment, the receiving module 510 is further used to: receive the first original data and generate a timestamp of the first original data; and store the first original data in the first time partition based on the timestamp.

[0153] In a possible embodiment, the receiving module 510 is also used to: receive a query instruction, which is used to query a first aggregation result after aggregating the second original data of the second time partition based on the first aggregation rule, wherein the second time partition is any one of the multiple time partitions, and the data currently stored in the second time partition is obtained by aggregating the second original data based on at least one aggregation rule; determine a second aggregation rule according to the first aggregation rule and the above-mentioned at least one aggregation rule; aggregate the data currently stored in the second time partition according to the second aggregation rule to obtain a first aggregation result.

[0154] Specifically, the specific implementation of various operations of the data downsampling method performed by the data downsampling device 500 can refer to the description of the relevant contents in the above method embodiment, which will not be repeated here.

[0155] Among them, the receiving module 510, the processing module 520 and the down-sampling module 530 can all be implemented by software, or can be implemented by hardware. Exemplarily, the implementation of the receiving module 510 is introduced below by taking the receiving module 510 as an example. Similarly, the implementation of the processing module 520 and the down-sampling module 530 can refer to the implementation of the receiving module 510.

[0156] As an example of a software functional unit, the receiving module 510 may include code running on a computing instance. Among them, the computing instance may include at least one of a physical host (computing device), a virtual machine, and a container. Further, the above-mentioned computing instance may be one or more. For example, the receiving module 510 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed in the same region (region) or in different regions. Furthermore, the multiple hosts / virtual machines / containers used to run the code may be distributed in the same availability zone (AZ) or in different AZs, each AZ including a data center or multiple data centers with similar geographical locations. Among them, usually a region may include multiple AZs.

[0157] Similarly, multiple hosts / virtual machines / containers used to run the code can be distributed in the same virtual private cloud (VPC) or in multiple VPCs. Usually, a VPC is set up in a region. For cross-region communication between two VPCs in the same region and between VPCs in different regions, a communication gateway needs to be set up in each VPC to achieve interconnection between VPCs through the communication gateway.

[0158] As an example of a hardware functional unit, the receiving module 510 may include at least one computing device, such as a server, etc. Alternatively, the receiving module 510 may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL) or any combination thereof.

[0159] The multiple computing devices included in the receiving module 510 can be distributed in the same region or in different regions. The multiple computing devices included in the receiving module 510 can be distributed in the same AZ or in different AZs. Similarly, the multiple computing devices included in the receiving module 510 can be distributed in the same VPC or in multiple VPCs. The multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0160] It should be noted that, in other embodiments, the receiving module 510 can be used to execute any step in the data downsampling method, the processing module 520 can be used to execute any step in the data downsampling method, and the downsampling module 530 can be used to execute any step in the data downsampling method. The steps that the receiving module 510, the processing module 520 and the downsampling module 530 are responsible for implementing can be specified as needed. The receiving module 510, the processing module 520 and the downsampling module 530 respectively implement different steps in the data downsampling method to achieve all the functions of the data downsampling device 500.

[0161] The present application also provides a computing device 900. Fig. 9 As shown, the computing device 900 includes: a bus 902, a processor 904, a memory 906, and a communication interface 908. The processor 904, the memory 906, and the communication interface 908 communicate through the bus 902. The computing device 900 can be a server or a terminal device. It should be understood that the present application does not limit the number of processors and memories in the computing device 900.

[0162] The bus 902 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig. 9 The bus 902 may include a path for transmitting information between various components of the computing device 900 (eg, the memory 906, the processor 904, and the communication interface 908).

[0163] The processor 904 may include any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0164] The memory 906 may include a volatile memory, such as a random access memory (RAM). The memory 906 may also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD), or a solid state drive (SSD).

[0165] The memory 906 stores executable program codes, and the processor 904 executes the executable program codes to respectively implement the functions of the aforementioned receiving module 510, processing module 520 and downsampling module 530, thereby implementing the data downsampling method. That is, the memory 906 stores instructions for executing the data downsampling method.

[0166] The communication interface 908 uses a transceiver module such as, but not limited to, a network interface card or a transceiver to implement communication between the computing device 900 and other devices or communication networks.

[0167] The embodiment of the present application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smart phone.

[0168] like Fig.10 As shown, the computing device cluster includes at least one computing device 900. The memory 906 in one or more computing devices 900 in the computing device cluster may store the same instructions for executing the data downsampling method.

[0169] In some embodiments, the memory 906 of one or more computing devices 900 in the computing device cluster may also store partial instructions for executing the data downsampling method. In other words, the combination of one or more computing devices 900 may jointly execute instructions for executing the detection data downsampling method.

[0170] It should be noted that the memory 906 in different computing devices 900 in the computing device cluster may store different instructions, which are respectively used to execute part of the functions of the data downsampling device 500. That is, the instructions stored in the memory 906 in different computing devices 900 may implement the functions of one or more modules among the receiving module 510, the processing module 520 and the downsampling module 530.

[0171] In some embodiments, one or more computing devices in the computing device cluster may be connected via a network, which may be a wide area network or a local area network, etc.

[0172] Fig.11 A possible implementation method is proposed. Fig.11 As shown, two computing devices 900A and 900B are connected via a network. Specifically, the network is connected via a communication interface in each computing device. In this type of possible implementation, the memory 906 in the computing device 900A stores instructions for executing the functions of the receiving module 510 and the processing module 520. At the same time, the memory 906 in the computing device 900B stores instructions for the functions of the downsampling module 530.

[0173] Fig.11 The connection mode between the computing device clusters shown may be that considering that the data downsampling method provided in the present application requires a large number of multi-stage downsampling, it is considered to hand over the functions implemented by the processing module 520 to the computing device 900B for execution.

[0174] It should be understood that Fig.11 The functions of the computing device 900A shown in FIG. 9A may also be completed by multiple computing devices 900. Similarly, the functions of the computing device 900B may also be completed by multiple computing devices 900.

[0175] In some embodiments, the memory 906 of one or more computing devices 900 in the computing device cluster may also store partial instructions for executing the data downsampling method. In other words, the combination of one or more computing devices 900 may jointly execute instructions for executing the data downsampling method.

[0176] The embodiment of the present application also provides a computer program product including instructions. The computer program product may be a software or program product including instructions that can be run on a computing device or stored in any available medium. When the computer program product is run on at least one computing device, the at least one computing device executes the above-mentioned data downsampling method.

[0177] The embodiment of the present application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that can be stored by the computing device or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk). The computer-readable storage medium includes instructions that instruct the computing device to perform the above-mentioned data downsampling method.

[0178] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0179] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0180] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0181] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0182] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0183] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage media include: various media that can store program codes, such as USB flash drives, mobile hard disks, read-only memories (ROM), random access memories (RAM), magnetic disks or optical disks.

[0184] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A data downsampling method, characterized in that: The method is applied to a database, the database includes multiple time partitions, and the multiple time partitions are used to store data in different time periods. The method includes: Receiving multiple levels of downsampling strategies defined by a user, wherein each level of downsampling strategy indicates that after the original data is stored in the multiple time partitions and a defined time interval has passed, the data in the multiple time partitions is downsampled according to a defined sampling period; Determine a first downsampling strategy from the multiple levels of downsampling strategies according to a first time interval after the first original data is stored in the first time partition, wherein the longer the first time interval is, the longer the sampling period of the first downsampling strategy is, and the first time partition is any one of the multiple time partitions; Acquire first data currently stored in the first time partition, where the first data is the first original data, or data obtained by downsampling the first original data at least once; The first data is downsampled according to the first downsampling strategy to obtain second data.

2. The method according to claim 1, characterized in that The method further comprises: The second data is stored in the first time partition, and the first data is deleted.

3. The method according to claim 1 or 2, characterized in that: Before receiving the multiple levels of downsampling strategies defined by the user, the method further includes: receiving the first original data, and generating a timestamp of the first original data; Based on the timestamp, the first original data is stored in the first time partition.

4. The method according to any one of claims 1 to 3, characterized in that The downsampling the first data according to the first downsampling strategy to obtain second data includes: dividing the first data into a plurality of groups according to a sampling period of the first downsampling strategy; According to the aggregation rule of the first downsampling strategy, the data of each group in the multiple groups is aggregated to obtain the second data.

5. The method according to claim 4, characterized in that The second data includes: at least one of the minimum value, maximum value, sum, number of data points, average value, value of the earliest input data point, and value of the latest input data point of each group of data in the first data.

6. The method according to any one of claims 1 to 5, characterized in that The method further comprises: Receive a query instruction, the query instruction being used to query a first aggregation result after aggregating second original data of a second time partition based on a first aggregation rule, wherein the second time partition is any one of the multiple time partitions, and data currently stored in the second time partition is obtained by aggregating the second original data based on at least one aggregation rule; determining a second aggregation rule according to the first aggregation rule and the at least one aggregation rule; The data currently stored in the second time partition is aggregated according to the second aggregation rule to obtain the first aggregation result.

7. A data downsampling device, characterized in that: The device is applied to a database, the database includes multiple time partitions, the multiple time partitions are used to store data in different time periods, and the device includes: A receiving module, configured to receive multiple levels of downsampling strategies defined by a user, wherein each level of downsampling strategy indicates that after the original data is stored in the multiple time partitions and a defined time interval has passed, the data in the multiple time partitions are downsampled according to a defined sampling period; a processing module, configured to determine a first downsampling strategy from the multiple levels of downsampling strategies according to a first time interval after the first original data is stored in the first time partition, wherein the longer the first time interval is, the longer the sampling period of the first downsampling strategy is, and the first time partition is any one of the multiple time partitions; A downsampling module is used to obtain first data currently stored in the first time partition, where the first data is the first original data, or data that has been downsampled at least once for the first original data, and is used to downsample the first data according to the first downsampling strategy to obtain second data.

8. The device according to claim 7, characterized in that The downsampling module is also used for: The second data is stored in the first time partition, and the first data is deleted.

9. The device according to claim 7 or 8, characterized in that The receiving module is also used for: receiving the first original data, and generating a timestamp of the first original data; Based on the timestamp, the first original data is stored in the first time partition.

10. The device according to any one of claims 7 to 9, characterized in that The downsampling the first data according to the first downsampling strategy to obtain second data includes: dividing the first data into a plurality of groups according to a sampling period of the first downsampling strategy; According to the aggregation rule of the first downsampling strategy, the data of each group in the multiple groups is aggregated to obtain the second data.

11. The device according to claim 13, characterized in that The second data includes: at least one of the minimum value, maximum value, sum, number of data points, average value, value of the earliest input data point, and value of the latest input data point of each group of data in the first data.

12. The device according to any one of claims 7 to 11, characterized in that The receiving module is also used for: Receive a query instruction, the query instruction being used to query a first aggregation result after aggregating second original data of a second time partition based on a first aggregation rule, wherein the second time partition is any one of the multiple time partitions, and data currently stored in the second time partition is obtained by aggregating the second original data based on at least one aggregation rule; determining a second aggregation rule according to the first aggregation rule and the at least one aggregation rule; The data currently stored in the second time partition is aggregated according to the second aggregation rule to obtain the first aggregation result.

13. A computing device cluster, characterized in that: comprising at least one computing device, each computing device comprising a processor and a memory; The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster executes the method according to any one of claims 1 to 6.

14. A computer program product comprising instructions, characterized in that When the instructions are executed by a computing device cluster, the computing device cluster is caused to perform the method according to any one of claims 1 to 6.

15. A computer-readable storage medium, characterized in that: The method comprises computer program instructions. When the computer program instructions are executed by a computing device cluster, the computing device cluster performs the method according to any one of claims 1 to 6.

Images