Method and device for pre-aggregating time series data

Abstract

The invention relates to the field of computers, and provides a method and a device for pre-aggregating time series data. The method comprises the following steps: acquiring at least one time sequence data from a first device; the first device is a generation device of at least one time sequence data, or the first device is a node device of a time sequence database, and any time sequence data in the at least one time sequence data comprises an identifier; determining a pre-polymerization method according to the identifier of the at least one time sequence data, wherein the pre-polymerization method and the identifier of the at least one time sequence data have a preset binding relationship; processing the at least one time sequence data by using a pre-aggregation method to generate pre-aggregated data; and when the occupancy rate of the memory meets a trigger condition, writing the pre-aggregated data and the at least one time sequence data into a disk. According to the method, the time sequence data and the pre-aggregated data can be uniformly distributed, and the utilization rate of the storage space is improved.

Description

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office on August 2, 2023, with application number 202310966765.X and application name “Prepolymerization Method and Prepolymerization Device”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of computers, and in particular to a method and apparatus for pre-aggregating time series data. Background Art

[0003] Time series data refers to a series of data that is generated over time. With the advent of the fifth generation of mobile communication technology (5 th The continuous development of 5G mobile communication technology and the Internet of Things (IoT) has led to explosive growth in data volumes. Time series data has a wide range of applications, including IoT, the Internet of Vehicles, the Industrial Internet, and application performance monitoring. In these scenarios, time series data can record key information such as device operating status, operational data, and monitoring data. By analyzing and processing time series data, companies can predict failures, optimize production, and provide support for decision-making.

[0004] Time series data is characterized by its high frequency of data generation and continuous high concurrent writes. These characteristics make it time-consuming to process. Pre-aggregation is a method to address this time-consuming processing of time series data. This method pre-aggregates the time series data during the writing process to generate pre-aggregated data. During querying, the pre-aggregated data is used to perform secondary aggregation on the time series data, which can improve the efficiency of time series data queries.

[0005] When pre-aggregating time series data, it is necessary to manually set the pre-aggregation time range. However, the generation frequency of time series data in different time periods is different. Manually setting the pre-aggregation time range is difficult to ensure that the data volume of time series data and pre-aggregated data in different time ranges is the same. The uneven distribution of time series data and pre-aggregated data will lead to a decrease in storage space utilization. Summary of the Invention

[0006] Embodiments of the present application provide a method, apparatus, computer-readable storage medium, and computer program product for pre-aggregating time series data, which can evenly distribute time series data and pre-aggregated data and improve storage space utilization.

[0007] In the first aspect, an embodiment of the present application provides a method for pre-aggregating time series data, the execution subject of the method can be a server or a chip applied to a server, and the following description is made by taking the execution subject being a server as an example. The method includes: obtaining at least one time series data from a first device, the first device is a device for generating at least one time series data, or the first device is a node device of a time series database, and any one of the at least one time series data includes an identifier; determining a pre-aggregation method based on the identifier of at least one time series data, and the pre-aggregation method has a preset binding relationship with the identifier of at least one time series data; using the pre-aggregation method to process at least one time series data to generate pre-aggregated data; writing the pre-aggregated data and at least one time series data into a memory; when the memory occupancy rate meets the trigger condition, writing the pre-aggregated data and at least one time series data into a disk.

[0008] The server can obtain time series data directly from the device that generates the time series data, or it can obtain time series data from the node device in the server cluster. Time series data generally includes an identifier (also known as an "indicator"). Based on the binding relationship between the pre-aggregation method and the identifier of the time series data, the server can determine the pre-aggregation method to be used. Subsequently, the server processes the time series data based on the determined pre-aggregation method, and writes the pre-aggregation results and the time series data to the memory. When the memory does not meet the trigger conditions, the server can continue to obtain time series data and pre-aggregate the time series data. When the memory meets the trigger conditions, the server can write the time series data and pre-aggregation data currently stored in the memory to the disk. Since the trigger conditions remain unchanged, the amount of time series data and pre-aggregation data written to the disk at different times is the same, thereby ensuring that the time series data and pre-aggregation data are evenly distributed, and improving the storage space utilization of time series data and pre-aggregation data.

[0009] Optionally, writing the pre-aggregated data and the at least one time series data to a disk includes: writing the at least one time series data to a first file in the disk; and writing the pre-aggregated data to a second file in the disk.

[0010] The pre-aggregation method may include multiple pre-aggregation functions, and different pre-aggregation functions have different functions. By using multiple pre-aggregation functions, comprehensive pre-aggregation data can be obtained, which is conducive to accelerating the query efficiency of time series data. However, the amount of pre-aggregation data generated by multiple pre-aggregation functions may be very large. For a time series database with a built-in pre-aggregation function, it usually writes the time series data and the pre-aggregation data into the same file, which will cause the data volume of the file to increase, and then cause the query efficiency of the time series data to decrease. Therefore, the time series database with a built-in pre-aggregation function can usually only preset a small number of pre-aggregation functions, and it is difficult to implement a custom pre-aggregation function. In this embodiment, the pre-aggregation data and the time series data are written into different files, so that the pre-aggregation data does not affect the file size of the time series data. Under the premise of not affecting the query efficiency of the time series data, more pre-aggregation functions can be applied to pre-aggregate the time series data to realize the function of the custom pre-aggregation function.

[0011] Optionally, the first file and the second file are stored in the same directory.

[0012] The first file and the second file are stored in the same directory. The server does not need to search across directories for the second file corresponding to the first file, which can improve the query efficiency of time series data.

[0013] Optionally, writing the pre-aggregated data and the at least one time series data to the disk includes: writing the pre-aggregated data and the at least one time series data to the disk in parallel.

[0014] Time series data is characterized by rapid data generation and continuous high-concurrency writes. If the time series data and pre-aggregated data in memory cannot be written to disk in a timely manner, insufficient memory may occur, and new time series data cannot be saved in a timely manner. This embodiment uses multi-threaded parallel writing. Even if the writing of one of the pre-aggregated data and the time series data is blocked, the writing of the other data can continue. The server can quickly free up memory space and avoid the loss of new time series data due to insufficient memory.

[0015] Optionally, the at least one time series data includes multiple time series data, the multiple time series data belong to the same type of time series data, and the multiple time series data are stored in the same memory block of the memory.

[0016] Different types of time series data may use different pre-aggregation methods. By storing the same type of time series data in the same memory block, the server no longer needs to determine whether the time series data matches the pre-aggregation method when pre-aggregating the time series data in the memory block, thereby improving the efficiency of pre-aggregation.

[0017] Optionally, the triggering condition includes: the memory occupancy is greater than or equal to the occupancy threshold; or, the occupancy of a memory block in the memory is greater than or equal to the occupancy threshold, and the memory block is used to store pre-aggregated data and at least one time series data.

[0018] Writing data from memory to disk incurs other time overheads besides the data writing time. When the memory or memory block occupancy rate is greater than or equal to the occupancy threshold, a large amount of data to be written to disk has accumulated. Writing a large amount of data to disk at once can spread out other time overheads and improve the efficiency of writing time series data to disk. The server can free up memory space as quickly as possible to avoid congestion caused by acquiring new time series data.

[0019] Optionally, before using the pre-aggregation method to process at least one time series data, it also includes: receiving first indication information, the first indication information is used to indicate a binding relationship; saving the identifier of at least one time series data and the identifier of the pre-aggregation method in a metadata module according to the first indication information; determining the pre-aggregation method according to the identifier of at least one time series data, including: determining the identifier of the pre-aggregation method from the metadata module according to the identifier of at least one time series data; determining the pre-aggregation method according to the identifier of the pre-aggregation method.

[0020] Different types of time series data may use different pre-aggregation methods. The user can instruct the server to pre-bind the time series data with the matching pre-aggregation method through the first indication information. The server can save the identifier of the time series data and the identifier of the pre-aggregation method in the metadata module according to the first indication information. In this way, after obtaining the time series data, the server can directly determine the pre-aggregation method that matches the time series data based on the binding relationship saved in the metadata module, and process the time series data according to the pre-aggregation method without waiting for the user's instructions, thereby improving the pre-aggregation efficiency of the time series data.

[0021] Optionally, after writing the pre-aggregated data and at least one time series data to the disk, the method further includes: receiving second indication information, the second indication information being used to indicate the release of the binding relationship; and deleting the identifier of the pre-aggregation method stored in the metadata module according to the second indication information.

[0022] For the same type of time series data, the pre-aggregation methods used in different scenarios may differ. After processing the current time series data using the pre-aggregation method, unbinding the relationship allows for the use of the new pre-aggregation method to process new time series data. Therefore, this embodiment can flexibly adapt to different time series data pre-aggregation scenarios.

[0023] Optionally, the method further includes: receiving third indication information, where the third indication information is used to indicate cancellation of the pre-aggregation method; and deleting the pre-aggregation method stored in the metadata module according to the third indication information.

[0024] In some scenarios, there are a large number of pre-aggregation methods. Before pre-aggregation processing, it is necessary to query the pre-aggregation methods with binding relationships from the large number of pre-aggregation methods. Unregistering the unbound pre-aggregation methods can improve the query efficiency of pre-aggregation methods.

[0025] In a second aspect, an embodiment of the present application provides a device for pre-aggregating time series data. The device may include an input unit, a processing unit, and an output unit;

[0026] The input unit is used to: obtain at least one time series data from a first device, where the first device is a device that generates at least one time series data, or the first device is a node device of a time series database, and any one of the at least one time series data includes an identifier;

[0027] The processing unit is configured to: determine a pre-aggregation method according to an identifier of at least one time series data, wherein the pre-aggregation method and the identifier of at least one time series data have a preset binding relationship; process the at least one time series data using the pre-aggregation method to generate pre-aggregated data; and write the pre-aggregated data and the at least one time series data into a memory;

[0028] The output unit is used to write pre-aggregated data and at least one time series data to a disk when the memory usage meets a trigger condition.

[0029] Optionally, the processing unit is specifically configured to: write at least one time series data into a first file in a disk; and write the pre-aggregated data into a second file in the disk.

[0030] Optionally, the first file and the second file are stored in the same directory.

[0031] Optionally, the processing unit is specifically configured to write the pre-aggregated data and at least one time series data to a disk in parallel.

[0032] Optionally, the at least one time series data includes multiple time series data, the multiple time series data belong to the same type of time series data, and the multiple time series data are stored in the same memory block of the memory.

[0033] Optionally, the triggering condition includes: the memory occupancy is greater than or equal to the occupancy threshold; or, the occupancy of a memory block in the memory is greater than or equal to the occupancy threshold, and the memory block is used to store pre-aggregated data and at least one time series data.

[0034] Optionally, before using the pre-aggregation method to process at least one time series data, the input unit is also used to: receive first indication information, the first indication information is used to indicate a binding relationship; the processing unit is also used to: save the identifier of at least one time series data and the identifier of the pre-aggregation method in the metadata module according to the first indication information; the processing unit is specifically further used to: determine the identifier of the pre-aggregation method from the metadata module according to the at least one time series data identifier; determine the pre-aggregation method according to the identifier of the pre-aggregation method.

[0035] Optionally, after the pre-aggregated data and at least one time series data are written to the disk, the input unit is also used to: receive second indication information, the second indication information is used to indicate the release of the binding relationship; the processing unit is also used to: delete the identifier of the pre-aggregation method saved in the metadata module according to the second indication information.

[0036] Optionally, the input unit is further configured to: receive third indication information, where the third indication information is used to instruct cancellation of the pre-aggregation method; and the processing unit is further configured to: delete the pre-aggregation method stored in the metadata module according to the third indication information.

[0037] In a third aspect, embodiments of the present application provide a device for pre-aggregating time series data, which may be a server or a chip applied to a server. The device may include a processor configured to execute any of the methods in the first aspect and its optional embodiments.

[0038] Optionally, the device may further include a transceiver. When the device is a server, the transceiver may be a transceiver circuit, an antenna, etc.; when the device is a chip used in a server, the transceiver may be an input / output interface, a pin, a circuit, etc.

[0039] Optionally, the device may further include a memory for storing instructions, and the processor executes the instructions stored in the memory to cause the device to perform any of the methods in the first aspect and its optional embodiments. When the device is a server, the memory may be a read-only memory, a random access memory, or the like; when the device is a chip used in a server, the memory may be a register, a cache, or the like.

[0040] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed on a computer, the computer executes any one of the methods in the first aspect and its optional implementation methods.

[0041] In the fifth aspect, an embodiment of the present application provides a computer program product, which includes: computer program code or computer program instructions, which, when the computer program code or computer program instructions are executed by a device for pre-aggregating time series data, enables the device to execute any one of the methods in the first aspect and its optional embodiments.

[0042] The beneficial effects of the second to fifth aspects can refer to the beneficial effects of the first aspect and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 is a schematic diagram of a system suitable for use in the present application;

[0044] Figure 2 is a schematic diagram of a method for pre-aggregating time series data provided in an embodiment of the present application;

[0045] Figure 3 is a schematic diagram of another method for pre-aggregating time series data provided by an embodiment of the present application;

[0046] Figure 4 This is a schematic diagram of a method for registering a pre-aggregation function provided in an embodiment of the present application;

[0047] Figure 5 This is a schematic diagram of a method for binding a pre-aggregation function provided in an embodiment of the present application;

[0048] Figure 6 is a schematic diagram of a method for using a pre-aggregation function provided in an embodiment of the present application;

[0049] Figure 7 Schematic diagram of a method for unbinding a pre-aggregation function provided in an embodiment of the present application;

[0050] Figure 8 This is a schematic diagram of a method for canceling a pre-aggregation function provided in an embodiment of the present application;

[0051] Figure 9 is a schematic diagram of a device for pre-aggregating time series data provided in an embodiment of the present application;

[0052] Figure 10 This is a schematic diagram of a server provided in an embodiment of the present application. DETAILED DESCRIPTION

[0053] The technical solution in this application will be described below with reference to the accompanying drawings.

[0054] To facilitate understanding of the technical solutions of the embodiments of the present application, a brief introduction to the concepts involved in the embodiments of the present application is first given.

[0055] 1. Time series data

[0056] Time series data refers to data of the same category and metric recorded in chronological order, representing data generated at a specific point in time. Time series data is characterized by the presence of a timestamp for each piece of data. The data consists of a triplet (metric, timestamp, value). Multiple metrics of the same type are referred to as a time series.

[0057] Taking the smart meter that records household electricity consumption as an example, the two time series data generated by the smart meter are<power,2023-6-20 20:30,99> and<power,2023-6-20 20:31,100> , where power represents the amount of electricity recorded by the smart meter, i.e., the indicator of the two time series data. 2023-6-20 20:30 and 2023-6-20 20:31 represent the times corresponding to the amount of electricity recorded by the smart meter, i.e., the timestamps of the two time series data. 99 and 100 represent the specific amount of electricity recorded by the smart meter, i.e., the numerical values ​​of the two time series data. The two power values ​​constitute a time series.

[0058] Time series data differs from other data in that it is more suitable for reflecting the process of data "change." By connecting the values ​​in time series data into lines along the time coordinate, multi-dimensional reports can be formed. These reports can reveal data trends and regularities, detect anomalies, and enable predictions and early warnings.

[0059] In recent years, the application of time series data has become increasingly widespread. It has been widely used in fields such as IoT, economics and finance, environmental monitoring, medicine, industrial manufacturing, agricultural production, and hardware / software system monitoring. Time series data can reveal trends, patterns, and anomalies within research objects. With the rise of artificial intelligence, the role of time series data as foundational data in big data, machine learning, real-time forecasting, and early warning has become increasingly prominent. Consequently, the research and application of time series data has become increasingly in-depth and important.

[0060] For example, in the field of autonomous driving, a vehicle's location changes over time, while other vehicle attributes (such as model, color, and license plate number) remain constant. This time-related location data constitutes a set of time series data. Time series data is also widely present on the Internet, such as records of user website visits and system log data.

[0061] Time series data has the following characteristics:

[0062] 1) The most obvious characteristic of time series data is its unique timestamp. Using timestamps as a unique identifier is the biggest difference between time series data and relational data. Relational data typically uses other fields as unique identifiers, such as student data, which is often identified by their student ID number.

[0063] 2) Time series data is constantly growing, generating new data at each time granularity. The volume of time series data continues to grow linearly, generating massive amounts of data. However, the growth of relational data is typically not continuous over time. For example, the volume of student data for a school is typically relatively stable over time.

[0064] 3) Time series data rarely needs to be updated. Once a measurement value is recorded, it will not be changed again, so updating time series data is rarely necessary. For example, a temperature sensor only records a temperature value once during a measurement cycle. For relational data, existing data is frequently updated. For example, a student's personal information (such as home address) may change frequently.

[0065] 4) Time series data has hot and cold characteristics. Time series data close to the current time is of high value and can be stored as hot data. Time series data farther in time has a gradually decreasing value and can be archived as cold data.

[0066] In addition, time series data also has the characteristics of high data generation frequency and continuous high concurrent writing.

[0067] 2. Time series database.

[0068] Time series databases have emerged to meet the rapidly growing demand for time series data applications and their distinct characteristics from traditional relational data. A time series database is a database system specifically designed for storing and querying time series data. Compared to traditional relational databases, time series databases focus more on writing and querying massive amounts of data and do not require complex transaction management capabilities.

[0069] Time series databases have the following characteristics:

[0070] 1) High-throughput and high-speed data writing capabilities: In time series data businesses, massive amounts of time series data are continuously generated, and high writing speeds are required. Therefore, time series databases need to have high-throughput data writing capabilities to ensure the timeliness and reliability of time series data.

[0071] 2) High compression rate: Since time series data is large in volume and needs to be stored for a long time, it is necessary to compress the time series data to save storage space and improve query efficiency.

[0072] 3) Efficient time window query capabilities: Time series data query requirements are generally divided into two categories: real-time data query and historical data query. Historical data query usually requires querying a large amount of data within a certain time window, so data query optimization is required to improve query efficiency.

[0073] 4) Efficient aggregation capabilities: Time series data businesses are usually concerned with the aggregate values ​​of data, such as aggregation functions such as mean and count, to reflect the data status within a certain time period. Therefore, time series databases need to provide efficient aggregation capabilities.

[0074] 5) Batch overwrite and batch delete capabilities: For expired time series data, batch overwrite or batch delete operations need to be performed promptly to ensure the stability and performance of the time series database.

[0075] 6) High scalability and reliability: Time series databases can support distributed architecture and dynamically expand the number of nodes to meet the needs of different data scales. At the same time, they can implement operations such as data backup and disaster recovery, improving the reliability of time series data.

[0076] 7) Large-scale parallel computing capabilities: Able to process time series data on multiple nodes, execute complex queries concurrently, and improve query efficiency.

[0077] In addition, time series databases usually need to provide rich data processing and analysis capabilities, such as data cleaning, statistics, analysis, and prediction, to provide more value to the business.

[0078] Based on the above characteristics, the time domain database can serve the following scenarios:

[0079] IoT: IoT uses sensors to collect large amounts of time-series data, such as temperature, humidity, and pressure, which requires fast and efficient storage and query. Time series databases provide efficient data storage and query capabilities, providing important support for IoT applications.

[0080] Finance: Financial data is time-series in nature, and data such as stock prices, trading volumes, and exchange rates require real-time processing and monitoring. Time-series databases help data analysts and traders quickly query data for decision-making.

[0081] Commercial retail: Time series databases can process order transaction amounts and payment data, product inventory, and logistics data in e-commerce systems.

[0082] Industry: Time series databases can process industrial machine data, such as real-time rotation speed, wind speed data, and power generation data of wind turbines.

[0083] Development operations (DevOps): In a DevOps environment, various logs and metrics need to be collected, stored, and analyzed to quickly locate and resolve problems. Time series databases can provide reliable data storage and query support for DevOps.

[0084] Artificial Intelligence: Artificial Intelligence applications require processing large amounts of time series data, including video, audio, and text data. Time series databases can support artificial intelligence algorithms in processing and analyzing this data.

[0085] Energy and Utilities: The energy and utility sectors require real-time monitoring of sensor data, grid status, weather information, and more to ensure the normal operation of the system. Time series databases provide efficient data storage and query capabilities for measuring and controlling power grid systems.

[0086] Smart city construction: Time series databases can be used to analyze urban operation data in real time, optimize urban public services, and improve urban water supply, power supply, and public transportation.

[0087] Scientific research: Time series databases can be used to store and analyze various scientific data, such as meteorological data, earthquake data, and biological data.

[0088] The above application scenarios are examples rather than limitations. The embodiments of the present application do not limit the application scenarios of the time series database.

[0089] The following describes a system suitable for the embodiment of the present application. Figure 1 As shown, the system 100 includes a client 110 and at least one server 120. When the system 100 includes one server 120, the server 120 can be called a stand-alone server. When the system 100 includes multiple servers 120, the server 120 can be called a cluster server.

[0090] The client 110 can be a mobile phone, a tablet computer, a computer with wireless transceiver function, a virtual reality (VR) terminal, an augmented reality (AR) terminal, a wireless terminal in industrial control, a whole vehicle, a wireless communication module in the whole vehicle, a telematics box (T-box), a roadside unit (RSU), a wireless terminal in unmanned driving, a smart speaker in IoT, a wireless user device in remote medical, a wireless user device in a smart grid, a wireless user device in transportation safety, a wireless user device in a smart city, or a wireless user device in a smart home. The embodiments of the present application are not limited to this.

[0091] As an example and not a limitation, in the embodiment of the present application, the client 110 may also be a wearable device. Wearable devices may also be referred to as wearable smart devices, which are a general term for wearable devices that are intelligently designed and developed using wearable technology for daily wear, such as glasses, gloves, watches, clothing, and shoes. A wearable device is a portable device that is worn directly on the body or integrated into the user's clothes or accessories. Wearable devices are not only hardware devices, but also achieve powerful functions through software support, data interaction, and cloud interaction. Broadly speaking, wearable smart devices include electronic devices that are fully functional, large in size, and can achieve complete or partial functions without relying on smartphones, such as smart watches or smart glasses, or electronic devices that only focus on a certain type of application function and need to be used in conjunction with other devices such as smartphones, such as various smart bracelets and smart jewelry for measuring vital signs.

[0092] The server 120 includes at least one processor core 121 and at least one memory 122. Optionally, the server 120 may further include at least one memory 123, that is, the memory 123 may be integrated into the server 120 or provided outside the server 120.

[0093] The at least one processor core 121 may be located in one processor or in different processors. The processor may be a central processor unit (CPU), a system on chip (SoC), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a microcontroller unit (MCU), a programmable logic device (PLD), or other logic devices such as discrete gates, transistor logic devices, or discrete hardware components.

[0094] Memory 122 serves as a cache memory, which may also be referred to as a memory, and is typically a volatile memory. By way of example and not limitation, memory 122 may be a random access memory (RAM), such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), and direct RAMbus RAM (DR RAM).

[0095] Memory 123 is a persistent storage device, which may also be referred to as a disk or hard disk, and is typically a non-volatile memory. By way of example and not limitation, memory 122 may be a read-only memory (ROM), such as a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory.

[0096] The processor core 121, memory 122, and storage 123 can be interconnected via a bus or other technology. The embodiments of the present application do not limit the specific types of the processor core 121, memory 122, and storage 123, nor do they limit the communication method between the processor core 121, memory 122, and storage 123.

[0097] By way of example and not limitation, in the embodiments of the present application, the server 120 may be a tower server, a blade server, a rack server, or a cabinet server, or the server 120 may be a complex instruction set computer (CISC) server, a reduced instruction set computer (RISC) server, or an explicitly parallel instruction computing (EPIC) server. The server 120 may also be a virtual server, such as a virtual machine (VM) or a container (Docker).

[0098] The client 110 and the server 120 can communicate via a wired connection or a wireless connection. The wired connection can be an optical fiber or cable, and the wireless connection can be a cellular network connection, wireless fidelity (WiFi), or Bluetooth. The embodiment of the present application does not limit the connection method between the client 110 and the server 120.

[0099] The server 120 is installed with a time series database. The time series database performs pre-aggregation processing on the time series data when it is stored. For example, the count function is used to count the number of time series data in a time window, and the max function is used to count the maximum value of the time series data in a time window. The number of time series data and the maximum value of the time series data are pre-aggregated data. Therefore, the time series database stores at least one time series data and the pre-aggregated data corresponding to the at least one time series data.

[0100] The client 110 is used to provide a query entry for the user, that is, the user can enter a query request through the client 110, and the query request is used to query the time series data in the time series database on the server 120. After receiving the query request, the client 110 sends the query request to the server 120.

[0101] After receiving the query request, the server 120 executes the query task through the time series database, and the time series database determines the content required by the user based on the pre-aggregated results.

[0102] For example, the query request is used to query the number of time series data from June 20 to June 21. If the time window of the count function in the time series database is one day, the time series database can add the result of the count function on June 20 and the result of the count function on June 21 (i.e., secondary aggregation) to obtain the content required by the user.

[0103] For another example, the query request is used to query the maximum value of the time series data on June 20. If the time window of the count function in the time series database is one day, the time series database can determine the content required by the user based on the result of the max function statistics on June 20.

[0104] After the time series database obtains the content corresponding to the query request, it sends the content to the client 110 through the communication interface between the server 120 and the client 110, and the client 110 displays the content to the user.

[0105] With the growth of IoT time series data, the performance requirements for aggregate queries of time series data under large data volumes are becoming increasingly stringent. High-performance time series queries have become a key requirement for more and more businesses. Aggregate queries are extremely time-consuming, for example: millisecond-level single-metric aggregate queries and millisecond-level multi-metric aggregate queries.

[0106] Time series data is generated frequently, written with high concurrency, and has a large data base, which undoubtedly makes aggregate queries difficult. Therefore, the industry has proposed pre-aggregation technology. Pre-aggregation technology pre-generates and stores aggregated data during the data writing process. This pre-aggregated data can be used for secondary aggregation during query time, accelerating aggregate queries. Pre-aggregation technology has become one of the important means to improve the efficiency of aggregate queries.

[0107] When pre-aggregating time series data, it's often necessary to manually set the pre-aggregation time range. However, the frequency of time series data generation may vary across different time periods. For example, in the financial sector, when transaction volume is high over a period, more transaction data (i.e., time series data) is generated, while when transaction volume is low over a period, less transaction data (i.e., time series data) is generated. Manually setting the pre-aggregation time range makes it difficult to ensure that the amount of time series data and pre-aggregated data within different time ranges is the same. Uneven distribution of time series data and pre-aggregated data can lead to reduced storage space utilization.

[0108] The following describes a method for pre-aggregating time series data provided by an embodiment of the present application. The execution subject of the method can be a server or a chip applied to a server, and the following description takes the execution subject being a server as an example.

[0109] like Figure 2As shown, the method 200 includes the following contents.

[0110] S210, obtaining at least one time series data from a first device, where the first device is a device that generates at least one time series data, or the first device is a node device of a time series database, and any one of the at least one time series data includes an identifier.

[0111] The server can directly obtain time series data from a device that generates time series data (such as an IoT device). For example, after a smart meter generates time series data, it transmits the time series data to the server 120 via IoT.

[0112] The server may also obtain time series data from the node device. For example, the server 120 obtains at least one time series data from other servers in the server cluster where it is located.

[0113] The embodiment of the present application does not limit the specific method of obtaining at least one time series data.

[0114] S220: Determine a pre-aggregation method according to an identifier of at least one time series data, where a preset binding relationship exists between the pre-aggregation method and the identifier of at least one time series data.

[0115] The identifier of time series data is the indicator of time series data. For example, a time series data generated by a smart meter is<power,2023-6-2020:30,99> , where power represents the amount of electricity recorded by the smart meter, which is the identifier or indicator of time-series data.

[0116] The metadata module of the server stores a preset binding relationship, which includes the identifier of at least one time series data and the identifier of the associated pre-aggregation method. The server can determine the identifier of the pre-aggregation function based on the binding relationship and the identifier of at least one time series data, thereby determining the pre-aggregation function that needs to be used.

[0117] S230: Process at least one time series data using a pre-aggregation method to generate pre-aggregated data.

[0118] The pre-aggregation method can be a pre-aggregation function, such as the count function, the max function, and the sum function. The count function is used to count the number of time series data in a time window, the max function is used to count the maximum value of time series data in a time window, and the sum function is used to count the sum of time series data in a time window.

[0119] The server can execute the pre-aggregation function once each time a new time series data is obtained, or it can execute the pre-aggregation function once after obtaining multiple time series data.

[0120] For example, after the server obtains time series data 1, it executes the max function once. If the value of time series data 1 is the largest in the current time window, time series data 1 is recorded in the pre-aggregated data. After the server obtains time series data 2, it executes the max function again. If the value of time series data 2 is the largest in the current time window, the time series data corresponding to the maximum value in the pre-aggregated data is updated to time series data 2.

[0121] For another example, after the server obtains time series data 1, because the amount of newly obtained time series data does not exceed the quantity threshold (assuming that there is only time series data 1 and the quantity threshold is 2), the server does not execute the max function; after the server obtains time series data 2, the amount of newly obtained time series data still does not exceed the quantity threshold, and the server does not execute the max function; after the server obtains time series data 3, the amount of newly obtained time series data exceeds the quantity threshold, and the server executes the max function once to determine the maximum value of the time series data in the current time window. Executing the pre-aggregation function when the amount of newly obtained time series data exceeds the quantity threshold can reduce the number of executions of the pre-aggregation function, thereby reducing the server's computing overhead.

[0122] The server can process time series data using one pre-aggregation function or multiple pre-aggregation functions. When using one pre-aggregation function to process time series data, the server can generate one pre-aggregated data set; when using multiple pre-aggregation functions to process time series data, the server can generate multiple pre-aggregated data sets.

[0123] For example, the server can execute the count function on the time series data in the current time window to generate a count result, which is the pre-aggregated data; or, the server can execute the count function and the max function on the time series data in the current time window respectively to generate a count result and a max result, which are two pre-aggregated data.

[0124] After the server obtains time series data and generates pre-aggregated data, perform the following steps.

[0125] S240: Write the time series data and pre-aggregated data into the memory.

[0126] S250: When the memory usage meets the trigger condition, write the pre-aggregated data and at least one time series data to the disk.

[0127] Optionally, the trigger condition may be: the memory occupancy is greater than or equal to the occupancy threshold. In this case, the memory in S250 should be understood as the entire memory space.

[0128] For example, after the server writes 1,000 time series data and the pre-aggregated data of the 1,000 time series data into the memory, the memory occupancy rate reaches 81%. If the occupancy rate threshold is 80%, the server can perform a disk write operation to write the 1,000 time series data and the corresponding pre-aggregated data to the disk and release memory space.

[0129] Optionally, the trigger condition may also be: the occupancy rate of some memory blocks in the memory is greater than or equal to the occupancy rate threshold. In this case, the memory in S250 should be understood as the memory space (or memory block) storing the pre-aggregated data and at least one time series data.

[0130] For example, the memory includes multiple memory blocks, different memory blocks are used to store time series data and pre-aggregated data corresponding to different indicators. The multiple memory blocks include memory block 1. After the server writes the time series data and pre-aggregated data of the smart meter into memory block 1, the occupancy rate of memory block 1 reaches 81%. If the occupancy rate threshold is 80%, the server can perform a write disk operation to write the time series data and corresponding pre-aggregated data in memory block 1 to the disk and release memory block 1. Different memory blocks can be different physical spaces or different logical spaces, which is not limited in this embodiment of the present application.

[0131] Memory usage can also be replaced by other equivalent parameters. For example, when the amount of time series data and pre-aggregated data in memory to be written to disk is greater than or equal to the data volume threshold, the server can perform the disk write operation. Any parameter equivalent to memory usage falls within the scope of protection of this application.

[0132] Writing data from memory to disk incurs other time overheads besides the data writing time. When the memory usage is greater than or equal to the memory usage threshold, a large amount of data to be written to disk has accumulated. Writing a large amount of data to disk at once can spread out other time overheads and improve the efficiency of writing time series data to disk. This allows the server to free up memory space as quickly as possible, avoiding congestion associated with retrieving new time series data.

[0133] When the memory usage doesn't meet the trigger conditions, the server can continue to acquire and pre-aggregate time series data. When the memory meets the trigger conditions, the server can write the time series data and pre-aggregated data currently stored in memory to disk. Because the trigger conditions remain unchanged, the amount of time series data and pre-aggregated data written to disk at different times is the same, ensuring even distribution of time series data and pre-aggregated data and improving storage space utilization for time series data and pre-aggregated data.

[0134] Optionally, when executing S250, the server may specifically perform the following steps:

[0135] At least one time series data is written to a first file in a disk; and pre-aggregated data is written to a second file in the disk.

[0136] In each embodiment of the present application, "first" and "second" are used to distinguish different objects, and there are no other limitations. For example, the first file and the second file represent two files. The first file can be a data file, and the second file can be an agg file. The embodiments of the present application do not limit the specific types of the first file and the second file.

[0137] The pre-aggregation method may include multiple pre-aggregation functions, and different pre-aggregation functions have different functions. By using multiple pre-aggregation functions, comprehensive pre-aggregation data can be obtained, which is conducive to accelerating the query efficiency of time series data. However, the amount of pre-aggregation data generated by multiple pre-aggregation functions may be very large. For a time series database with a built-in pre-aggregation function, it usually writes the time series data and the pre-aggregation data into the same file, which will cause the data volume of the file to increase, and then cause the query efficiency of the time series data to decrease. Therefore, the time series database with a built-in pre-aggregation function can usually only preset a small number of pre-aggregation functions, and it is difficult to implement a custom pre-aggregation function. In this embodiment, the pre-aggregation data and the time series data are written into different files, so that the pre-aggregation data does not affect the file size of the time series data. Under the premise of not affecting the query efficiency of the time series data, more pre-aggregation functions can be applied to pre-aggregate the time series data to realize the function of the custom pre-aggregation function.

[0138] Optionally, the first file and the second file are stored in the same directory.

[0139] Taking the smart meter as an example, if the first file corresponding to the time series data of the smart meter is 12345678.data, and the second file corresponding to the pre-aggregated data of the smart meter is 12345678.agg, then the directory structure of the first file can be ···. / power / 20220105 / 12345678.data, and the directory structure of the second file can be ···. / power / 20220105 / 12345678.agg.

[0140] The first file and the second file are stored in the same directory. The server does not need to search across directories for the second file corresponding to the first file, which can improve the query efficiency of time series data.

[0141] Optionally, when executing S250, the server may specifically perform the following steps:

[0142] Write pre-aggregated data and at least one time series data to disk in parallel.

[0143] For example, the server can use two threads to process the disk write operations of pre-aggregated data and time series data respectively. In this way, pre-aggregated data and time series data can be written to disk in parallel. If the disk write operation of pre-aggregated data is blocked, the disk write operation of time series data can still be carried out; if the disk write operation of time series data is blocked, the disk write operation of pre-aggregated data can still be carried out.

[0144] Time series data is characterized by rapid data generation and continuous high-concurrency writes. If the time series data and pre-aggregated data in memory cannot be written to disk in a timely manner, insufficient memory may occur, and new time series data cannot be saved in a timely manner. This embodiment uses multi-threaded parallel writing. Even if the writing of one of the pre-aggregated data and the time series data is blocked, the writing of the other data can continue. The server can quickly free up memory space and avoid the loss of new time series data due to insufficient memory.

[0145] Optionally, the at least one time series data includes multiple time series data, the multiple time series data belong to the same type of time series data, and the multiple time series data are stored in the same memory block of the memory.

[0146] Different types of time series data may use different pre-aggregation methods. By storing the same type of time series data in the same memory block, the server no longer needs to determine whether the time series data matches the pre-aggregation method when pre-aggregating the time series data in the memory block, thereby improving the efficiency of pre-aggregation.

[0147] For example, the server obtains multiple time series data generated by smart meters and multiple time series data generated by the financial system. The server can store the multiple time series data generated by the smart meter in memory block 1 and the multiple time series data generated by the financial system in memory block 2, and process the time series data in memory block 1 and memory block 2 respectively through different pre-aggregation functions, thereby improving the efficiency of pre-aggregation.

[0148] The above describes in detail the method of processing time series data based on the pre-aggregation method. Since different time series data may be suitable for different pre-aggregation methods, the server needs to bind the pre-aggregation method with the time series data before executing the pre-aggregation method.

[0149] Optionally, before executing S230, the server may save the binding relationship between the pre-aggregation method and at least one time series data; subsequently, when executing S230, the pre-aggregation method may be used to process the at least one time series data. For example, method 200 further includes:

[0150] receiving first indication information, where the first indication information is used to indicate a binding relationship;

[0151] According to the first indication information, an identifier of at least one time series data and an identifier of a pre-aggregation method are saved in a metadata module.

[0152] When the server executes S220, it may specifically execute:

[0153] determining an identifier of a pre-aggregation method from a metadata module according to at least one time series data identifier;

[0154] Determine the pre-polymerization method based on the identification of the pre-polymerization method.

[0155] For example, the user can instruct the server to bind the time series power and count functions through the first indication information. The server can generate metadata including the identifiers of the power and count functions based on the first indication information, and save the metadata in the metadata module. In this way, when the server pre-aggregates the time series data with the indicator power, it queries the pre-aggregation function corresponding to power from the metadata module, obtains the identifier of the count function, and thus determines to use the count function to pre-aggregate the time series data with the indicator power.

[0156] The server can also bind multiple pre-aggregation functions to a time series. For example, the server can bind the count function and the max function to the time series power, and save the binding relationship as metadata in the metadata module. In this way, when the server pre-aggregates the time series data with the power indicator, it can use the count function and the max function for pre-aggregation.

[0157] By pre-binding the time series with the matching pre-aggregation method, the server can obtain information related to the pre-aggregation method in advance based on the binding relationship. In this way, after obtaining the time series data, the server can directly process the time series data without waiting for user instructions, thereby improving the pre-aggregation efficiency of the time series data.

[0158] Optionally, after executing S250, the server may further execute:

[0159] receiving second indication information, where the second indication information is used to indicate the release of the binding relationship;

[0160] The identifier of the pre-aggregation method stored in the metadata module is deleted according to the second instruction information.

[0161] For the same type of time series data, the pre-aggregation methods used in different scenarios may differ. After processing the current time series data using a pre-aggregation method, unbinding the relationship allows you to use a new pre-aggregation method to process new time series data, flexibly adapting to different time series data pre-aggregation scenarios.

[0162] For example, if client A needs to determine the amount of power in a time series within time window A, the server can bind the count function to the power of the time series and save the binding relationship in the metadata module. In this way, when the server pre-aggregates the time series data whose indicator is power in time window A, it can use the count function for pre-aggregation. After completing client A's task, the server can unbind the time series power from the count function. If client B needs to determine the maximum value of the power of the time series within time window B, the server can bind the max function to the power of the time series and save the binding relationship in the metadata module. In this way, when the server pre-aggregates the time series data whose indicator is power in time window B, it can use the max function for pre-aggregation.

[0163] Optionally, the server can also execute:

[0164] receiving third indication information, where the third indication information is used to indicate cancellation of the pre-aggregation method;

[0165] The pre-aggregation method stored in the metadata module is deleted according to the third instruction information.

[0166] For example, when supporting custom pre-aggregation functions, a large amount of metadata of pre-aggregation functions may be stored in the metadata module. Before executing the pre-aggregation function, the server needs to query the metadata module for pre-aggregation functions with binding relationships. Canceling the pre-aggregation method from which the binding relationship has been unbound (i.e., deleting the metadata of the pre-aggregation function from which the binding relationship has been unbound) can improve the query efficiency of the pre-aggregation method.

[0167] Another method for pre-aggregating time series data provided by an embodiment of the present application is described below.

[0168] like Figure 3 As shown, the method includes the following contents.

[0169] S310, register a pre-aggregation function.

[0170] S320, bind a pre-aggregation function.

[0171] S330, use pre-aggregation function.

[0172] S340, unbinding the pre-aggregation function.

[0173] S350, cancel the pre-aggregation function.

[0174] The following describes these steps in detail.

[0175] like Figure 4 As shown, S310 may include the following contents:

[0176] S311: Generate an executable file for the pre-aggregation function.

[0177] Users can write pre-aggregation functions according to the custom function rules of the time series database and generate executable files for the pre-aggregation functions.

[0178] For example, a custom function rule allows multiple inputs and a single output. Custom function rules also support setting the data types of the input and output data. You can create a sum function based on this custom function rule, setting the input data types to int, double, and float, and the output data type to double. Once the function is created, the executable file sunFunction.jar is generated.

[0179] S312: Upload the executable file to the pre-aggregation function directory.

[0180] For example, a user may upload an executable file to the pre-aggregation function directory through the client 110 so that the time series database can read the executable file.

[0181] S313, distribute the executable file.

[0182] In some cases, the server architecture of the time series database is a cluster architecture. After a node in the cluster reads the executable file, it can distribute the executable file to other nodes in the server cluster. For stand-alone servers, this step is not required.

[0183] S314, register a pre-aggregation function.

[0184] After the time series database reads the executable file, it loads the executable file into the executor and writes the metadata of the pre-aggregation function into the metadata module, that is, registering the pre-aggregation function. Taking the sum function as an example, the command example for registering the pre-aggregation function is as follows:

[0185] CREATE PREAGGFUNCTION sum AS'org.apache.udf.sumFunction'location' / srv / sumFunction.jar'.

[0186] Users can determine the pre-aggregation functions supported by the time series data block by querying the metadata module.

[0187] like Figure 5 As shown, S320 may include the following contents:

[0188] S321: The user creates a time series and binds a pre-aggregation function.

[0189] For example, a user may create a time series power through the client 110 and bind one or more pre-aggregation functions to the time series power.

[0190] The following are examples of commands for creating a time series and binding a pre-aggregation function:

[0191] CREATE TIMESERIES power with datatype=INT32,pre_agg=sum.

[0192] S322: Save the binding relationship to the metadata module.

[0193] Users can determine the pre-aggregation function bound to the time series by querying the metadata module.

[0194] like Figure 6 As shown, S330 may include the following contents:

[0195] S331, input node to obtain time series data.

[0196] Time series databases are designed to process massive amounts of data and therefore require efficient writing, storage, and reading capabilities. However, in a standalone environment, neither the hardware nor the software can meet these requirements. Therefore, time series databases need to adopt distributed storage to solve this problem. This involves distributing data across clustered servers for storage, processing, and reading, achieving high performance, high availability, and high fault tolerance.

[0197] The input node is the first server in the cluster that obtains time series data.

[0198] S332: The input node routes the time series data to the target node according to the data distribution.

[0199] If the input node has a high load, the input node can execute S332 and route the time series data to a server with a lower load, i.e., the target node, based on the data distribution. If the input node has a low load, the input node can skip S332 and process the time series data locally. In this case, the input node is the target node.

[0200] S333: The target node writes the time series data into memory and groups them.

[0201] Since time series data has the characteristics of high generation frequency and continuous high concurrent writing, the target node may obtain a large amount of time series data in a short period of time. The target node can group the time series data according to the indicators and store the time series data with the same indicators in the same area to facilitate pre-aggregation processing.

[0202] For example, the target node obtains two time series data generated by the smart meter, which are<power,2023-6-2020:30,99> and<power,2023-6-20 20:31,100> , the target node can store the two time series data in a memory block according to the indicator power of the two time series data.

[0203] S334: The target node obtains the pre-aggregation function bound to the time series from the metadata module.

[0204] For example, the target node obtains two time series data generated by the smart meter, which are<power,2023-6-2020:30,99> and<power,2023-6-20 20:31,100> , the target node can obtain the pre-aggregation function information bound to the time series power from the metadata module according to the indicator power of the two time series data, and call the pre-aggregation function based on the pre-aggregation function information.

[0205] The target node typically queries the metadata for the bound pre-aggregation function after first acquiring the smart meter's time series data. Subsequently, when acquiring the smart meter's time series data again, the target node can directly perform pre-aggregation processing without having to query the bound pre-aggregation function again. Furthermore, the grouping of time series data and the acquisition of the bound pre-aggregation function can be performed simultaneously or sequentially, and this is not limited in this embodiment of the present application.

[0206] S335: The target node calls the pre-aggregation function to pre-aggregate the time series data in the group to generate pre-aggregated data.

[0207] For example, there are two time series data in the current group, namely<power,2023-6-20 20:30,99> and<power,2023-6-20 20:31,100> If the pre-aggregation function bound to the power time series is the sum function, the target node can execute the sum function to calculate the sum of 99 and 100, and obtain the pre-aggregated data 199. The pre-aggregated data 199 is cached in the memory. After obtaining new power time series data, the sum function is executed on the new power time series data to update the pre-aggregated data.

[0208] S336: The memory triggers the disk write condition, and the time series data and pre-aggregated data are written to different files on the disk in parallel.

[0209] For example, when memory usage exceeds the memory usage threshold, the target node executes the Flush command, writing the time series data in memory and the pre-aggregated data to separate files on disk. The pre-aggregation range is automatically determined based on the Flush command, eliminating the need to pre-set the pre-aggregation time range. Pre-aggregated data files and time series data files are stored in the same directory, facilitating data deletion and merging.

[0210] An optional directory structure is as follows:

[0211] … / power / 20220105 / 12343678.data;

[0212] … / power / 20220105 / 12343678.agg.

[0213] Users can send query commands to the target node to view the pre-aggregated data corresponding to the time series. An optional query command is as follows:

[0214] Show timeseries power.

[0215] like Figure 7 As shown, S340 may include the following contents:

[0216] S341: Receive a command for a pre-aggregation function bound to a contact time series.

[0217] For example, a user can issue the following command:

[0218] ALTER timeseries power DELETE AGG max.

[0219] This command instructs the target node to unbind the time series power from the pre-aggregation function max. After receiving this command, the target node executes the following steps.

[0220] S342: Determine whether the time series is bound to a pre-aggregation function.

[0221] If not, end the unbinding process; if so, execute the following steps.

[0222] S343, unbind the pre-aggregation function of the time series.

[0223] For example, the target node can delete the binding between the time series power and the pre-aggregation function max in the metadata module. For clustered servers, the target node can also notify other nodes to synchronize the latest metadata, that is, to notify other nodes to delete the binding between the time series power and the pre-aggregation function max.

[0224] like Figure 8 As shown, S350 may include the following:

[0225] S351, receiving a command to cancel a pre-aggregation function.

[0226] In some cases, if you no longer want to use some pre-aggregation functions, you can send a command to the target node to cancel the pre-aggregation function. After receiving the command, the target node executes the following steps.

[0227] S352: Determine whether the pre-aggregation function is bound to a time series.

[0228] If a pre-aggregation function is bound to a time series, it means that the pre-aggregation function may be in use. Unregistering the pre-aggregation function will interrupt the processing of the time series data. The target node can prompt the user to release all binding relationships of the pre-aggregation function and end the unregistration process.

[0229] If the pre-aggregation function is not bound to a time series, it means that the pre-aggregation function is not used. Canceling the pre-aggregation function will not affect the processing of time series data. The target node can perform the following steps.

[0230] S353, cancel the pre-aggregation function.

[0231] For example, the target node can delete the information related to the pre-aggregation function max in the metadata module to complete the deregistration of the pre-aggregation function max. For a clustered server, the target node can also notify other nodes to synchronize the latest metadata, that is, notify other nodes to deregister the pre-aggregation function max.

[0232] The above describes in detail the method examples provided by the embodiments of the present application. It is understandable that, in order to implement the above functions, the corresponding device includes hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art should easily appreciate that, in combination with the units and algorithm steps of each example described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in a hardware or computer software driven hardware manner depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0233] Figure 9 1 is a schematic diagram of the structure of an apparatus 900 for pre-aggregating time series data provided in an embodiment of the present application. The apparatus 900 includes a processing unit 910, an input unit 920, and an output unit 930. The input unit 920 performs a receiving step or an input step under the control of the processing unit 910, and the output unit 930 performs a sending step or an output step under the control of the processing unit 910.

[0234] The input unit 920 is configured to: obtain at least one time series data from a first device, where the first device is a device that generates at least one time series data, or the first device is a node device of a time series database, and any one of the at least one time series data includes an identifier;

[0235] The processing unit 910 is configured to: determine a pre-aggregation method according to an identifier of at least one time series data, where the pre-aggregation method and the identifier of at least one time series data have a preset binding relationship; process the at least one time series data using the pre-aggregation method to generate pre-aggregated data; and write the pre-aggregated data and the at least one time series data into a memory;

[0236] The output unit 930 is configured to write the pre-aggregated data and at least one time series data to a disk when the memory occupancy rate meets a trigger condition.

[0237] Optionally, the processing unit 910 is specifically configured to: write at least one time series data into a first file in a disk; and write the pre-aggregated data into a second file in the disk.

[0238] Optionally, the first file and the second file are stored in the same directory.

[0239] Optionally, the processing unit 910 is specifically configured to write the pre-aggregated data and at least one time series data to a disk in parallel.

[0240] Optionally, the at least one time series data includes multiple time series data, the multiple time series data belong to the same type of time series data, and the multiple time series data are stored in the same memory block of the memory.

[0241] Optionally, the triggering condition includes: the memory occupancy is greater than or equal to the occupancy threshold; or, the occupancy of a memory block in the memory is greater than or equal to the occupancy threshold, and the memory block is used to store pre-aggregated data and at least one time series data.

[0242] Optionally, before using the pre-aggregation method to process at least one time series data, the input unit 920 is also used to: receive first indication information, the first indication information is used to indicate a binding relationship; the processing unit 910 is also used to: save the identifier of at least one time series data and the identifier of the pre-aggregation method in the metadata module according to the first indication information; the processing unit 910 is specifically used to: determine the identifier of the pre-aggregation method from the metadata module according to the at least one time series data identifier; determine the pre-aggregation method according to the identifier of the pre-aggregation method.

[0243] Optionally, after the pre-aggregated data and at least one time series data are written to the disk, the input unit 920 is also used to: receive second indication information, the second indication information is used to indicate the release of the binding relationship; the processing unit 910 is also used to: delete the identifier of the pre-aggregation method saved in the metadata module according to the second indication information.

[0244] Optionally, the input unit 920 is further used to: receive third indication information, where the third indication information is used to instruct to cancel the pre-aggregation method; and the processing unit 910 is further used to: delete the pre-aggregation method saved in the metadata module according to the third indication information.

[0245] Those skilled in the art can clearly understand that the specific working process of the device 900 and the technical effects produced by the execution steps can be referred to the description in the aforementioned corresponding method embodiment. For the sake of brevity, they will not be repeated here.

[0246] Device 900 may be a server or a chip. Processing unit 910 may be implemented in hardware or software. When implemented in hardware, processing unit 910 may be a logic circuit, an integrated circuit, or the like. When implemented in software, processing unit 910 may be a general-purpose processor implemented by reading software code stored in a storage unit. The storage unit may be integrated into processing unit 910 or located independently of processing unit 910.

[0247] Figure 10 This is a schematic diagram of the structure of a server provided in an embodiment of the present application. For ease of explanation, Figure 10 Only the main components of the server are shown. Figure 10 As shown, the server 1000 includes a processor 1010, a memory 1020, and an input / output device 1030. The processor 1010 is mainly used to process the time series database protocol and time series data, as well as to control the entire server 1000, execute software programs, and process the data of software programs, for example, to support the server 1000 in performing the actions described in the above method embodiments. The memory 1020 is mainly used to store software programs and data. The input / output device 1030 is, for example, a network card, an antenna, etc., and is mainly used to receive data and output data to the user. The processor 1010, the memory 1020, and the input / output device 1030 can be connected through various buses.

[0248] The processor 1010 and memory 1020 can serve one or more boards. In other words, each board can have its own memory and processor. Alternatively, multiple boards can share the same memory and processor. In addition, each board can also have necessary circuits.

[0249] Those skilled in the art will understand that for ease of explanation, Figure 10 Only one memory and one processor are shown. In an actual server, there may be multiple processors and multiple memories. The memory may also be referred to as a storage medium or a storage device, etc., which is not limited in this application.

[0250] It is understood that the processor in each embodiment of the present application can be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method embodiment can be completed by hardware integrated logic circuits in the processor or software instructions. The above processor can be a CPU, SoC, ASIC, FPGA, MCU, PLD, or other logic device, such as discrete gates, transistor logic devices, or discrete hardware components. The disclosed methods, steps, and logic block diagrams in the embodiments of the present application can be implemented or executed.

[0251] It will be appreciated that the memory in each embodiment of the present application may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a ROM, PROM, EPROM, EEPROM, or flash memory. The volatile memory may be a RAM, which serves as an external cache. By way of example and not limitation, many forms of RAM are available, such as SRAM, DRAM, SDRAM, DDR SDRAM, ESDRAM, SLDRAM, and DR RAM. It should be noted that the memory in the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0252] During implementation, each step of the above method can be completed by an integrated logic circuit of the hardware in the processor or by instructions in the form of software. The steps of the method disclosed in conjunction with the embodiments of the present application can be directly embodied as being executed by a hardware processor, or can be executed by a combination of hardware and software modules in the processor. The software module can be located in a storage medium mature in the art such as a random access register, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in a memory, and the processor reads the information in the memory and completes the steps of the above method in conjunction with its hardware. To avoid repetition, it will not be described in detail here.

[0253] The present application also provides a computer-readable medium having a computer program stored thereon, which implements the functions of any of the above method embodiments when executed by a computer.

[0254] The present application also provides a computer program product, which implements the functions of any of the above method embodiments when executed by a computer.

[0255] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless or microwave, etc.) method. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (eg, a floppy disk, a hard disk, a magnetic tape), an optical medium (eg, a high-density digital video disc (DVD)), or a semiconductor medium (eg, a solid state disk (SSD)).

[0256] It should be understood that the “embodiment” mentioned throughout the specification means that the specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, the various embodiments in the entire specification do not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It should be understood that in various embodiments of the present application, the terminal device and / or the network device can perform some or all of the steps in each embodiment. These steps or operations are merely examples, and the embodiments of the present application can also perform other operations or variations of various operations. In addition, the various steps can be performed in a different order than presented in the various embodiments, and it is possible that not all operations in the embodiments of the present application need to be performed. Moreover, the size of the sequence number of each of the above processes does not mean the order of execution. The order of execution of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0257] It should also be understood that in this application, "when", "if" and "if" all mean that the UE or base station will take corresponding measures under certain objective circumstances. It does not limit the time, and does not require the UE or base station to take judgment actions when implementing it, nor does it mean that there are other limitations.

[0258] Additionally, the terms "system" and "network" are often used interchangeably. The term "and / or" is simply used to describe an association between related objects, indicating that three possible relationships exist. For example, "A and / or B" can mean: A exists alone, A and B exist simultaneously, or B exists alone.

[0259] It should be understood that in each embodiment of the present application, "B corresponding to A" means that B is associated with A and B can be determined based on A. However, it should also be understood that determining B based on A does not mean determining B based solely on A, but B can also be determined based on A and / or other information.

[0260] The above contents are optional embodiments of the technical solution of this application and are not intended to limit the scope of protection of this application. Any modifications, equivalent replacements, improvements, etc. made within the principles of this application should be included in the scope of protection of this application.

Claims

1. A method for pre-aggregating time series data, characterized in that: include: Acquire at least one time series data from a first device, where the first device is a device for generating the at least one time series data, or the first device is a node device of a time series database, and any one of the at least one time series data includes an identifier; Determining a pre-aggregation method according to the identifier of the at least one time series data, wherein the pre-aggregation method has a preset binding relationship with the identifier of the at least one time series data; Processing the at least one time series data using the pre-aggregation method to generate pre-aggregated data; Writing the pre-aggregated data and the at least one time series data into a memory; When the memory occupancy rate satisfies a trigger condition, the pre-aggregated data and the at least one time series data are written to a disk.

2. The method according to claim 1, characterized in that Writing the pre-aggregated data and the at least one time series data into a disk includes: Writing the at least one time series data into a first file in the disk; and, The pre-aggregated data is written to a second file in the disk.

3. The method according to claim 2, characterized in that The first file and the second file are stored in the same directory.

4. The method according to any one of claims 1 to 3, characterized in that Writing the pre-aggregated data and the at least one time series data into a disk includes: The pre-aggregated data and the at least one time series data are written to the disk in parallel.

5. The method according to any one of claims 1 to 4, characterized in that The at least one time series data includes a plurality of time series data, the plurality of time series data are time series data of the same type, and the plurality of time series data are stored in a same memory block of the memory.

6. The method according to any one of claims 1 to 5, characterized in that The trigger conditions include: The memory occupancy rate is greater than or equal to an occupancy rate threshold; or, An occupancy rate of a memory block in the memory is greater than or equal to an occupancy rate threshold, and the memory block is used to store the pre-aggregated data and the at least one time series data.

7. The method according to any one of claims 1 to 6, characterized in that Before using the pre-aggregation method to process the at least one time series data, the method further includes: receiving first indication information, where the first indication information is used to indicate the binding relationship; storing, according to the first indication information, an identifier of the at least one time series data and an identifier of the pre-aggregation method in a metadata module; The determining of the pre-aggregation method according to the identifier of the at least one time series data includes: Determining an identifier of the pre-aggregation method from the metadata module according to the at least one time series data identifier; The pre-polymerization method is determined according to the identifier of the pre-polymerization method.

8. The method according to claim 7, characterized in that After writing the pre-aggregated data and the at least one time series data to a disk, the method further includes: receiving second indication information, where the second indication information is used to indicate to release the binding relationship; The identifier of the pre-aggregation method stored in the metadata module is deleted according to the second indication information.

9. The method according to claim 8, characterized in that The method further comprises: receiving third indication information, where the third indication information is used to instruct cancellation of the pre-aggregation method; The pre-aggregation method stored in the metadata module is deleted according to the third indication information.

10. A device for pre-aggregating time series data, characterized in that: include: Module for executing the method as claimed in any one of claims 1 to 9.

11. A device for pre-aggregating time series data, characterized in that: include: A processor, configured to execute a computer program stored in the memory so that the apparatus performs the method according to any one of claims 1 to 9; A communication interface is coupled to the processor and is used to input or output information.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor is caused to perform the method according to any one of claims 1 to 9.

13. A computer program product, characterized in that When the computer program product is executed by a processor, the method according to any one of claims 1 to 9 is performed.

Images