Record management in time series databases

Abstract

The present disclosure relates to record management in time series databases. Computer- implemented methods, systems, and computer program products for record management in time series databases (TSDB) are disclosed. In a computer-implemented method for record management in a TSDB, time series records can be obtained by one or more processing units from distributed sources. The records can be divided by the one or more processing units into different record groups based on a timestamp of each record. The records in each record group can be normalized by the one or more processing units into normalized records. The normalized records can be stored by the one or more processing units in the TSDB.

Description

Background Technology

[0001] This disclosure relates to record management, and more specifically, to computer-implemented methods, systems, and computer program products for record management in time series databases (TSDBs).

[0002] A Time Series Database (TSDB) is a database that stores and manages time-stamped data, also known as a time series record or sample. Time series records can be measurements or events that are tracked or monitored over time, such as stock market prices, environmental temperatures, CPU usage, etc. There are many TSDBs, such as OpenTSDB, Prometheus, and InfluxDB. TSDBs can be applied to various fields, including financial trading systems, hardware monitoring systems, and environmental monitoring systems. Summary of the Invention

[0003] According to one embodiment of the present invention, a computer-implemented method for record management in a Time Series Database (TSDB) is provided. In this method, time series records can be obtained from a distributed source by one or more processing units. The records can be divided into different record groups by one or more processing units based on the timestamp of each record. Records in each record group can be normalized into normalized records by one or more processing units. The normalized records can be stored in the TSDB by one or more processing units. This method provides the benefit of managing distributed metrics in the TSDB. The managed distributed metrics maintain a mapping when tags are encoded to generate UUIDs.

[0004] In some embodiments, the method may include mapping recorded metrics and labels to normalized metrics and labels based on mappings between metrics and labels stored in a dictionary, and adapting recorded values ​​based on the mappings of metrics and labels. Furthermore, the mappings stored in the dictionary may include predefined mappings. Some embodiments of this disclosure may also include updating the mappings stored in the dictionary based on trends in recorded values. Mappings can improve the efficiency of managing and using distributed metrics.

[0005] In some embodiments of this disclosure, storing normalized records in a TSDB may include storing only one of the normalized records that have the same metric, the same label, and the same value as a shared record.

[0006] In some embodiments, the method may include sorting the normalized records in the time series using the same metric and the same label based on the timestamps of the normalized records. The timestamps of the normalized records provide benefits for the efficient management and use of distributed metrics.

[0007] In some embodiments of this disclosure, arbitrarily dividing records into different record groups may include grouping records with the same timestamp into the same record group, or grouping records sampled at the same time interval into the same record group, or grouping records sampled at time intervals with multiple relationships into the same record group.

[0008] In some embodiments of this disclosure, any storage of normalized records in the TSDB may include converting the key of the normalized record into a universally unique identifier (UUID). The key may include the metric and tag of the normalized record. Storing normalized records may also include storing the UUID, value, and timestamp of the normalized record in the TSDB. Furthermore, the method may also include maintaining a mapping between keys and UUIDs in the TSDB. The method may further include parsing a requested timestamp and a requested key with the requested metric and the requested tag from a request for a time-series record, converting the requested key into a requested UUID, and searching for the requested time-series record in the TSDB based on the requested UUID and the requested timestamp. Storage performed in this manner may preserve the mapping when the tag is encoded to produce a UUID and subsequently decoded.

[0009] In some embodiments of this disclosure, a method may include: parsing a requested timestamp and a requested key having a requested metric and a requested label from a request for a time series record; mapping the requested metric and the requested label to normalized requested metric and normalized requested label based on mappings between metrics and between labels stored in a dictionary; converting the normalized requested metric and normalized requested label to a requested UUID; and searching for the requested time series record in a TSDB based on the requested UUID and the requested timestamp.

[0010] The method may also include embodiments in which obtaining records from a distributed source includes pulling records from a distributed source or obtaining records pushed from a distributed source. Obtaining records in this way can effectively manage distributed metrics in the TSDB.

[0011] According to another embodiment of the present invention, a system for record management in a TSDB is provided. The system includes one or more processors, a memory coupled to at least one of the processors, and a set of computer program instructions stored in the memory. When executed by at least one of the processors, the set of computer program instructions performs the following actions: Time-series records can be obtained from a distributed source. Records can be divided into different record groups based on the timestamp of each record. Records in each record group can be normalized into normalized records. Normalized records can be stored in the TSDB.

[0012] According to another embodiment of this disclosure, a computer program product for record management in a TSDB is provided. The computer program product includes a computer-readable storage medium having program instructions embodied therein. The program instructions are executable by a processor to cause the processor to perform the following actions: Time-series records can be obtained from a distributed source; Records can be divided into different record groups based on the timestamp of each record; Records in each record group can be normalized into normalized records; Normalized records can be stored in the TSDB. Attached Figure Description

[0013] The above and other objects, features and advantages of this disclosure will become more apparent from the more detailed description of some embodiments thereof in the accompanying drawings, wherein like reference numerals in the embodiments of this disclosure generally refer to like parts.

[0014] Figure 1 A cloud computing node according to an embodiment of the present disclosure is described.

[0015] Figure 2 A cloud computing environment according to an embodiment of the present disclosure is described.

[0016] Figure 3 An abstract model layer according to an embodiment of this disclosure is described.

[0017] Figure 4 A flowchart of a record management method in a TSDB according to an embodiment of the present disclosure is shown.

[0018] Figure 5 An exemplary process is described in the steps of a method for record management according to embodiments of the present disclosure.

[0019] Figure 6 An exemplary process is described for another step in a method for record management according to embodiments of the present disclosure.

[0020] Figure 7 An illustrative example of a key-to-UUID conversion according to an embodiment of the present disclosure is described.

[0021] Figure 8 The process of searching for time series records in a TSDB according to an embodiment of the present disclosure is described.

[0022] Figure 9 Another method for searching time series records in a TSDB according to an embodiment of the present invention is described.

[0023] Figure 10 An illustrative example is shown of the result of a step in a method for record management according to an embodiment of the present disclosure.

[0024] Figure 11 A record management system in a TSDB according to another embodiment of the present invention is described. Detailed Implementation

[0025] Some embodiments will be described in more detail with reference to the accompanying drawings, in which embodiments of this disclosure are illustrated. However, this disclosure may be implemented in various ways and should therefore not be construed as being limited to the embodiments disclosed herein.

[0026] It should be understood that while this disclosure includes a detailed description of cloud computing, the implementation of the teachings set forth herein is not limited to cloud computing environments. Rather, embodiments of this disclosure can be implemented in conjunction with any other type of computing environment now known or hereafter developed.

[0027] Cloud computing is a service delivery model that enables convenient, on-demand access to a shared pool of configurable computing resources (e.g., a shared pool of configurable computing resources). These resources include networks, network bandwidth, servers, processing power, storage, applications, virtual machines, and services, which can be rapidly provisioned and released with minimal management effort or interaction with the service provider. This cloud model may include at least five features, at least three service models, and at least four deployment models.

[0028] The features are as follows:

[0029] On-demand self-service: Cloud consumers can unilaterally and automatically provide computing power, such as server time and network storage, as needed, without requiring human interaction with the service provider.

[0030] Extensive network access: Capabilities are available through networks and accessed via standard mechanisms that facilitate the use of heterogeneous thin client or thick client platforms (e.g., mobile phones, laptops, and PDAs).

[0031] Resource pooling: A provider's computing resources are pooled to serve multiple consumers using a multi-tenant model, where different physical and virtual resources are dynamically assigned and reassigned as needed. There is a sense of location independence because consumers typically do not have control or knowledge of the exact location of the resources provided, but may be able to specify the location at a higher level of abstraction (e.g., country, state, or data center).

[0032] Rapid flexibility: The ability to provide capacity quickly and flexibly, automatically scaling down and up rapidly in some situations to scale up rapidly. For consumers, the available supply capacity often appears unlimited and can be purchased in any quantity at any time.

[0033] Measuring services: Cloud systems automatically control and optimize resource usage by leveraging metering capabilities at a level of abstraction appropriate to the service type (e.g., storage, processing, bandwidth, and active user accounts). Resource usage can be monitored, controlled, and reported, providing transparency to both service providers and consumers.

[0034] The service model is as follows:

[0035] Software as a Service (SaaS): This provides consumers with the ability to use the provider's applications running on cloud infrastructure. Applications can be accessed from different client devices via thin client interfaces such as web browsers (e.g., web-based email). Consumers do not manage or control the underlying cloud infrastructure, including the network, servers, operating system, storage, or even individual application capabilities, with possible exceptions such as limited user-specific application configuration settings.

[0036] Platform as a Service (PaaS): This provides consumers with the ability to deploy applications created or acquired by the consumer using programming languages ​​and tools supported by the provider onto cloud infrastructure. Consumers do not manage or control the underlying cloud infrastructure, including networks, servers, operating systems, or storage, but they have control over the deployed applications and the configuration of any application hosting environment.

[0037] Infrastructure as a Service (IaaS): The capabilities offered to consumers are processing, storage, networking, and other basic computing resources that enable consumers to deploy and run arbitrary software, which may include operating systems and applications. Consumers do not manage or control the underlying cloud infrastructure, but rather have control over the operating system, storage, deployed applications, and potentially limited control over selected networking components (e.g., host firewalls).

[0038] The deployment model is as follows:

[0039] Private cloud: A cloud infrastructure that operates solely for an organization. It can be managed by the organization or a third party and can exist on-site or off-site.

[0040] Community cloud: A cloud infrastructure shared by several organizations and supporting a specific community with shared concerns (e.g., tasks, security requirements, policies, and compliance considerations). It can be managed by an organization or a third party and can exist on-site or off-site.

[0041] Public cloud: Makes cloud infrastructure available to the public or large industry groups and is owned by an organization that sells cloud services.

[0042] Hybrid cloud: A cloud infrastructure is a combination of two or more clouds (private, community, or public) that remain a single entity but are bound together by normalization or proprietary technologies that enable data and applications to be portable (e.g., cloud bursting for load balancing between clouds).

[0043] Cloud computing environments are service-oriented, focusing on statelessness, loose coupling, modularity, and semantic interoperability. At the heart of cloud computing is the infrastructure comprising a network of interconnected nodes.

[0044] See now Figure 1 The diagram illustrates an example of a cloud computing node. Cloud computing node 10 is merely one example of a suitable cloud computing node and is not intended to impose any limitation on the scope or functionality of the embodiments described herein. In any case, cloud computing node 10 can be implemented and / or perform any of the functions set forth above.

[0045] Within cloud computing node 10, there exists a computer system / server 12 or a portable electronic device such as a communication device, which can operate alongside many other general-purpose or special-purpose computing system environments or configurations. Examples of known computing systems, environments, and / or configurations that may be suitable for computer system / server 12 include, but are not limited to, personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputer systems, mainframe computer systems, and distributed cloud computing environments that include any of the aforementioned systems or devices.

[0046] Computer system / server 12 can be described in the general context of computer system executable instructions (e.g., program modules) executed by the computer system. Generally, program modules may include routines, programs, objects, components, logic, data structures, etc., that perform specific tasks or implement specific abstract data types. Computer system / server 12 can be implemented in a distributed cloud computing environment, where tasks are performed by remote processing devices linked via a communication network. In a distributed cloud computing environment, program modules can reside in local and remote computer system storage media, including memory storage devices.

[0047] like Figure 1 As shown, the computer system / server 12 in cloud computing node 10 is illustrated in the form of a general-purpose computing device. The components of the computer system / server 12 may include, but are not limited to, one or more processors or processing units 16, system memory 28, and a bus 18 that couples the various system components, including the system memory 28, to the processor 16.

[0048] Bus 18 represents any one or more of several types of bus architectures, including memory buses or memory controllers, peripheral buses, accelerated graphics ports, and processor or local buses using any of the various bus architectures. By way of example and not limitation, such architectures include Industry Standard Architecture (ISA) buses, Micro Channel Architecture (MCA) buses, Enhanced ISA (EISA) buses, Video Electronics Standards Association (VESA) local buses, and Peripheral Component Interconnect (PCI) buses.

[0049] Computer system / server 12 typically includes various computer system readable media. Such media can be any available media that can be accessed by computer system / server 12, and includes volatile and non-volatile media, removable and non-removable media.

[0050] System memory 28 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache memory 32. Computer system / server 12 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, storage system 34 may be provided for reading from and writing to non-removable, non-volatile magnetic media (not shown, and generally referred to as "hard disk drives"). Although not shown, disk drives for reading from or writing to removable non-volatile disks (e.g., "floppy disks") and optical disk drives for reading from or writing to removable non-volatile optical disks (such as CD-ROMs, DVD-ROMs, or other optical media) may be provided. In such cases, each may be connected to bus 18 via one or more data media interfaces. As will be further described and depicted below, memory 28 may include at least one program product having at least one set of program modules configured to perform the functions of embodiments disclosed herein.

[0051] A program / utility 40 having a set (at least one) of program modules 42, along with an operating system, one or more applications, other program modules, and program data, may be stored in memory 28 by way of example and not limitation. Each or some combination of the operating system, one or more applications, other program modules, and program data may include an implementation of a network environment. Program modules 42 typically perform functions and / or methods as described in embodiments of this disclosure herein.

[0052] Computer system / server 12 can also communicate with one or more external devices 14 (e.g., keyboard, pointing device, display 24, etc.); and / or any device that enables computer system / server 12 to communicate with one or more other computing devices (e.g., network interface card, modem, etc.). This communication can be performed via input / output (I / O) interface 22. Furthermore, computer system / server 12 can communicate with one or more networks, such as local area networks (LANs), general area networks (WANs), and / or public networks (e.g., the Internet), via network adapter 20. As shown, network adapter 20 communicates with other components of computer system / server 12 via bus 18. It should be understood that, although not shown, other hardware and / or software components can be used in conjunction with computer system / server 12. Examples include, but are not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data archiving storage systems.

[0053] See now Figure 2 This describes an illustrative cloud computing environment 50. As shown, the cloud computing environment 50 includes one or more cloud computing nodes 10 to which local computing devices used by cloud consumers can communicate. These local computing devices include, for example, personal digital assistants (PDAs) or cellular phones 54A, desktop computers 54B, laptop computers 54C, and / or automotive computer systems 54N. The nodes 10 can communicate with each other. They can be physically or virtually grouped (not shown) in one or more networks, such as private clouds, community clouds, public clouds, or hybrid clouds, or combinations thereof, as described above. This allows the cloud computing environment 50 to provide infrastructure, platforms, and / or software as services that cloud consumers do not need to maintain on their local computing devices. It should be understood that... Figure 2 The types of computing devices 54A-N shown are intended to be illustrative only, and computing node 10 and cloud computing environment 50 can communicate with any type of computerized device via any type of network and / or network-addressable connectivity (e.g., using a web browser).

[0054] See now Figure 3 This demonstrates a cloud computing environment of 50 ( Figure 2 This provides a set of functional abstractions. It should be understood beforehand that... Figure 3 The components, layers, and functions shown are intended to be illustrative only, and embodiments of this disclosure are not limited thereto. As described, the following layers and corresponding functions are provided:

[0055] The hardware and software layer 60 includes hardware and software components. Examples of hardware components include: a mainframe 61; a RISC (Reduced Instruction Set Computer) based server 62; a server 63; a blade server 64; a storage device 65; and network and networking components 66. In some embodiments, software components include network application server software 67 and database software 68.

[0056] The virtualization layer 70 provides an abstraction layer from which the following examples of virtual entities can be provided: virtual server 71; virtual storage 72; virtual network 73, including virtual private network; virtual application and operating system 74; and virtual client 75.

[0057] In one example, management layer 80 may provide the following functionalities: Resource Provisioning 81 provides dynamic procurement of computing resources and other resources used to perform tasks within the cloud computing environment. Metering and Pricing 82 provides cost tracking as resources are utilized within the cloud computing environment and bills or invoices for the consumption of these resources. In one example, these resources may include application software licenses. Security provides authentication for cloud consumers and tasks, as well as protection for data and other resources. User Portal 83 provides access to the cloud computing environment for consumers and system administrators. Service Level Management 84 provides cloud resource allocation and management to ensure that required service levels are met. Service Level Agreement (SLA) Planning and Fulfillment 85 provides pre-scheduling and procurement of cloud resources, anticipating future requirements for those resources according to the SLA.

[0058] Workload layer 90 provides examples of functionalities that can be leveraged in a cloud computing environment. Examples of workloads and functionalities that can be provided from this layer include: mapping and navigation 91; software development and lifecycle management 92; virtual classroom education delivery 93; data analytics and processing 94; transaction processing 95; and records management 96.

[0059] Figure 3 The functional abstraction layer shown is merely an example. One or more layers can be added as needed, and they can be merged or omitted. Figure 3 One or more layers within. Furthermore, in Figure 3 In each layer, some components can be omitted or combined, and one or more components can be added.

[0060] TSDB can store and manage time-series records, which can be timestamped records collected from a sampling source. The format of a time-series record can be: <key><timestamp><value>, which is a key-value pair with a timestamp. The key can include a metric reflecting the record's attributes and a label reflecting the metric's dimensional characteristics, and can be represented as <metric>{<label>}. A label can include one or more sub-labels, each consisting of a sub-label name and a sub-label value. In this respect, the key can also be represented as <metric>{<sublabel name 1> = <sublabel value 1>, <sublabel name 2> = <sublabel value 2>…}. For example, the key can be represented as http_request_total{method='POST',status='200'}, which represents the total number of HTTP requests with the request type POST and a response status of 200.

[0061] The following are four illustrative examples of time series records.

[0062] 1cpu_use{ip='10.27.120.7',id='3'}948258276 14.5%

[0063] 2cpu_use{ip='10.27.120.7',id='3'}948258286 14.0%

[0064] 3http_request_total{status='200',method='GET'}948258296 94355

[0065] 4meterCurrent{device='d01',location='bj'}948278275 10.2

[0066] For the first record, the key is `cpu_use{ip='10.27.120.7', id='3'}`, which consists of the metric `cpu_use` and the tag `{ip='10.27.120.7', id='3'}`; the timestamp is 948258276, and the value is 14.5%. For the second record, the key is `cpu_use{ip='10.27.120.7', id='3'}`, the same as the first record. This means that both records point to the same metric (i.e., CPU usage) for the same server or host (with the IP 10.27.120.7 and the ID 3); the timestamp is 948258286, and the value is 14.0%. For the third record, the key is...

[0067] The record `http_request_total{status='200',method='GET'}` consists of the metric `http_request_total` and the tag `{status='200',method='GET'}`; the timestamp is 948258296; and the value is 94355. For the fourth record, the key is `meterCurrent{device='d01',location='bj'}`, which consists of the metric `meterCurrent` and the tag `{device='d01',location='bj'}`; the timestamp is 948278275; and the value is 10.2.

[0068] Time series records can be sampled from various sampling sources across a variety of targets, and some records from different sampling sources can represent the same attribute of the same sampling target. For example, multiple applications or servers running on a physical machine can sample records of attributes of that machine separately, such as CPU usage and IP address. Therefore, different sampling sources (e.g., different applications and / or servers) can sample records representing the same attribute (e.g., CPU usage and / or IP address of the same sampling target, such as the physical machine). As another example, processes or threads of an application running on a physical machine can sample records of application and / or physical machine attributes separately, and virtual machines on a host can sample records of host attributes separately. Therefore, there may be a large number of duplicate or redundant records, which increases storage costs. On the other hand, the metrics and labels of records from different sampling sources can be named differently. Therefore, TSDB needs to store and manage records with all kinds of metrics and labels, leading to inefficient record management and use.

[0069] The embodiments of this disclosure aim to solve at least one of the above-described technical problems, and provide a method, system, and computer program product for record management in a TSDB, enabling the unification of metric and tag naming, and thus enabling the effective management and use of records in a TSDB.

[0070] Now for reference Figure 4 The diagram illustrates a flowchart of a record management method 400 in a TSDB according to an embodiment of the present disclosure. Method 400 may be implemented by one or more processing units, for example, by... Figure 1 The computer system / server 12 is implemented.

[0071] In step 410, time-series records can be obtained from a distributed source. The distributed source can be different applications, servers, processes, threads, and / or virtual machines as described above. TSDB can obtain records in different ways. In some embodiments, step 410 may include at least one of retrieving records from a distributed source and obtaining records pushed from the distributed source. For example, records can be retrieved from an application server or API server, records can be pushed by an exporter (a data acquisition component in an existing TSDB), or records can be retrieved from service discovery in an existing TSDB that dynamically discovers targets to be sampled or monitored.

[0072] In step 420, records can be grouped into different groups based on the timestamp of each record. This is to classify potentially related records. The timestamp of a record can reflect the time when the record was sampled; the timestamps of multiple records with the same key and sampled by the same sampling source can reflect the sampling interval or sampling rate. For example, if a record of the city's temperature sampled by a first sampling source has a timestamp of 11:00 AM on a day, a subsequent record of the city's temperature sampled by the first sampling source has a timestamp of 12:00 PM on the same day, and a previous record of the city's temperature sampled by the first sampling source has a timestamp of 10:00 AM on the same day, it can be concluded that the sampling interval for the city's temperature by the first sampling source is 1 hour. Records sampled at the same time or at the same time interval can be related. For example, records sampled by the first sampling source at 1-hour time intervals and records sampled by the second sampling source at 1-hour time intervals can be related because they can both be records of the city's temperature. On the other hand, records sampled by different sampling sources at different times or time intervals may be less related. For example, records sampled by a first sampling source at 10-minute intervals and records sampled by a second sampling source at 3-minute intervals may be unrelated, because the former could be a record of website clicks, while the latter could be a record of CPU usage. Furthermore, records sampled at the first interval can be related to records sampled at a second interval that has multiple relationships with the first interval. For example, records sampled by a first sampling source at 5-minute intervals and records sampled by a second sampling source at 10-minute intervals (twice the length of a 5-minute interval) can be related because they could be records of voltage for electrical components. Therefore, there are various ways to divide records into different groups based on the timestamp of each record. For example, a first method is to group records with the same timestamp into the same group, which can be done directly by referencing the timestamps. A second method is to group records sampled at the same time interval into the same group. For example, records sampled every 5 minutes from the first sampling source and records sampled every 5 minutes from the second sampling source can be grouped into the same group. A third method is to group records sampled at time intervals that have multiple relationships into the same group. For example, records sampled every 5 minutes from a first sampling source and records sampled every 15 minutes from a second sampling source can be grouped into the same record group. The 15-minute time interval has a triple relationship with the 5-minute time interval. As another example, records sampled every 5 minutes from a first sampling source, records sampled every 15 minutes from a second sampling source, records sampled every 5 minutes from a third sampling source, and records sampled every 60 minutes from a fourth sampling source can be grouped into the same record group.

[0073] In step 430, the records in each record group can be normalized into normalized records. This step aims to standardize the measurement of records and the naming of labels. Figure 5 The present disclosure illustrates how to normalize records in each record group into normalized records according to an embodiment of the present disclosure.

[0074] According to embodiments of the present invention, a mapping relationship between metrics and tags can be stored in a dictionary within the TSDB, and this mapping relationship can be used for normalization processing. The dictionary can store mapping relationships indicating the correspondence between each normalized metric and its associated unnormalized metric, and the correspondence between each normalized tag and its associated unnormalized tag. Therefore, for each record group, in step 510, the metrics and tags recorded in that record group can be mapped to normalized metrics and normalized tags based on the mapping relationship between metrics and tags stored in the dictionary. In other words, if the metrics and tags of a record are different from their corresponding normalized metrics and tags, the metrics and tags will be converted to their corresponding normalized metrics and tags. For example, the dictionary can store a mapping relationship indicating that the metrics http_requests_total and httprequest_total can be mapped to the normalized metric http_request_total, so that both the metrics http_requests_total and httprequest_total can be mapped to the normalized metric http_request_total based on this mapping relationship. For example, a dictionary can store a mapping that indicates a measured temperature_Fahrenheit can be mapped to a normalized measured temperature_Celsius, and thus a measured temperature_Fahrenheit can be mapped to a normalized measured temperature_Celsius based on that mapping.

[0075] After mapping the measurement and label of the record, at step 520, the value of the record can be adapted based on the mapping of the measurement and label. For example, in the example above of mapping the measured temperature_Fahrenheit to the normalized measured temperature_Celsius, the value of the corresponding record needs to be converted based on the equation C = (F-32) / 1.8, where C represents the value in Celsius and F represents the value in Fahrenheit. In another example, if the measurement or label of the record is mapped from a decimal representation to a binary representation, then the value also needs to be converted accordingly.

[0076] In some embodiments, the mappings stored in the dictionary may include predefined mappings. These predefined mappings can be generated based on knowledge of known naming conventions. In some embodiments, the mappings stored in the dictionary can be updated based on the trends in the values ​​of the obtained records. If the values ​​of a first set of records with a first key and the values ​​of a second set of records with a second key have similar trends (i.e., changes over time), then the first key and the second key may be different expressions of the same meaning. Therefore, by observing the trends in the values ​​of the records, different naming conventions for measures and labels with the same meaning can be detected and added to the dictionary. Thus, mappings between measures of the first key and measures of the second key, as well as mappings between labels of the first key and labels of the second key, can be added to the dictionary. It should be noted that updating the mappings stored in the dictionary also includes building the dictionary from scratch, without saving the predefined mappings. The updating of the mappings can be performed before step 430.

[0077] After the normalization in step 430, measures with different representations but the same meaning can be unified into a normalized measure, and labels with different representations but the same meaning can be unified into normalized labels. This greatly reduces the number of measures and labels that should be managed in the TSDB. Then, in step 440, the normalized records can be stored in the TSDB.

[0078] After normalization, duplicate or redundant records with the same metric, label, and value can be identified. As mentioned above, there may be a large number of duplicate or redundant records. To save storage space, duplicate or redundant records may be stored in only one copy. Accordingly, in some embodiments, step 440 may include storing only one of the normalized records with the same metric, label, and value as a shared record, thereby reducing the need for storage space.

[0079] In some embodiments, to reduce storage space requirements, instead of storing the metrics and tags of normalized records, a universally unique identifier (UUID) converted from the metrics and tags can be stored. Metrics and tags are typically long strings, while UUIDs can be shorter. Figure 6 The process of storing normalized records in a TSDB according to an embodiment of this disclosure is shown. For each normalized record, in step 610, the key of the normalized record can be converted into a UUID. As described above, the key of the normalized record may include the metric and tag of the normalized record. As an example, a hash function such as Message Digest Algorithm 5 (MD5) can be used for the conversion. As another example, a lookup table can be built in the TSDB for the conversion. Figure 7 This illustrates a conversion from a key to a UUID. Figure 7As can be seen, the long string of keys can be converted into a much shorter representation format. After the conversion, in step 620, the UUID, value, and timestamp of the normalized record can be stored in the TSDB. In some embodiments, the mapping relationship between keys and UUIDs can also be stored in the TSDB, so that the key of the normalized record can be retrieved based on a certain UUID. Furthermore, normalized records can be retrieved based on UUIDs.

[0080] In some embodiments, to improve write performance, step 440 may include using a Log Structured Merge (LSM) tree for storage.

[0081] In some embodiments, a user can access the TSDB, for example, by submitting a request for a time-series record. A request for a time-series record may include a requested key for the requested record. The requested key can be converted to a requested UUID, and the requested UUID can be compared with UUIDs stored in the TSDB to search for the requested time-series record. Method 400 may further include the step of searching for time-series records in the TSDB.

[0082] Figure 8 The process 800 for searching time-series records in a TSDB is illustrated. In step 810, a requested timestamp and a requested key can be parsed from the request for a time-series record. The requested key may include a requested metric and a requested tag. The parsing of the requested timestamp and the requested key can be achieved using any suitable means for extracting information from structured or unstructured text. For example, any kind of tokenizer or solution from natural language processing techniques can be used. At step 820, the requested key can be converted into a requested UUID. For example, the same hash function or lookup table used in step 610 can be used in the conversion from the requested key to the requested UUID. In step 830, the requested time-series record can be searched in the TSDB based on the requested UUID and the requested timestamp. As described in step 620, each normalized record can be stored in the TSDB with a UUID, a value, and a timestamp. By using the requested UUID and the requested timestamp, if they exist, a time-series record with the same UUID and the same timestamp can be found in TSDB as the requested record.

[0083] In some cases, a match may not be found in the TSDB for a UUID directly converted from the metric and label requested in the request, because the TSDB may only store UUIDs corresponding to the normalized metric and normalized label. Therefore, in some implementations, method 400 may further include Figure 9The process 900 shown describes searching for time-series records in TSDB, where the request key is mapped to a normalized request metric and a normalized request tag before being converted to the request's UUID. (Example...) Figure 9 As shown, in step 910, the request timestamp and request key can be parsed from the request of the time series record. The requested key may include the requested metric and the requested tag. At step 920, the requested metric and the requested tag can be mapped to normalized requested metric and normalized requested tag based on the mapping between metrics and tags stored in a dictionary. The dictionary used here is the same as the dictionary used in step 510. In step 930, the normalized request metric and the normalized request tag can be converted into a requested UUID. For example, the same hash function or lookup table used in step 610 can be used in the conversion from the requested key to the requested UUID. In step 940, the requested time series record can be searched in the TSDB based on the requested UUID and the requested timestamp. As described in step 620, each normalized record can be stored in the TSDB with a UUID, a value, and a timestamp. By using the requested UUID and the requested timestamp, if they exist, a time-series record with the same UUID and the same timestamp can be found in TSDB as the requested record.

[0084] In some embodiments, records with the same key (i.e., the same metric and the same label) obtained from different sources can be sampled at different times, and these records can therefore be sorted chronologically to form a list of records with more complete data for the metric and label. Therefore, in some embodiments, method 400 may further include sorting the normalized records chronologically with the same metric and the same label based on the timestamps of the normalized records. Figure 10An illustrative example of the sorting results is described. Normalized records from source A can be records with a series of timestamps such as 10:00, 10:03, 10:06, ... . Normalized records from source B can be records with a series of timestamps such as 10:01, 10:04, 10:07, ... . Normalized records from source C can be records with a series of timestamps such as 10:02, 10:05, 10:08, ... . After sorting, a list of records with a series of timestamps 10:00, 10:01, 10:02, 10:03, 10:04, ... can be created. Therefore, automatic filling of data with the same metric and the same label can be achieved. It should be noted that the sorting step can be performed before storage step 440, or after search step 830 or search step 940. When the sorting step is performed before storage step 440, records with the same key can be stored in a timeline manner. When the sorting step is performed after search steps 830 or 940, a series of records can be output to the user in chronological order.

[0085] In view of the above, the embodiments of this disclosure can achieve unified naming of metrics and tags, and can further achieve deduplication of records, and / or efficient storage, searching, and / or sorting of records. Therefore, the embodiments of the present invention can provide effective management of records in TSDB, improve the operational performance of TSDB, and / or reduce the storage space required for TSDB.

[0086] Now for reference Figure 11 This image illustrates a system 1100 for record management in a Time Series Database (TSDB) according to an embodiment of the present invention. System 1100 may include one or more processors 1110 and a memory 1120 coupled to at least one of the processors 1110. A set of computer program instructions is stored in the memory 1120. When executed by at least one of the processors 1110, the set of computer program instructions performs the following series of actions for record management in the TSDB. Time series records may be obtained from a distributed source. Records may be divided into different record groups based on the timestamp of each record. Records in each record group may be normalized into normalized records. Normalized records may be stored in the TSDB.

[0087] In some embodiments, records can be pulled from a distributed source and / or pushed from a distributed source.

[0088] In some embodiments, records with the same timestamp can be grouped into the same record group, records sampled at the same time interval can be grouped into the same record group, or records sampled at time intervals with multiple relationships can be grouped into the same record group.

[0089] In some embodiments, the recorded metrics and labels can be mapped to normalized metrics and normalized labels based on mappings between metrics and between labels stored in a dictionary. The recorded values ​​can be adapted based on this mapping of metrics and labels.

[0090] In some embodiments, the mappings stored in the dictionary include predefined mappings.

[0091] In some embodiments, the mapping relationships stored in the dictionary can be updated based on the trend of the values ​​of the records before normalization.

[0092] In some embodiments, for each normalized record, the key of the normalized record can be converted into a universally unique identifier (UUID), where the key includes the metric and label of the normalized record. The UUID, value, and timestamp of the normalized record can be stored in a TSDB.

[0093] In some embodiments, the mapping between keys and UUIDs can be stored in TSDB.

[0094] In some embodiments, only one of the normalized records having the same metric, the same label, and the same value may be stored as a shared record.

[0095] In some embodiments, a requested timestamp and a requested key, including the requested metric and the requested label, can be parsed from a request for a time series record. The requested key can be converted into a requested UUID. The requested time series record can be searched in the TSDB based on the requested UUID and the requested timestamp.

[0096] In some embodiments, a requested timestamp and a requested key including the requested metric and the requested label can be parsed from a request for a time series record. The requested metric and the requested label can be mapped to normalized requested metrics and normalized requested labels based on mappings between metrics and between labels stored in a dictionary. The normalized requested metric and normalized requested label can be converted into a requested UUID. The requested time series record can be searched in the TSDB based on the requested UUID and the requested timestamp.

[0097] In some embodiments, normalized records with the same metric and the same label can be ordered chronologically based on the timestamps of the normalized records.

[0098] The above description of the process of method 400 can be applied to system 1000. For the sake of brevity, details are omitted here.

[0099] It should be noted that the processing implemented by the record management system in the TSDB according to an embodiment of the present invention can be performed by... Figure 1 The computer system / server 12 is implemented.

[0100] According to another embodiment of this disclosure, a computer program product for record management in a TSDB is disclosed. The computer program product includes a computer-readable storage medium having program instructions contained therein, and the program instructions are executable by a processor. When executed, the program instructions cause the processor to perform one or more of the processes described above.

[0101] It should also be noted that the terms “first” and “second” as used herein are used only for the purpose of distinguishing one element from another, and do not indicate the order and / or importance of the corresponding elements.

[0102] The sequence of actions described in conjunction with the accompanying drawings is merely exemplary and should not be construed as limiting this disclosure. The sequence of actions can be modified as necessary. Furthermore, one or more actions may be omitted, and more actions may be added.

[0103] Various aspects of this disclosure are described by narrative text, flowcharts, block diagrams of computer systems, and / or block diagrams of machine logic included in embodiments of a computer program product (CPP). Regarding any flowchart, depending on the technology involved, operations may be performed in a different order than that shown in a given flowchart. For example, also depending on the technology involved, two operations shown in successive flowchart blocks may be performed in reverse order, as a single integrated step, simultaneously, or in a manner that at least partially overlaps in time.

[0104] Computer program product embodiment (“CPP embodiment” or “CPP”) is a term used in this disclosure to describe any collection of one or more storage media (also referred to as “media”) commonly included in a collection of one or more storage devices, which commonly include machine-readable code corresponding to instructions and / or data for performing the computer operations specified in a given CPP claim. A “storage device” is any tangible device capable of holding and storing instructions used by a computer processor. Computer-readable storage media can be electronic storage media, magnetic storage media, optical storage media, electromagnetic storage media, semiconductor storage media, mechanical storage media, or any suitable combination of the foregoing, but is not limited thereto. Some known types of storage devices that include these media include: disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), compact disk read-only memory (CD-ROM), digital universal disk (DVD), memory sticks, floppy disks, mechanical encoding devices (such as punch cards or pits / platforms formed in the main surface of the disk), or any suitable combination of the foregoing. As used in this disclosure, computer-readable storage medium should not be construed as storing transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides, optical pulses through fiber optic cables, electrical signals transmitted through wires, and / or other transmission media. As those skilled in the art will understand, during normal operation of a storage device (such as during access, defrauding, or garbage collection), data is typically moved at some occasional points in time, but this does not make the storage device transient, because the data is not transient when it is stored.

[0105] This document describes aspects of the disclosure with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the disclosure. It should be understood that each block of the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer-readable program instructions.

[0106] These computer-readable program instructions may be provided to a processor of a computer or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / actions specified in one or more blocks of a flowchart and / or block diagram. These computer-readable program instructions may also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner, such that the computer-readable storage medium storing the instructions includes an article of manufacture containing instructions that implement aspects of the functions / actions specified in one or more blocks of a flowchart and / or block diagram.

[0107] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus, or other device to produce computer-implemented processing, such that the instructions executed on the computer, other programmable apparatus, or other device perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0108] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. Each block in a flowchart or block diagram may represent a module, segment, or portion of instructions, including one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than indicated in the figures. For example, two consecutively shown blocks may actually be completed as a single step, executed simultaneously, substantially simultaneously, or with partial or complete temporal overlap, or the blocks may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action or executes a combination of dedicated hardware and computer instructions.

[0109] Various embodiments of this disclosure have been described for illustrative purposes, but are not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein has been chosen to best explain the principles of the embodiments, their practical application, or technical improvements to technologies found in the market, or to enable those skilled in the art to understand the embodiments disclosed herein.

Claims

1. A computer-implemented method for record management in a time-series database, comprising: Time-series records are obtained from a distributed source by one or more processing units, wherein the records include timestamps, metrics, and labels; One or more processing units divide the records into different record groups based on the timestamp of each record. One or more processing units normalize the records in each record group into normalized records, wherein normalization includes mapping the measure and label of the record to normalized measure and normalized label based on the mapping relationship stored in a dictionary; as well as Normalized records are stored in a time-series database through one or more processing units.

2. The computer-implemented method according to claim 1, further comprising: The value of the record is adapted by one or more processing units based on the mapping between the metric and the label.

3. The computer-implemented method according to claim 1, wherein, The mapping relationships stored in the dictionary include predefined mapping relationships.

4. The computer-implemented method according to claim 2, further comprising: The mapping relationship stored in the dictionary is updated based on the trend of the recorded values.

5. The computer-implemented method according to claim 1, wherein, Storing the normalized records in the time series database includes: One or more processing units store only one of the normalized records that have the same metric, the same label, and the same value as a shared record.

6. The computer-implemented method according to claim 1, further comprising: One or more processing units sort the normalized records in the time series using the same metric and the same label, based on the timestamps of the normalized records.

7. The computer-implemented method according to claim 1, wherein, Dividing the records into different record groups includes one of the following: Records with the same timestamp are grouped into the same record group by one or more processing units; One or more processing units group records sampled at the same time interval into the same record group; as well as Records sampled at time intervals with multiple relationships are grouped into the same record group by one or more processing units.

8. The computer-implemented method according to claim 1, wherein, Storing the normalized records in the time series database includes: for each normalized record... One or more processing units convert the keys of the normalized records into universally unique identifiers, wherein the keys include the metric and label of the normalized record; and One or more processing units store the universally unique identifiers, values, and timestamps of the normalized records in the time series database.

9. The computer-implemented method according to claim 8, further comprising: One or more processing units store the mapping relationship between keys and universally unique identifiers in the time series database.

10. The computer-implemented method according to claim 8, further comprising: One or more processing units parse the timestamp of the request and the key of the request, including the metric and tag of the request, from the request for time series recording. One or more processing units convert the request key into a universally unique identifier for the request; as well as One or more processing units search for the requested time series records in the time series database based on the requested universally unique identifier and the requested timestamp.

11. The computer-implemented method according to claim 8, further comprising: One or more processing units parse the timestamp of the request and the key of the request, including the metric and tag of the request, from the request for time series recording. One or more processing units map the request's metrics and tags to normalized request metrics and tags based on the mapping relationships between metrics and tags stored in a dictionary; One or more processing units convert the normalized request metric and the normalized request label into a universally unique identifier for the request. as well as One or more processing units search for the requested time series records in the time series database based on the requested universally unique identifier and the requested timestamp.

12. The computer-implemented method according to claim 1, wherein, Obtaining time series records from a distributed source includes at least one of the following: The records are pulled from the distributed source by one or more processing units; and The records pushed from the distributed source are obtained by one or more processing units.

13. A system for record management in a time-series database, comprising: One or more processors; A memory coupled to at least one of the processors; A set of computer program instructions stored in the memory, when executed by at least one of the processors, performs the following actions: Time-series records are obtained from a distributed source, wherein the records include timestamps, metrics, and labels; The records are divided into different record groups based on the timestamp of each record. The records in each of the record groups are normalized into normalized records, wherein normalization includes mapping the measure and label of the record to a normalized measure and a normalized label based on a mapping relationship stored in a dictionary; as well as Normalized records are stored in a time series database.

14. The system according to claim 13, wherein, The set of computer program instructions further performs the following actions: The value of the record is adapted based on the mapping between the metric and the label.

15. The system according to claim 14, wherein, The set of computer program instructions performs the following actions: The mapping relationship stored in the dictionary is updated based on the trend of the recorded values.

16. The system according to claim 13, wherein, Storing the normalized records in a time series database includes: Only one of the normalized records that has the same metric, the same label, and the same value is stored as a shared record.

17. The system according to claim 13, wherein, The set of computer program instructions further performs the following actions: Based on the timestamps of the normalized records, the normalized records are sorted in the time series using the same metric and the same label.

18. The system according to claim 13, wherein, Dividing the records into different record groups includes one of the following: Group records with the same timestamp into the same record group; Records sampled at the same time intervals are grouped into the same record group; as well as Records sampled at time intervals with multiple relationships are grouped into the same record group.

19. The system of claim 13, wherein storing normalized records in the time series database comprises: For each normalized record, The key of the normalized record is converted into a universally unique identifier, wherein the key includes the measure and label of the normalized record; and The universally unique identifier, value, and timestamp of the normalized record are stored in the time series database.

20. A computer program product for record management in a time series database, the computer program product comprising program instructions executable by a processor to cause the processor to perform the method steps as described in any one of claims 1-12.

Images