Method and related device for solving OpenTSDB data hotspots
By reconstructing the metrics of OpenTSDB data, generating new metrics, and re-storing them, we solved the data hotspot problem, improved query and storage efficiency under high concurrency, and achieved balanced data distribution.
Patent Information
- Application Number
- CN202510723003.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-05-30
- Publication Date
- 2025-09-12
AI Technical Summary
When OpenTSDB stores data, the same measurement points are concentrated on the same machine, resulting in data hotspots. This affects query and storage efficiency under high-concurrency requests. The existing bucketing strategy cannot completely solve this problem.
The original metric of the time series data is reconstructed to generate a new metric, which is encrypted using MD5 and the first 5 digits are used as a prefix. The metric is then stored back in the OpenTSDB database to ensure balanced data distribution.
This effectively avoids data hotspots, improves the query and storage efficiency of OpenTSDB in high-concurrency situations, and evenly distributes data to every machine in the cluster.
Smart Images
Figure CN120632166A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of time series data storage and relates to a method and related device for resolving OpenTSDB data hotspots. Background Art
[0002] OpenTSDB is a distributed, scalable time series database based on HBase, supporting second-level data collection. As an open-source, distributed time series database, it is widely used in the industrial field to store monitoring, alarms, IoT, and indicators.
[0003] OpenTSDB data storage is stored in order after being sorted by RowKey, which may cause the same measurement points on different devices to be stored on the same machine. If the measurement point is a frequently used measurement point, it will eventually cause a small number of machines in the cluster to store most of the hot data (such as Figure 1 This directly impacts storage and query efficiency under high-concurrency conditions, and can even lead to service failures in business systems. OpenTSDB's built-in bucketing strategy can alleviate this issue to some extent, but it cannot fundamentally resolve the problem. Furthermore, setting an unreasonable number of buckets can actually increase cluster pressure. Therefore, designing OpenTSDB's distributed data storage is crucial for improving cluster query and storage efficiency. Summary of the Invention
[0004] The purpose of the present invention is to overcome the shortcomings of the above-mentioned prior art and provide a method and related device for solving OpenTSDB data hotspots. This method and related device can improve the query and storage efficiency of OpenTSDB under high concurrency conditions.
[0005] To achieve the above objectives, the present invention discloses a method for resolving OpenTSDB data hotspots, comprising:
[0006] Get the original metric and all tag information of time series data;
[0007] Reconstruct the original metric of the time series data to obtain a new metric of the time series data;
[0008] Write the new metric and original tag information, timestamp, and value of the time series data into the OpenTSDB database.
[0009] The method for solving OpenTSDB data hotspots described in the present invention is further improved in that:
[0010] Furthermore, the process of reconstructing the original metric of the time series data to obtain a new metric of the time series data is as follows:
[0011] The Metric Reconstructor is used to combine all tag values into a string, which is then encrypted with MD5 to obtain an encrypted string. The first 5 digits of the encrypted string are used as a prefix and appended to the front of the Metric to form a new Metric for the time series data.
[0012] Furthermore, the process of writing the new metric and the original tag information, timestamp, and value of the time series data into the OpenTSDB database is as follows:
[0013] Call the data writing service, obtain the new metric from the metric reconstructor, and write the new metric and the original tag information, timestamp, and value into the OpenTSDB database.
[0014] Furthermore, it also includes:
[0015] Receive a user's request for querying data, which includes metric, tag information and time range, call the metric reconstructor to generate a new metric for the request based on the metric and tag information in the request, call the OpenTSDB database API based on the new metric and the time range to obtain data, and return it to the user.
[0016] The present invention discloses a system for solving OpenTSDB data hotspots, comprising:
[0017] The acquisition module is used to obtain the original metrics and all tag information of time series data;
[0018] A reconstruction module is used to reconstruct the original metric of the time series data to obtain a new metric of the time series data;
[0019] The writing module is used to write the new metric, original tag information, timestamp and value of the time series data into the OpenTSDB database.
[0020] Further improvements to the system for resolving OpenTSDB data hotspots described in the present invention are:
[0021] Furthermore, the process of reconstructing the original metric of the time series data to obtain a new metric of the time series data is as follows:
[0022] The Metric Reconstructor is used to combine all tag values into a string, which is then encrypted with MD5 to obtain an encrypted string. The first 5 digits of the encrypted string are used as a prefix and appended to the front of the Metric to form a new Metric for the time series data.
[0023] Furthermore, the process of writing the new metric and the original tag information, timestamp, and value of the time series data into the OpenTSDB database is as follows:
[0024] Call the data writing service, obtain the new metric from the metric reconstructor, and write the new metric and the original tag information, timestamp, and value into the OpenTSDB database.
[0025] Furthermore, it also includes:
[0026] The reading module is used to receive a user's request for querying data, which includes metric, tag information and time range, call the metric reconstructor to generate a new metric for the request based on the metric and tag information in the request, call the OpenTSDB database API based on the new metric and the time range to obtain data, and return it to the user.
[0027] The present invention discloses a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the method for resolving OpenTSDB data hotspots are implemented.
[0028] The present invention discloses a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method for resolving OpenTSDB data hotspots are implemented.
[0029] The present invention has the following beneficial effects:
[0030] The method and related devices for solving OpenTSDB data hotspots described in the present invention abandon the bucketing strategy of OpenTSDB during specific operation. By reconstructing the original Metric of the time series data to obtain a new Metric of the time series data, only the new Metric of the time series data is stored. This can flexibly control the degree of dispersion of data storage, evenly distribute data to each machine in the cluster, completely avoid the data hotspot problem, and greatly improve the query and storage efficiency of OpenTSDB under high concurrency conditions. BRIEF DESCRIPTION OF THE DRAWINGS
[0031] The accompanying drawings, which constitute part of the present invention, are intended to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are intended to explain the present invention and do not constitute an undue limitation of the present invention. In the accompanying drawings:
[0032] Figure 1 A diagram of the OpenTSDB storage structure with hotspot issues.
[0033] Figure 2 is a flow chart of the method of the present invention;
[0034] Figure 3 This is a schematic diagram of the data storage structure in the OpenTSDB database. DETAILED DESCRIPTION
[0035] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.
[0036] In the description of the present invention, it is to be understood that the terms “include” and “comprise” indicate the presence of the described features, wholes, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or collections thereof.
[0037] It should also be understood that the terms used in the present specification are only for the purpose of describing particular embodiments and are not intended to limit the present invention. As used in the present specification and the appended claims, the singular forms "a", "an", and "the" are intended to include the plural forms unless the context clearly indicates otherwise.
[0038] It should be further understood that the term "and / or" as used in the present specification and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items. For example, A and / or B may represent: A exists alone, A and B exist simultaneously, and B exists alone. In addition, the character " / " in the present invention generally indicates that the associated objects are in an "or" relationship.
[0039] It should be understood that although the terms "first," "second," and "third" may be used to describe preset ranges in embodiments of the present invention, these preset ranges should not be limited to these terms. These terms are merely used to distinguish one preset range from another. For example, without departing from the scope of embodiments of the present invention, the first preset range may also be referred to as the second preset range, and similarly, the second preset range may also be referred to as the first preset range.
[0040] The word "if," as used herein, may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to the determination" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)," depending on the context.
[0041] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings herein can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.
[0042] The accompanying drawings illustrate various schematic diagrams of structures according to embodiments disclosed herein. These figures are not drawn to scale; for clarity, some details are exaggerated and some details may be omitted. The shapes of the various regions and layers shown in the figures, as well as their relative sizes and positional relationships, are merely exemplary and may deviate in practice due to manufacturing tolerances or technical limitations. Those skilled in the art may design regions / layers with different shapes, sizes, and relative positions as needed.
[0043] Example 1
[0044] refer to Figure 2 The method for solving OpenTSDB data hotspots of the present invention includes the following steps:
[0045] 1) Reconstruct metrics for time series data;
[0046] The RowKey of OpenTSDB data consists of Metric, Timestamp, and Tag. OpenTSDB data is sorted by RowKey and stored in order. Metric is an indicator. For example, in a photovoltaic system, the metric is current, voltage, and temperature. Timestamp is the timestamp of the data. Tag is a label that indicates the dimension. For example, for data on photovoltaic equipment, the device ID can be used as a tag. In OpenTSDB, tags are in the form of kv, where k is the key and v is the value. For example, device_id = 1001 is a tag, and a time series data item can have multiple tags.
[0047] The data writing service calls the metric reconstructor method, passing in the original metric and all tag information of the time series data as parameters. The metric reconstructor concatenates the values of all tags into a string, then encrypts the string with MD5, takes the first 5 characters of the encrypted string as a prefix and appends it to the front of the original metric to form a new metric. The new metric is then returned to the data writing service.
[0048] For example, there are two time series data:
[0049] The metric of the first time series data is A1, and the tag information is device_id = 1001 and plant_id = 2002. The values of all tags are concatenated into the string "10012002". After MD5 encryption of this string, the first five digits of the encrypted string are taken as "f34bc". These five digits are concatenated to the front of the original metric to generate the new metric "f34bcA1".
[0050] The metric of the second time series data is A1, and the tag information is device_id = 1002 and plant_id = 2002. The new metric generated is "8aceeA1".
[0051] The purpose of this is to convert the same metrics on different devices into different metrics, making the time series data stored in OpenTSDB more distributed.
[0052] 2) Write the metrics of the time series data into the OpenTSDB database;
[0053] The data writing service obtains the new metric from the metric reconstructor and writes the original timestamp, tag, and value of the time series data into the OpenTSDB database. At this time, the data storage structure in the OpenTSDB database is as follows: Figure 3 As shown in the figure, it can be seen that the time series data originally stored on the same machine is evenly distributed on different machines, thus solving the problem of data hotspots.
[0054] 3) Query time series data
[0055] The data query service receives user requests for data query, which include metrics, tags, and time ranges. Similarly, it calls the metric reconstructor method to obtain a new metric, then uses the new metric, tags, and time range information to call the OpenTSDB database API to obtain data and return it to the user.
[0056] Example 2
[0057] The system for resolving OpenTSDB data hotspots described in the present invention includes:
[0058] The acquisition module is used to obtain the original metrics and all tag information of time series data;
[0059] A reconstruction module is used to reconstruct the original metric of the time series data to obtain a new metric of the time series data;
[0060] The writing module is used to write the new metric of the time series data and the original timestamp, tag and value of the time series data into the OpenTSDB database.
[0061] In this embodiment, the process of reconstructing the original metric and all tag information of the time series data to obtain a new metric of the time series data is as follows:
[0062] The Metric Reconstructor is used to combine all tag values into a string, which is then encrypted with MD5 to obtain an encrypted string. The first 5 digits of the encrypted string are used as a prefix and appended to the front of the Metric to form a new Metric for the time series data.
[0063] In this embodiment, the process of writing the new metric of the time series data into the OpenTSDB database is as follows:
[0064] Call the data writing service, obtain the new metric from the metric reconstructor, and write the new metric and the original timestamp, tag, and value of the time series data into the OpenTSDB database.
[0065] In this embodiment, it also includes:
[0066] The reading module is used to receive a user's request for querying data, which includes metric, tag information and time range, call the metric reconstructor to generate a new metric for the request based on the metric and tag information in the request, call the OpenTSDB database API based on the new metric and the time range to obtain data, and return it to the user.
[0067] By reconstructing the original metrics and all tag information of the time series data, a new metric for the time series data is obtained. Only the new metric for the time series data is stored, which greatly improves the query and storage efficiency of OpenTSDB in high-concurrency situations. In addition, when reading data, the reverse operation is used to achieve data reading.
[0068] The division of modules in the embodiments of the present application is illustrative and is merely a logical functional division. In actual implementation, other division methods may be used. Furthermore, the functional modules in the various embodiments of the present application may be integrated into a single processor, or may exist physically separately, or two or more modules may be integrated into a single module. The aforementioned integrated modules may be implemented in the form of hardware or software functional modules.
[0069] Example 3
[0070] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the method for resolving OpenTSDB data hotspots are implemented, for example, including: obtaining the original metric and all tag information of the time series data; using a metric reconstructor to combine all tag values into a string, then encrypting the string with MD5 to obtain an encrypted string, taking the first 5 digits of the encrypted string as a prefix and appending it to the front of the metric to form a new metric for the time series data; writing the new metric of the time series data and the original timestamp, tag, and value of the time series data into the OpenTSDB database. Receive a user's request for data query, the request containing metric, tag information, and a time range, call the metric reconstructor to generate the requested new metric based on the metric and tag information in the request, call the OpenTSDB database API based on the requested new metric and the time range to obtain data, and return it to the user. The memory may include internal memory, such as high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device. The processor, network interface, and memory are interconnected via an internal bus. This internal bus may be an Industry Standard Architecture bus, a Peripheral Component Interconnect Standard bus, an Extended Industry Standard Architecture bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. The memory is used to store programs. Specifically, the program may include program code, and the program code includes computer operating instructions. The memory may include internal memory and non-volatile memory, and provides instructions and data to the processor.
[0071] Example 4
[0072] A computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of a method for resolving OpenTSDB data hotspots. For example, the method includes: obtaining the original metric and all tag information of time series data; reconstructing the original metric and all tag information of the time series data to obtain a new metric for the time series data; and writing the new metric and the original timestamp, tag, and value of the time series data into an OpenTSDB database. A user request for data is received, the request including metric, tag information, and a time range. A metric reconstructor is invoked to generate the requested new metric based on the metric and tag information in the request. The data is retrieved using an OpenTSDB database API based on the requested new metric and the time range, and then returned to the user. Specifically, the computer-readable storage medium includes, but is not limited to, volatile memory and / or non-volatile memory. The volatile memory may include random access memory (RAM) and / or cache memory. The non-volatile memory may include read-only memory (ROM), a hard disk, flash memory, an optical disk, a magnetic disk, etc.
[0073] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.
[0074] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0075] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.
[0076] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.
[0077] Those skilled in the art will readily identify other embodiments of the present invention after considering the specification and disclosure of the invention. This application is intended to cover any variations, uses, or adaptations of the present invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered as exemplary only, with the true scope and spirit of the invention being indicated by the following claims.
[0078] It should be understood that the present invention is not limited to the exact construction described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.
[0079] The above description is only a preferred embodiment of the present invention and does not limit the present invention in any way. Any simple modification, change and equivalent structural change made to the above embodiment based on the technical essence of the present invention shall still fall within the scope of protection of the technical solution of the present invention.
Claims
1. A method for solving OpenTSDB data hotspots, characterized by: include: Get the original metric and all tag information of time series data; Reconstruct the original metric of the time series data to obtain a new metric of the time series data; Write the new metric and original tag information, timestamp, and value of the time series data into the OpenTSDB database.
2. The method for solving OpenTSDB data hotspots according to claim 1, characterized in that: The process of reconstructing the original metric of the time series data to obtain a new metric of the time series data is as follows: The Metric Reconstructor is used to combine all tag values into a string, which is then encrypted with MD5 to obtain an encrypted string. The first 5 digits of the encrypted string are used as a prefix and appended to the front of the original metric to form a new metric for the time series data.
3. The method for solving OpenTSDB data hotspots according to claim 1, characterized in that: The process of writing the new metric and the original tag information, timestamp, and value of the time series data into the OpenTSDB database is as follows: Call the data writing service, obtain the new metric from the metric reconstructor, and write the new metric and the original tag information, timestamp, and value of the time series data into the OpenTSDB database.
4. The method for solving OpenTSDB data hotspots according to claim 1, characterized in that: Also includes: Receive a user's request for querying data, which includes metric, tag information and time range, call the metric reconstructor to generate a new metric for the request based on the metric and tag information in the request, call the OpenTSDB database API based on the new metric and the time range to obtain data, and return it to the user.
5. A system for solving OpenTSDB data hotspots, characterized by: include: The acquisition module is used to obtain the original metrics and all tag information of time series data; A reconstruction module is used to reconstruct the original metric of the time series data to obtain a new metric of the time series data; The writing module is used to write the new metric and the original tag information, timestamp and value of the time series data into the OpenTSDB database.
6. The system for solving OpenTSDB data hotspots according to claim 5, characterized in that: The process of reconstructing the original metric of the time series data to obtain a new metric of the time series data is as follows: The Metric Reconstructor is used to combine all tag values into a string, which is then encrypted with MD5 to obtain an encrypted string. The first 5 digits of the encrypted string are used as a prefix and appended to the front of the Metric to form a new Metric for the time series data.
7. The system for solving OpenTSDB data hotspots according to claim 5, characterized in that: The process of writing the new metric and the original tag information, timestamp, and value of the time series data into the OpenTSDB database is as follows: Call the data writing service, obtain the new metric from the metric reconstructor, and write the new metric and the original tag information, timestamp, and value of the time series data into the OpenTSDB database.
8. The system for solving OpenTSDB data hotspots according to claim 5, characterized in that: Also includes: The reading module is used to receive a user's request for querying data, which includes metric, tag information and time range, call the metric reconstructor to generate a new metric for the request based on the metric and tag information in the request, call the OpenTSDB database API based on the new metric and the time range to obtain data, and return it to the user.
9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the method for solving OpenTSDB data hotspots according to any one of claims 1 to 4 are implemented.
10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method for solving OpenTSDB data hotspots according to any one of claims 1 to 4 are implemented.