Method, system and device for improving Opentsdb writing stability and medium

Abstract

The invention discloses a method, a system, equipment and a medium for improving Opentsdb writing stability. The method comprises the following steps: judging whether a to-be-written measuring point is a written measuring point or not; when the to-be-written measuring point is a non-written measuring point, executing UID (User Identifier) creating operation, and writing the to-be-written measuring point into Opentsdb; when the to-be-written measuring point is a written measuring point, whether the to-be-written measuring point exists or not is inquired in a cache, and when the to-be-written measuring point exists in the cache, UID creating operation continues to be executed; and when the to-be-written measuring point does not exist in the cache, writing the to-be-written measuring point into the Opentsdb. According to the method, the system, the equipment and the medium, the Opentsdb writing stability can be improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of big data and relates to a method, system, device and medium for improving the writing stability of Opentsdb. Background Art

[0002] With the development of the Internet and sensor technology, the Internet of Things has become an important bridge connecting the physical world and the digital world. The data sources of the Internet of Things are diverse, including but not limited to sensors, smart devices, automobiles, industrial equipment, home appliances, etc. These devices and sensors can collect various types of data, such as temperature, humidity, light, pressure, location, movement, sound, etc.

[0003] IoT data is usually generated and collected in real time, and can reflect changes in equipment and environment in a timely manner. This makes IoT data of great application value in monitoring, early warning and control. With the popularization of IoT devices and the expansion of their application scope, the amount of IoT data is growing exponentially. These huge amounts of data require the help of big data technologies and tools for effective storage, management and analysis.

[0004] IoT data has a wide range of application scenarios, including smart cities, smart transportation, smart healthcare, smart manufacturing, smart agriculture, etc. Through the analysis and mining of IoT data, it can help people better understand and utilize the physical world, improve production efficiency, improve the quality of life, and achieve goals such as resource optimization and environmental protection.

[0005] Therefore, the storage of IoT data is the most critical part of IoT. IoT data is generally time series data, that is, each piece of data reported is timestamped to indicate the time when the data occurred. OpenTSDB (Open Time Series Database) is an open source, distributed time series database for storing and retrieving large-scale time series data. It is built on HBase and is specifically designed to process time series data, such as server metrics, sensor data, application metrics, etc.

[0006] OpenTSDB provides an efficient storage mechanism that can store large amounts of time series data. It uses a time-based sharding storage method to divide data into multiple shards according to the time range. Each shard can be stored independently in different HBase tables to achieve horizontal expansion and high availability. OpenTSDB is a distributed system that can process large amounts of data through multiple nodes. It uses HBase as the underlying storage engine and uses the distributed computing and storage capabilities provided by the Hadoop ecosystem to achieve distributed storage and query. OpenTSDB provides a rich query function that can query and aggregate time series data in real time. Users can use SQL-like query language to perform various query operations, including range queries, aggregation queries, filter queries, etc. to meet different analysis needs. OpenTSDB is a highly scalable system that can flexibly expand and reduce nodes as needed. It can dynamically add or remove nodes according to the growth of data volume to achieve automatic scaling of storage and processing capabilities. OpenTSDB is an open source project with an active development and user community. Users can obtain technical support and exchange experiences through official documents, mailing lists, forums and other channels, and can also participate in the development and improvement of the project.

[0007] The storage of measurement point data in Opentsdb involves uid_table, meta_table, tree_table and data_table. Among them, uid_table is used to store a unique UID for each measurement point. When the measurement point data is written to Opentsdb, if the measurement point is a new measurement point (first write), a corresponding unique UID will be created (the option of automatically creating a new measurement point is turned on). When the data is stored in data_table, the metric of the measurement point is converted to the corresponding UID, which can reduce the storage space required for the data.

[0008] The default length of UID is three bytes. If the number of measurement points is large, the number of bytes occupied by UID can be adjusted according to the actual situation to modify the available number of UIDs. In the actual application scenario, due to the large number of measurement points, the length of UID is set to four bytes.

[0009] When a measurement point is written to Opentsdb, if the corresponding UID of the measurement point has not been created before, Opentsdb can automatically create the corresponding UID of the measurement point (automatic creation of UID can be turned on or off). When the number of created UIDs is too large, it becomes increasingly slow to find a UID that has not been created before (due to conflicts, etc.). By observing the write log, the time to create a new UID is as long as about 1 minute, and some even take 2 minutes. When creating a new UID, the writing of the measurement point data will be blocked, resulting in poor writing stability, which is manifested as the write speed fluctuating. Summary of the invention

[0010] The purpose of the present invention is to overcome the shortcomings of the above-mentioned prior art and provide a method, system, device and medium for improving the writing stability of Opentsdb. The method, system, device and medium can improve the writing stability of Opentsdb.

[0011] In order to achieve the above object, the present invention adopts the following technical scheme:

[0012] In one aspect, the present invention provides a method for improving the write stability of Opentsdb, comprising:

[0013] Determine whether the measurement point to be written is a measurement point that has been written;

[0014] When the measurement point to be written is a measurement point that has not been written, the UID creation operation is performed, and the measurement point to be written is written into Opentsdb;

[0015] When the measurement point to be written is a measurement point that has been written, the cache is queried to see whether the measurement point to be written exists. If the measurement point to be written exists in the cache, the UID creation operation is continued; if the measurement point to be written does not exist in the cache, the measurement point to be written is written to Opentsdb.

[0016] The method for improving the write stability of Opentsdb described in the present invention is further improved in that:

[0017] Furthermore, the process of determining whether the measurement point to be written is a measurement point that has been written is as follows:

[0018] The Bloom filter is used to determine whether the measurement point to be written is a measurement point that has already been written.

[0019] Furthermore, before determining whether the measurement point to be written is a measurement point that has been written, the process further includes:

[0020] Create a Bloom filter.

[0021] Furthermore, the cache time is 10 minutes.

[0022] In a second aspect of the present invention, the present invention provides a system for improving the writing stability of Opentsdb, comprising:

[0023] A judgment module, used to judge whether the measurement point to be written is a measurement point that has been written;

[0024] The first writing module is used for executing a UID creation operation when the measurement point to be written is a measurement point that has not been written, and writing the measurement point to be written into Opentsdb;

[0025] The second writing module is used to query whether the measurement point to be written exists in the cache when the measurement point to be written is a measurement point that has been written. If the measurement point to be written exists in the cache, the UID creation operation continues to be executed; if the measurement point to be written does not exist in the cache, the measurement point to be written is written into Opentsdb.

[0026] The method for improving the write stability of Opentsdb described in the present invention is further improved in that:

[0027] Furthermore, the process of determining whether the measurement point to be written is a measurement point that has been written is as follows:

[0028] The Bloom filter is used to determine whether the measurement point to be written is a measurement point that has already been written.

[0029] Furthermore, it also includes:

[0030] Create a module for creating Bloom filters.

[0031] Furthermore, the cache time is 10 minutes.

[0032] In a third aspect of the present invention, the present invention provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method for improving the write stability of Opentsdb when executing the computer program.

[0033] In a fourth aspect of the present invention, the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and is characterized in that when the computer program is executed by a processor, the steps of the method for improving the write stability of Opentsdb are implemented.

[0034] The present invention has the following beneficial effects:

[0035] The method, system, device and medium for improving the writing stability of Opentsdb described in the present invention, in specific operation, because the speed of creating UID is slow, if there is no such caching process, then the Bloom filter already has the measuring point when the measuring point is written next time, and the measuring point will be directly written into Opentsdb. At this time, the UID of the measuring point has not been created yet, and the UID will be created first when writing. Since the creation of UID is slow, it will affect the normal writing rate and reliability. Therefore, in the present invention, when the measuring point to be written is a new measuring point, caching is performed, that is, the corresponding UID is created first to improve the stability of Opentsdb writing and ensure the timeliness of the business. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] The accompanying drawings constituting a part of the present invention are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the accompanying drawings:

[0037] Figure 1 The present invention is a flow chart of the method. DETAILED DESCRIPTION

[0038] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0039] In the description of the present invention, it should be understood that the terms “include” and “comprises” indicate the presence of described features, wholes, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or collections thereof.

[0040] It should also be understood that the terms used in the present specification are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the present specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0041] It should be further understood that the term "and / or" used in the present specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes these combinations. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in the present invention generally indicates that the associated objects are in an "or" relationship.

[0042] It should be understood that, although the terms first, second, third, etc. may be used to describe preset ranges, etc. in the embodiments of the present invention, these preset ranges should not be limited to these terms. These terms are only used to distinguish preset ranges from each other. For example, without departing from the scope of the embodiments of the present invention, the first preset range may also be referred to as the second preset range, and similarly, the second preset range may also be referred to as the first preset range.

[0043] The word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to determining" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)", depending on the context.

[0044] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. The components of the embodiments of the present invention described and shown in the drawings here can usually be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the present invention. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0045] Various structural schematic diagrams of the embodiments disclosed in the present invention are shown in the accompanying drawings. These figures are not drawn to scale, and some details are magnified and some details may be omitted for the purpose of clear expression. The shapes of various regions and layers shown in the figures and the relative sizes and positional relationships therebetween are only exemplary, and may deviate in practice due to manufacturing tolerances or technical limitations, and those skilled in the art may additionally design regions / layers with different shapes, sizes, and relative positions according to actual needs.

[0046] Embodiment 1

[0047] refer to Figure 1 The method for improving the writing stability of Opentsdb according to the present invention comprises the following steps:

[0048] 1) Create a Bloom filter;

[0049] 2) Using the Bloom filter, determine whether the measurement point to be written is a measurement point that has already been written;

[0050] 3) When the measurement point to be written is a measurement point that has not been written, the UID creation operation is performed and written to Opentsdb; when the measurement point to be written is a measurement point that has been written, the cache is queried to see whether the measurement point to be written exists. When the measurement point to be written exists in the cache, the UID creation operation continues to be performed; when the measurement point to be written does not exist in the cache, the measurement point to be written is written to Opentsdb.

[0051] It should be noted that the reason for caching for 10 minutes is that the speed of creating UID is slow. If there is no caching process, the next time the measurement point is written, the Bloom filter already has the measurement point, and the measurement point will be written directly to Opentsdb. At this time, the UID of the measurement point has not been created yet. The UID will be created first when writing. Since the creation of UID is slow, it will affect the normal writing rate and reliability.

[0052] The present invention uses a Bloom filter to determine whether a measurement point is a new measurement point or a measurement point that has been written. If it is a new measurement point, the corresponding UID is created first; if it is a measurement point that has been written, it is directly written to Opentsdb, thereby improving the stability of Opentsdb writing and ensuring the timeliness of the business.

[0053] Embodiment 2

[0054] The present invention provides a system for improving the writing stability of Opentsdb, including:

[0055] A judgment module, used to judge whether the measurement point to be written is a measurement point that has been written;

[0056] The first writing module is used for executing a UID creation operation when the measurement point to be written is a measurement point that has not been written, and writing the measurement point to be written into Opentsdb;

[0057] The second writing module is used to query whether the measurement point to be written exists in the cache when the measurement point to be written is a measurement point that has been written. If the measurement point to be written exists in the cache, the UID creation operation continues to be executed; if the measurement point to be written does not exist in the cache, the measurement point to be written is written into Opentsdb.

[0058] As an implementation manner of the present invention, the process of determining whether the measurement point to be written is a measurement point that has been written is as follows:

[0059] The Bloom filter is used to determine whether the measurement point to be written is a measurement point that has already been written.

[0060] As an embodiment of the present invention, it also includes:

[0061] Create a module for creating Bloom filters.

[0062] As an implementation mode of the present invention, the cache time is 10 minutes. .

[0063] The division of modules in the embodiments of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation. In addition, each functional module in each embodiment of the present application may be integrated into a processor, or may exist physically separately, or two or more modules may be integrated into one module. The above-mentioned integrated modules may be implemented in the form of hardware or in the form of software functional modules.

[0064] Embodiment 3

[0065] A computer device comprises a memory, a processor and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the method for improving the write stability of Opentsdb are implemented, for example, comprising: determining whether a measurement point to be written is a measurement point that has been written; when the measurement point to be written is a measurement point that has not been written, executing a create UID operation and writing the measurement point to be written into Opentsdb; when the measurement point to be written is a measurement point that has been written, querying whether the measurement point to be written exists in the cache, and when the measurement point to be written exists in the cache, continuing to execute the create UID operation; when the measurement point to be written does not exist in the cache, writing the measurement point to be written into Opentsdb.

[0066] In addition, the computer device may be a computing device such as a desktop computer, a notebook, a PDA, and a cloud server. The computer device may include, but is not limited to, a processor and a memory. Those skilled in the art will appreciate that the above are examples of computer devices and do not constitute a limitation on computer devices. The computer device may include more components than the above, or a combination of certain components, or different components. For example, the computer device may also include input and output devices, network access devices, buses, etc. The memory may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the mobile phone (such as audio data, a phone book, etc.), etc. In addition, the memory may include a high-speed random access memory, and may also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (SmartMediaCard, SMC), a secure digital (SecureDigital, SD) card, a flash card (FlashCard), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. The memory may be used to store the computer programs and / or modules, and the processor implements various functions of the computer device by running or executing the computer programs and / or modules stored in the memory and calling the data stored in the memory.

[0067] The processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc. The processor is the control center of the computer device, and uses various interfaces and lines to connect various parts of the entire computer.

[0068] Embodiment 4

[0069] A computer-readable storage medium stores a computer program. When the computer program is executed by a processor, the steps of the method for improving the writing stability of Opentsdb are implemented, for example, including: determining whether the measurement point to be written is a measurement point that has been written; when the measurement point to be written is a measurement point that has not been written, performing a UID creation operation for the new measurement point and writing it to Opentsdb; when the measurement point to be written is a measurement point that has been written, querying whether the measurement point to be written exists in the cache, and when the measurement point to be written exists in the cache, continuing to perform the UID creation operation; when the measurement point to be written does not exist in the cache, writing the measurement point to be written to Opentsdb. Specifically, the computer-readable storage medium includes, but is not limited to, for example, volatile memory and / or non-volatile memory. The volatile memory may include random access memory (RAM) and / or cache memory (cache), etc. The non-volatile memory may include read-only memory (ROM), hard disk, flash memory, optical disk, magnetic disk, etc.

[0070] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0071] Based on this understanding, the present invention implements all or part of the process in the above-mentioned method of automatically detecting task status and starting and stopping, and can also be completed by instructing related hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, the steps of the above-mentioned method for improving the write stability of Opentsdb can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or preset intermediate form, etc.

[0072] It should be noted that the content contained in the computer-readable storage medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable storage media do not include electrical carrier signals and telecommunication signals.

[0073] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0074] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0075] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A method to improve the writing stability of Opentsdb, It is characterized in that include: Determine whether the measurement point to be written is a measurement point that has been written; When the measurement point to be written is a measurement point that has not been written, the UID creation operation is performed, and the measurement point to be written is written into Opentsdb; When the measurement point to be written is a measurement point that has been written, the cache is queried to see whether the measurement point to be written exists. If the measurement point to be written exists in the cache, the UID creation operation is continued; if the measurement point to be written does not exist in the cache, the measurement point to be written is written to Opentsdb.

2. The method for improving the write stability of Opentsdb according to claim 1, It is characterized in that The process of judging whether the measurement point to be written is a measurement point that has been written is as follows: The Bloom filter is used to determine whether the measurement point to be written is a measurement point that has already been written.

3. The method for improving the write stability of Opentsdb according to claim 1, It is characterized in that Before determining whether the measurement point to be written is a measurement point that has been written, the following step is further included: Create a Bloom filter.

4. The method for improving the write stability of Opentsdb according to claim 1, It is characterized in that The cache time is 10 minutes.

5. A system to improve Opentsdb write stability, It is characterized in that include: A judgment module, used to judge whether the measurement point to be written is a measurement point that has been written; The first writing module is used for executing a UID creation operation when the measurement point to be written is a measurement point that has not been written, and writing the measurement point to be written into Opentsdb; The second writing module is used to query whether the measurement point to be written exists in the cache when the measurement point to be written is a measurement point that has been written. If the measurement point to be written exists in the cache, the UID creation operation continues to be executed; if the measurement point to be written does not exist in the cache, the measurement point to be written is written into Opentsdb.

6. The system for improving Opentsdb write stability according to claim 1, It is characterized in that The process of judging whether the measurement point to be written is a measurement point that has been written is as follows: The Bloom filter is used to determine whether the measurement point to be written is a measurement point that has already been written.

7. The system for improving Opentsdb write stability according to claim 1, It is characterized in that Also includes: Create a module for creating Bloom filters.

8. The system for improving Opentsdb write stability according to claim 1, It is characterized in that The cache time is 10 minutes.

9. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, It is characterized in that When the processor executes the computer program, the steps of the method for improving the write stability of Opentsdb as described in any one of claims 1-4 are implemented.

10. A computer-readable storage medium storing a computer program. It is characterized in that When the computer program is executed by a processor, the steps of the method for improving the write stability of Opentsdb as described in any one of claims 1 to 4 are implemented.

Images