Method, system, device and medium for converting structured data and time series data
By converting the timing data of Prometheus into columnar data of ClickHouse and storing it in ClickHouse, the storage bottleneck and query delay problems of Prometheus when processing large-scale timing data is solved, efficient data storage and querying is achieved, and the system is improved Scalability and reliability.
Patent Information
- Application Number
- CN202411045655.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-01
- Publication Date
- 2024-11-05
- Estimated Expiration
- 2044-08-01
AI Technical Summary
Prometheus has storage bottlenecks, query latency and scalability when processing large-scale timing data, making it difficult to meet the data processing needs of high concurrency and high throughput.
By converting the time series data collected by Prometheus into columnar data of ClickHouse and storing it in ClickHouse, it utilizes its high-performance data storage capabilities, combining user authentication, permission management and data compression technologies to achieve seamless conversion and storage of data.
Improves data storage and query performance, reduces resource consumption, provides good scalability and data security, reduces latency, and improves the overall performance and reliability of the system.
Smart Images

Figure CN118568134B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of data processing technology, and in particular to a method, system, device and medium for converting structured data and time series data. Background Art
[0002] With the rapid development of cloud services, enterprises and organizations generate a large amount of time series data in their daily operations. These data cover multiple aspects such as system monitoring, performance analysis, and user behavior tracking, and are of great significance for improving service quality and business decision-making. As a popular open source time series database, Prometheus is widely used in monitoring and alarm systems. However, with the surge in data volume, Prometheus faces a series of problems in data storage and query:
[0003] ① Storage bottleneck: Prometheus's built-in storage is prone to performance bottlenecks when processing large-scale time series data, making it difficult to meet high-concurrency and high-throughput data processing requirements.
[0004] ②Query delay: As the amount of data increases, query performance decreases significantly, affecting the efficiency of real-time data analysis and monitoring.
[0005] ③ Insufficient scalability: Prometheus has limited scalability in large-scale cluster environments and is difficult to adapt to growing data needs.
[0006] Therefore, how to store the time series data collected by Prometheus in a higher-performance database to improve overall performance and reliability is a technical problem that needs to be solved urgently. Summary of the invention
[0007] The technical task of the present invention is to provide a method, system, device and medium for converting structured data and time series data to solve the problem of how to store the time series data collected by Prometheus into a database with higher performance, thereby improving the overall performance and reliability.
[0008] The technical task of the present invention is achieved in the following manner: a method for converting structured data and time series data, the method is specifically as follows:
[0009] User authentication: Perform user authentication to ensure user access rights and generate Prometheus requests;
[0010] Request reception: Receive Prometheus requests, which include data loading and reading requests;
[0011] Data transformation: Based on Prometheus requests, time series data is converted into columnar data within ClickHouse and ClickHouse data is converted into Prometheus result sets.
[0012] As a preference, user authentication is specifically as follows:
[0013] User identity authentication: Confirm the user's login information and use the OAuth2.0 protocol to authenticate the user to ensure the security of the authentication process;
[0014] Permission management: manage various permissions for data loading and reading requests to prevent unauthorized reading and writing of data;
[0015] Session management: Use JWT (JSON Web Token) for session management, support session renewal and forced logout functions to ensure security.
[0016] Preferably, the request is received as follows:
[0017] Data read request processing: Receive data read requests forwarded by Prometheus, and confirm whether the data read permission is within the user's permission range through user authentication, send data query to Clickhouse to obtain data and call data conversion to convert Clickhouse structure data into Prometheus time series data to provide services to users;
[0018] Data loading request processing: Receive Prometheus data writing requests, confirm whether the data writing permissions are within the user's permissions through user authentication, and call data conversion to convert Prometheus time series data into Clickhouse structured data.
[0019] Preferably, the data conversion is as follows:
[0020] promQL parsing: Define a time series data of Prometheus as a data table in Clickhouse structured data, and define the time series data as the corresponding label and value fields. According to different time series data read and write requests, promQL is converted into conditional splicing of the label and value of the Clickhouse data table to form the corresponding database query request;
[0021] Time series data conversion: Convert the labels of time series data into labels and values in Clickhouse structured data, perform corresponding data compression according to the data type, and create different Clickhouse data tables according to the name of the time series to perform reading and writing;
[0022] Structured data conversion: query to obtain structured data in the Clickhouse data table, combine it into Prometheus standard time series data according to the sorting dimension and return it.
[0023] A system for converting structured data and time series data, the system is used to implement the above-mentioned method for converting structured data and time series data, and the system includes:
[0024] User authentication module, used to implement user authentication and ensure user access rights;
[0025] The request receiving module is used to receive Prometheus requests, which include data loading and reading requests.
[0026] Data conversion module, used to convert time series data into columnar data within ClickHouse and convert ClickHouse data into Prometheus result sets.
[0027] Preferably, the user authentication module comprises:
[0028] User login submodule, used for user login via username and password;
[0029] The permission verification submodule is used to verify user access requests based on the user's identity permissions to ensure that only authorized users can access specific data and functions;
[0030] The session management submodule is used to manage user session information and supports session timeout and re-authentication mechanisms to ensure security.
[0031] Preferably, the request receiving module comprises:
[0032] The data loading request processing submodule is used to receive data writing requests sent by Prometheus and forward the requests to the data conversion module for processing;
[0033] The data read request processing submodule is used to receive data query requests sent by users and forward the requests to the data conversion module for processing.
[0034] Preferably, the data conversion module includes:
[0035] PromQL parsing submodule, used to convert time series data queries and operations in PromQL into structured data SQL operations for Clickhouse;
[0036] The time series data conversion submodule is used to convert the time series data received by Prometheus into a columnar data format compatible with ClickHouse and write it into the ClickHouse database in batches;
[0037] The structured data conversion submodule is used to convert ClickHous' structured columnar data into Prometheus's time series data for Prometheus to provide external services;
[0038] The data storage optimization submodule is used to compress data during data conversion to improve the efficiency of data storage and query.
[0039] An electronic device, characterized in that it comprises: a memory and at least one processor;
[0040] Wherein, the memory stores a computer program;
[0041] The at least one processor executes the computer program stored in the memory, so that the at least one processor performs the method for converting structured data and time series data as described above.
[0042] A computer-readable storage medium stores a computer program, and the computer program can be executed by a processor to implement the above-mentioned method for converting structured data and time series data.
[0043] The method, system, device and medium for converting structured data and time series data of the present invention have the following advantages:
[0044] (1) The present invention can improve data storage performance: by transferring Prometheus data to ClickHouse and utilizing ClickHouse's high-performance data storage capabilities, the efficiency of data storage and query is improved; at the same time, a seamless data conversion and storage solution from Prometheus to ClickHouse is provided;
[0045] (ii) The present invention reduces resource consumption: by compressing data, system resource expenditure is reduced;
[0046] (III) The present invention has good scalability: it solves the problem that Prometheus does not have a cluster structure, prevents single points in the production environment, and designs a write proxy service to support high concurrency and high throughput processing of large-scale data to meet the growing data demand;
[0047] (iv) The present invention ensures data security: through encrypted transmission and authentication mechanisms, it ensures the security of data during transmission and storage, and prevents data leakage and tampering; and 2) implements user authentication and access control to ensure the security and compliance of data access;
[0048] (V) The present invention improves system compatibility: the proxy service design is highly compatible and can be seamlessly integrated into the existing Prometheus and ClickHouse systems, facilitating system upgrades and maintenance;
[0049] (VI) The present invention reduces latency: reduces data query latency and improves response speed;
[0050] (VII) The present invention realizes user isolation: the proxy realizes user isolation, so that data that was originally not managed by users has permission control, realizing efficient, safe and reliable data management and access control;
[0051] (VIII) The present invention stores the time series data collected by Prometheus in a database with higher performance, thereby improving the overall performance and reliability of the system;
[0052] (IX) The present invention ensures efficient access and query of data in ClickHouse through data conversion and optimization, and can effectively improve the scalability and performance of Prometheus data storage, reduce data query latency, and improve the reliability and availability of the overall system; the present invention has good compatibility and can be easily integrated into existing Prometheus and ClickHouse systems, and is suitable for various complex business scenarios. BRIEF DESCRIPTION OF THE DRAWINGS
[0053] The present invention is further described below in conjunction with the accompanying drawings.
[0054] Attached Figure 1 It is a schematic diagram of the structure of the conversion system between structured data and time series data;
[0055] Attached Figure 2 This is a schematic diagram of a user authentication module;
[0056] Attached Figure 3 A schematic diagram of a user authentication module and a request receiving module;
[0057] Attached Figure 4 is a schematic diagram of a request receiving module;
[0058] Attached Figure 5 Schematic diagram of the working process of the conversion system for structured data and time series data. DETAILED DESCRIPTION
[0059] The method, system, device and medium for converting structured data and time series data of the present invention are described in detail below with reference to the accompanying drawings and specific embodiments.
[0060] Embodiment 1: This embodiment provides a method for converting structured data and time series data, and the method is specifically as follows:
[0061] User authentication: Perform user authentication to ensure user access rights and generate Prometheus requests;
[0062] Request reception: Receive Prometheus requests, which include data loading and reading requests;
[0063] Data transformation: Based on Prometheus requests, time series data is converted into columnar data within ClickHouse and ClickHouse data is converted into Prometheus result sets.
[0064] As a preference, user authentication is specifically as follows:
[0065] User identity authentication: Confirm the user's login information and use the OAuth2.0 protocol to authenticate the user to ensure the security of the authentication process;
[0066] Permission management: manage various permissions for data loading and reading requests to prevent unauthorized reading and writing of data;
[0067] Session management: Use JWT (JSON Web Token) for session management, support session renewal and forced logout functions to ensure security.
[0068] Preferably, the request is received as follows:
[0069] Data read request processing: Receive data read requests forwarded by Prometheus, and confirm whether the data read permission is within the user's permission range through user authentication, send data query to Clickhouse to obtain data and call data conversion to convert Clickhouse structure data into Prometheus time series data to provide services to users;
[0070] Data loading request processing: Receive Prometheus data writing requests, confirm whether the data writing permissions are within the user's permissions through user authentication, and call data conversion to convert Prometheus time series data into Clickhouse structured data.
[0071] Preferably, the data conversion is as follows:
[0072] promQL parsing: Define a time series data of Prometheus as a data table in Clickhouse structured data, and define the time series data as the corresponding label and value fields. According to different time series data read and write requests, promQL is converted into conditional splicing of the label and value of the Clickhouse data table to form the corresponding database query request;
[0073] Time series data conversion: Convert the labels of time series data into labels and values in Clickhouse structured data, perform corresponding data compression according to the data type, and create different Clickhouse data tables according to the name of the time series to perform reading and writing;
[0074] Structured data conversion: query to obtain structured data in the Clickhouse data table, combine it into Prometheus standard time series data according to the sorting dimension and return it.
[0075] Embodiment 2: As attached Figure 1 As shown, this embodiment provides a system for converting structured data and time series data, which is used to implement the method for converting structured data and time series data in Example 1. The system includes:
[0076] User authentication module, used to implement user authentication and ensure user access rights;
[0077] The request receiving module is used to receive Prometheus requests, which include data loading and reading requests.
[0078] Data conversion module, used to convert time series data into columnar data within ClickHouse and convert ClickHouse data into Prometheus result sets.
[0079] As attached Figure 2 and 3 As shown, the user authentication module in this embodiment includes:
[0080] User login submodule, used for user login via username and password;
[0081] The permission verification submodule is used to verify user access requests based on the user's identity permissions to ensure that only authorized users can access specific data and functions;
[0082] The session management submodule is used to manage user session information and supports session timeout and re-authentication mechanisms to ensure security.
[0083] As attached Figure 4 and5 As shown, the request receiving module in this embodiment includes:
[0084] The data loading request processing submodule is used to receive data writing requests sent by Prometheus and forward the requests to the data conversion module for processing;
[0085] The data read request processing submodule is used to receive data query requests sent by users and forward the requests to the data conversion module for processing.
[0086] The data conversion module in this embodiment includes:
[0087] PromQL parsing submodule, used to convert time series data queries and operations in PromQL into structured data SQL operations for Clickhouse;
[0088] The time series data conversion submodule is used to convert the time series data received by Prometheus into a columnar data format compatible with ClickHouse and write it into the ClickHouse database in batches;
[0089] The structured data conversion submodule is used to convert ClickHous' structured columnar data into Prometheus's time series data for Prometheus to provide external services;
[0090] The data storage optimization submodule is used to compress data during data conversion to improve the efficiency of data storage and query.
[0091] Embodiment 3: This embodiment also provides an electronic device, including: a memory and a processor;
[0092] Wherein, the memory stores computer-executable instructions;
[0093] The processor executes the computer-executable instructions stored in the memory, so that the processor executes the method for converting structured data and time series data in any embodiment of the present invention.
[0094] The processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The processor may be a microprocessor or any conventional processor, etc.
[0095] The memory can be used to store computer programs and / or modules. The processor realizes various functions of the electronic device by running or executing the computer programs and / or modules stored in the memory, and calling the data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, at least one application required for a function, etc.; the data storage area can store data created according to the use of the terminal, etc. In addition, the memory can also include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (SMC), a secure digital (SD) card, a flash memory card, at least one disk storage period, a flash memory device, or other volatile solid-state storage devices.
[0096] Embodiment 4: This embodiment also provides a computer-readable storage medium, in which a plurality of instructions are stored, and the instructions are loaded by a processor, so that the processor executes the method for converting structured data and time series data in any embodiment of the present invention. Specifically, a system or device equipped with a storage medium can be provided, on which a software program code for implementing the functions of any of the above embodiments is stored, and a computer (or CPU or MPU) of the system or device reads and executes the program code stored in the storage medium.
[0097] In this case, the program code itself read from the storage medium can realize the function of any one of the above-mentioned embodiments, and thus the program code and the storage medium storing the program code constitute a part of the present invention.
[0098] Examples of storage media for providing program code include floppy disks, hard disks, magneto-optical disks, optical disks (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RYM, DVD-RW, DVD+RW), magnetic tapes, non-volatile memory cards, and ROMs. Alternatively, program code may be downloaded from a server computer via a communication network.
[0099] In addition, it should be clear that the functions of any of the above embodiments can be implemented not only by executing the program code read by the computer, but also by enabling an operating system operating on the computer to complete part or all of the actual operations based on instructions from the program code.
[0100] In addition, it can be understood that the program code read from the storage medium is written to a memory provided in an expansion board inserted into the computer or written to a memory provided in an expansion unit connected to the computer, and then based on the instructions of the program code, a CPU installed on the expansion board or the expansion unit is enabled to perform part or all of the actual operations, thereby realizing the functions of any of the above-mentioned embodiments.
[0101] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.
Claims
1. A method for converting structured data and time series data, characterized in that: The method is as follows: User authentication: Perform user authentication to ensure user access rights and generate Prometheus requests; the details are as follows: User identity authentication: Confirm the user's login information and use the OAuth2.0 protocol to authenticate the user to ensure the security of the authentication process; Permission management: manage various permissions for data loading and reading requests to prevent unauthorized reading and writing of data; Session management: Use JWT for session management, support session renewal and forced logout functions; Request reception: Receive Prometheus requests, which include data loading and reading requests. The details are as follows: Data read request processing: Receive data read requests forwarded by Prometheus, and confirm whether the data read permission is within the user's permission range through user authentication, send data query to Clickhouse to obtain data and call data conversion to convert Clickhouse structure data into Prometheus time series data to provide services to users; Data loading request processing: Receive Prometheus data writing requests, confirm whether the data writing permission is within the user's permission through user authentication, and call data conversion to convert Prometheus time series data into Clickhouse structured data; Data conversion: According to the request of Prometheus, the time series data is converted into columnar data in ClickHouse and the ClickHouse data is converted into Prometheus result set; the details are as follows: promQL parsing: Define a time series data of Prometheus as a data table in Clickhouse structured data, and define the time series data as the corresponding label and value fields. According to different time series data read and write requests, promQL is converted into conditional splicing of the label and value of the Clickhouse data table to form the corresponding database query request; Time series data conversion: Convert the labels of time series data into labels and values in Clickhouse structured data, perform corresponding data compression according to the data type, and create different Clickhouse data tables according to the name of the time series to perform reading and writing; Structured data conversion: query to obtain structured data in the Clickhouse data table, combine it into Prometheus standard time series data according to the sorting dimension and return it.
2. A system for converting structured data and time series data, characterized in that: The system is used to implement the method for converting structured data and time series data as claimed in claim 1, and the system includes: User authentication module, used to implement user authentication and ensure user access rights; The request receiving module is used to receive Prometheus requests, which include data loading and reading requests. Data conversion module, which is used to convert time series data into columnar data in ClickHouse and convert ClickHouse data into Prometheus result sets; Among them, the user authentication module includes: User login submodule, used for user login via username and password; The permission verification submodule is used to verify user access requests based on the user's identity permissions to ensure that only authorized users can access specific data and functions; The session management submodule is used to manage user session information and supports session timeout and re-authentication mechanisms to ensure security; The request receiving module includes: The data loading request processing submodule is used to receive data writing requests sent by Prometheus and forward the requests to the data conversion module for processing; The data read request processing submodule is used to receive the data query request sent by the user and forward the request to the data conversion module for processing; The data conversion module includes: PromQL parsing submodule, used to convert time series data queries and operations in PromQL into structured data SQL operations for Clickhouse; The time series data conversion submodule is used to convert the time series data received by Prometheus into a columnar data format compatible with ClickHouse and write it into the ClickHouse database in batches; The structured data conversion submodule is used to convert ClickHous' structured columnar data into Prometheus's time series data for Prometheus to provide external services; The data storage optimization submodule is used to compress data during data conversion to improve the efficiency of data storage and query.
3. An electronic device, characterized in that: include: memory and at least one processor; Wherein, the memory stores a computer program; The at least one processor executes the computer program stored in the memory, so that the at least one processor executes the method for converting structured data and time series data as described in claim 1.
4. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program can be executed by a processor to implement the method for converting structured data and time series data as claimed in claim 1.
