Distributed timestamp generation method and system based on table number

Abstract

The invention discloses a distributed timestamp generation method and system based on the number of tables, and the method comprises the following steps: asynchronously receiving data through a message queue, carrying out the Hash calculation of a key field in each data, and determining a target table to which each data belongs according to a Hash value; the total number of machines in the cluster is obtained, each machine competes for a unique fragment number by using a distributed lock, and the distributed timestamp of each machine is calculated according to the current timestamp and the fragment number; traversing in a time window of a target table to which the data belongs, checking whether a distributed timestamp of a current machine is occupied, and if the distributed timestamp is occupied, adjusting the distributed timestamp according to the total number of machines in a current cluster; otherwise, marking the current distributed timestamp as occupied; and sending the distributed timestamps corresponding to the data to a storage queue, and writing the data into a time sequence database according to table names.

Description

Technical Field

[0001] The present invention relates to the technical field of distributed timestamp generation, and particularly relates to a method and system for generating a distributed timestamp based on the number of tables. Background Art

[0002] In many application scenarios, the generation and storage of timestamps are crucial for the integrity and accuracy of data. Timestamps not only need to accurately record the generation time of data but also need to maintain consistency and reliability during subsequent data processing and storage.

[0003] Currently, the generation of timestamps mainly relies on single-point timestamp systems. When generating timestamps, these systems usually adopt a data splicing method, generating timestamps by combining multiple data segments.

[0004] However, single-point timestamp systems have obvious reliability problems. Once a machine fails, the entire timestamp generation service will be affected, resulting in the interruption of recording the data generation time. The splicing method adopted by existing systems when generating timestamps may cause a large deviation between the generated timestamp and the real time of the original data, affecting the accuracy and credibility of the data. In addition, in high-concurrency scenarios, existing systems cannot completely avoid the situation of generating duplicate timestamps, which further increases the complexity and error risk of data processing. Summary of the Invention

[0005] The present invention proposes a method and system for generating a distributed timestamp based on the number of tables, which solves the problem of low reliability of existing timestamp generation methods.

[0006] To solve the above technical problems, the present invention provides a method for generating a distributed timestamp based on the number of tables, including the following steps: Step S1: Asynchronously receive data through a message queue, perform a hash calculation on the key fields in each data, and determine the target table to which each data belongs according to the hash value; Step S2: Obtain the total number of machines in the cluster. Each machine competes for a unique shard number using a distributed lock, and calculates the distributed timestamp of each machine according to the current timestamp and the shard number; Step S3: Traverse within the time window of the target table to which the data belongs, check whether the distributed timestamp of the current machine is occupied. If it is occupied, execute Step S4; otherwise, mark the current distributed timestamp as occupied and execute Step S5; Step S4: Adjust the distributed timestamp according to the total number of machines in the current cluster, and return to Step S3; Step S5: Send the distributed timestamp corresponding to each data to the storage queue and write it into the time series database according to the table name.

[0007] Preferably, the expression for calculating the distributed timestamp of each machine according to the shard number in step S2 is: ; In the formula, is the distributed timestamp; is the shard number of the machine; is the current timestamp; represents the modulo operation; is the total number of machines in the current cluster.

[0008] Preferably, in step S3, a dynamic array is obtained by expanding forward and backward centered on the distributed timestamp, and the dynamic array is used as the timestamp window , where is the window size.

[0009] Preferably, in S3, the size of the time window of the target table is adjusted according to the set accuracy of the distributed timestamp and the number of transactions per second TPS . The greater the accuracy of the distributed timestamp, the smaller the time window; the smaller the number of transactions per second TPS , the smaller the time window.

[0010] Preferably, in step S4, adjusting the distributed timestamp according to the total number of machines in the current cluster includes the following steps: advancing the distributed timestamp forward by steps. If the distributed timestamp exceeds the left boundary of the time window when advancing forward, then advance the distributed timestamp backward by steps, where is the total number of machines in the current cluster.

[0011] Preferably, in step S2, each machine competing for a unique shard number using a distributed lock includes the following steps: Step S21: Query the total number of machines in the current cluster , and determine that the value range of the shard number is ; Step S22: Each machine sends a lock request. Only one machine is allowed to obtain the lock at the same time. The machine that obtains the lock executes step S23, and other machines that do not obtain the lock wait for the release of the distributed lock; Step S23: Traverse the numbers from 0 to -1 N , and splice the fixed prefix with the current number N to generate the identifier of the machine; Step S24: Query whether the current identifier has been occupied in the coordination center. If it has not been occupied, mark the shard number of the current machine as N , otherwise letN = N +1, repeat steps S22 to S23; Step S25: Release the distributed lock, and return to step S22 until shard numbers are assigned to all machines.

[0012] The present invention also provides a distributed timestamp generation system based on the number of tables, which is implemented based on the above-mentioned distributed timestamp generation method based on the number of tables, and includes: a data receiving module, a distributed coordination module, a timestamp generation module, and a data storage module; The data receiving module: receives data sent by an external system through a message queue, extracts key fields in the data, determines the target table to which the data belongs after hashing the key fields, and binds the data to the target table and then sends it to the timestamp generation module; The distributed coordination module: maintains the total number of cluster machines and the list of live nodes through a coordination center, registers nodes when new machines join, automatically removes faulty nodes, and uses a distributed lock to assign a unique shard number to each machine; The timestamp generation module: calculates the distributed timestamp of the machine based on the current timestamp of the system and the shard number, checks the occupancy of the distributed timestamp within the independent time window of each target table, and if there is a conflict, adjusts the distributed timestamp in steps S ; The data storage module: receives the data processed by the timestamp generation module, and writes the data into the corresponding time series database according to the target table corresponding to the data.

[0013] Preferably, the system further includes a fault tolerance and monitoring module, and the fault tolerance and monitoring module: monitors the health status of machine nodes, automatically isolates faulty nodes, and reallocates the shard numbers and data of faulty nodes to healthy nodes.

[0014] The present invention also provides an electronic device, including: a memory, a processor, and a computer program, where the computer program is stored in the memory and is configured to be executed by the processor to implement the above-mentioned distributed timestamp generation method based on the number of tables.

[0015] The present invention further provides a computer-readable storage medium, in which a computer program is stored, and the computer program is executed by a processor to implement the above-mentioned distributed timestamp generation method based on the number of tables.

[0016] The advantages of the present invention at least include: 1. By performing hash calculations to distribute data to different target tables, distributed storage of data is achieved, which can effectively avoid the problem of excessive storage pressure on a single table and ensure uniform distribution of data; 2. Compete for the shard number through a distributed lock, and generate a distributed timestamp by combining the current timestamp and the shard number, ensuring the uniqueness of the timestamp and avoiding duplicate timestamps. 3. Through the mechanism of distributed lock and shard number, the timestamp generation strategy can be dynamically adjusted to avoid timestamp duplication or conflict caused by high concurrency. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is a schematic flowchart of the method of the embodiment of the present invention; Figure 2 It is a schematic diagram of the implementation process architecture of the embodiment of the present invention; Figure 3 It is a schematic flowchart of the process for a machine to obtain a shard number in the embodiment of the present invention; Figure 4 It is a schematic flowchart of the process for generating a table dimension timestamp within a single machine in the embodiment of the present invention; Figure 5 It is a schematic diagram of the relationship between the timestamp window and the table in the embodiment of the present invention; Figure 6 It is a running logic diagram of the time window in the embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0018] The following describes clearly and completely the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts fall within the protection scope of the present invention.

[0019] As Figure 1 and Figure 2 shown, the embodiment of the present invention provides a distributed timestamp generation method based on the number of tables, including the following steps: Step S1: Asynchronously receive data through a message queue, perform a hash calculation on the key fields in each data, and determine the target table to which each data belongs according to the hash value.

[0020] Specifically, before the system officially runs, the business party evaluates the data for which timestamps need to be generated to determine the number of tables in advance. The determination of the number of tables is crucial because once the system starts running and generates timestamps, the number of tables cannot be dynamically increased. If it is found later that the number of tables does not meet the requirements, the current machine must be stopped, the number of tables must be increased again, and the system must be redeployed. The number of tables has no direct connection with the shard number, but the reasonable setting of the number of tables will directly affect the size of the time window and the generation efficiency of timestamps.

[0021] In the embodiment of the present invention, the method of subscribing to the message queue kafka is adopted to asynchronously receive the data that needs to generate timestamps. After receiving the data, the system will perform hash calculation according to the key fields in the data to determine which table the data should be stored in.

[0022] Step S2: Obtain the total number of machines in the cluster. Each machine competes for a unique shard number using a distributed lock, and calculates the distributed timestamp of each machine according to the current timestamp and the shard number.

[0023] Specifically, when the timestamp generation cluster starts, each machine first obtains the total number of timestamp generation modules in the cluster through the unified coordination center , after obtaining the total number, each machine competes for a unique shard number using a distributed lock. As Figure 3 shown, it includes the following steps: Step S21: Each machine instance competes for the same distributed lock at the same time. Only one machine can obtain the lock at the same time, and other machines that do not obtain the lock enter the waiting state, waiting for the release of the distributed lock. In the embodiment of the present invention, the waiting time is set to 30 seconds.

[0024] Step S22: The machine instance that obtains the distributed lock loops from 0 to -1. Assume that the current loop reaches N , then combine the fixed prefix with the number N to form a new identifier key.

[0025] Step S23: Check whether the identifier key is occupied in the unified coordination center. If it is not occupied, write the identifier into the unified coordination center, set the shard number corresponding to the current machine instance to N , and then release the distributed lock; otherwise, execute step S24.

[0026] Step S24: Let N = N +1, regenerate a new identifier, and repeat steps S21 to S23 until all machines successfully write the shard number.

[0027] Since the total number of machines in the current cluster is known, the key written in the unified coordination center is at the millisecond level, and the distributed lock can ensure that only one machine instance can successfully occupy the lock at the same time. Therefore, during the startup process, only one instance will go to the unified coordination center to write the key at the same time. Through the above mechanism, it can be ensured that each machine can obtain a unique shard number N , and N the range of is between 0 and

[0028] Step S3: Traverse within the time window of the target table to which the data belongs, check whether the distributed timestamp of the current machine is occupied. If it is occupied, execute Step S4; otherwise, mark the current distributed timestamp as occupied and execute Step S5.

[0029] Step S4: Adjust the distributed timestamp according to the total number of machines in the current cluster, and return to Step S3.

[0030] Step S5: Send the distributed timestamp corresponding to each data to the storage queue and write it into the time series database according to the table name.

[0031] Specifically, as Figure 4 shown, after obtaining the data for which the timestamp needs to be generated, first obtain the current system long - type timestamp , and then calculate the distributed timestamp : ; In the formula, is the shard number of the machine; represents the modulo operation; is the total number of machines in the current cluster.

[0032] This formula ensures that the timestamp generated by each machine, after taking the modulo of , is equal to the corresponding machine shard number N , thus ensuring that the timestamps generated by different machines do not repeat.

[0033] As Figure 5 shown, after obtaining the distributed timestamp , starting from as the starting node, combined with the total number of machines in the cluster, the finally generated timestamp is obtained through the following timestamp window. According to the table information to which the data is assigned, obtain the timestamp window of the table. The timestamp window is a dynamic array centered on the current time and extended several seconds forward and backward , where is the window size. Traverse in the array to query whether the generated distributed timestamp is occupied. If is occupied, subtract from the current timestamp to get a new timestamp, and check the occupancy again. If the timestamp advanced forward reaches the left boundary of the time window, add to the current timestamp and continue to check until an unoccupied timestamp is found within the window.

[0034] The sizes of the left and right boundaries of the time window have nothing to do with the shard number, are indirectly related to the number of tables, and are directly related to the timestamp accuracy required by the business requirements and the number of transactions per second of the timestamp data actually needed to be generated. TPS They are directly related. Since the data for which timestamps actually need to be generated will ultimately be stored in sharded tables, and each table can only distinguish timestamps at the millisecond level at most, the actual insertion speed of each table cannot exceed 1000 records per second. Therefore, for a certain type of data, the more tables are sharded, the smaller the insertion TPS of each table after hash distribution, and the smaller the time window. Therefore, the size of the time window is related to the number of tables and the TPS of the actual data, and needs to be determined in advance according to the actual business requirements before accessing the data. If the accuracy requirement for generating timestamps is high, and the number of transactions per second of the timestamp data currently needed to be generated TPS can be met, the left and right windows can be appropriately adjusted smaller. If the accuracy requirement is not high, they can be appropriately adjusted larger. Through this mechanism, it is ensured that the generated timestamps are unique within a single table, avoiding conflicts.

[0035] According to the obtained timestamp information, the time series data, the generated timestamp information, and the corresponding storage table information are sent to another message queue. After receiving these data, the storage module stores the data in the specified time series database according to the table name and the timestamp.

[0036] To verify that the timestamp generation method in the embodiment of the present invention generates non-repeating timestamps for the time series data on a single table, the following assumptions are made: Case 1: As Figure 6 shown, assume there are two time series data D1 and D2 that need to be stored in table TableA, and they are respectively assigned to machine A with shard number N1 and machine B with shard number N2 through the message queue. According to the timestamp generation algorithm in the embodiment of the present invention, the timestamp T1 generated by D1 must be equal to N1 after taking the modulus of the total number of servers and the timestamp T2 generated by D2 must be equal to N2 after taking the modulus of the total number of servers . At the same time, since both D1 and D2 are stored in TableA, T1 and T2 must be different.

[0037] Case 2: Assume there are two time series data D1 and D2 that need to be stored in table TableA, and they are assigned to the same machine with shard number N1 through the message queue. Since the messages in the message queue need to be processed in order, assume the timestamp obtained after processing D1 is T1. According to the timestamp generation algorithm, when D2 obtains the timestamp T2, T1 has been occupied in the time window, so the timestamp assigned to T2 must be different from T1.

[0038] The embodiment of the present invention also provides a distributed timestamp generation system based on the number of tables, which is implemented based on the above-mentioned distributed timestamp generation method based on the number of tables, and includes: a data receiving module, a distributed coordination module, a timestamp generation module, a data storage module, and a fault tolerance and monitoring module.

[0039] Data receiving module: Receive data sent by an external system through a message queue, send data to the message queue in json format. The data contains key fields for uniform hashing. Extract the key fields in the data, hash the key fields to determine the target table to which the data belongs, bind the data to the target table, and send it to the processing queue of the timestamp generation module.

[0040] Distributed coordination module: Maintain the total number of cluster machines and the list of surviving nodes through a coordination center. When a new machine joins, register the node, and automatically remove the faulty node. Use a distributed lock to assign a unique shard number to each machine.

[0041] Timestamp generation module: Deployed in a cluster mode, deploy an appropriate number of server instances in the system according to the size of the business volume. At the same time, deploy a coordination center for communication such as number confirmation between server instances. An independent cache service also needs to be deployed for the generation of distributed locks during the shard acquisition process. In the embodiment of the present invention, redis is used as the cache, and zookeeper is used as the coordination center. Among them, redis is used for the distributed lock, and zookeeper is used to obtain the total number of servers. Calculate the distributed timestamp of the machine according to the current timestamp of the system and the shard number, and check the occupancy of the distributed timestamp within the independent time window of each target table. If there is a conflict, adjust the distributed timestamp in steps Adjust the distributed timestamp.

[0042] Data storage module: Used to receive the data processed by the timestamp generation module, and write the data into the corresponding time series database according to the target table corresponding to the data. Deploy an independent multi-node time series library, which is used for data storage and the configuration generation of the time window at the table dimension. In the embodiment of the present invention, the time series database TDengine is used as the storage module for time series data.

[0043] Fault Tolerance and Monitoring Module: Monitor the health status of machine nodes, automatically isolate faulty nodes, and reassign the shard numbers and data of faulty nodes to healthy nodes. If a single-machine instance fails at a certain point in time, the system can automatically exclude the relevant nodes. When a machine instance node fails, it will no longer pull messages from Kafka, and the data originally assigned to this machine instance will be randomly assigned to other normal machines through Kafka's allocation mechanism to complete the generation of timestamps on other machines. After the operation and maintenance personnel discover a machine failure, they can restart a new instance. The new instance will obtain the shard number corresponding to the faulty machine according to the algorithm rules, and at the same time Kafka will assign the original data to this new machine, thereby dynamically ensuring the availability of the overall system.

[0044] After deploying this system in the intranet of a certain company, it receives all the network switch data within the entire company and other operation and maintenance time-series data. At the peak of the data reception module TPS With 13,000 in peak value, a 6-instance cluster is deployed in the timestamp generation module, 10,000 tables are split by table dimension, 10,000 time windows are generated for the single-machine table structure, and it can still run stably when the total amount of time-series data stored in a week is about 1TB.

[0045] An embodiment of the present invention also provides an electronic device, including: a memory, a processor, and a computer program. The computer program is stored in the memory and is configured to be executed by the processor to implement the above-mentioned distributed timestamp generation method based on the number of tables.

[0046] An embodiment of the present invention further provides a computer-readable storage medium. A computer program is stored in the computer-readable storage medium, and the computer program is executed by the processor to implement the above-mentioned distributed timestamp generation method based on the number of tables.

[0047] Most of the existing timestamp generation technologies adopt non-distributed architectures, which are difficult to meet the ever-growing demand for time-series data generation. Single-machine systems have performance bottlenecks and single-point failure risks. Once a machine fails, the entire timestamp generation service will be affected. In addition, most of the existing timestamp generation algorithms adopt the method of adding a specified bit and a random number. In theory, there is a possibility of generating duplicate timestamps when the data volume increases infinitely. Most of the existing patents related to timestamps focus on embedded systems or electronic devices, and their applications in distributed large-scale application systems are relatively rare, making it difficult to meet the storage requirements of large-scale time-series data.

[0048] A distributed timestamp generation method and system based on the number of tables proposed by the embodiments of the present invention adopt a distributed architecture. Through the combination of multiple machines and multiple tables, it can be infinitely horizontally scaled, effectively meeting the storage requirements of large-scale time-series data. By introducing a time window mechanism, as long as the business volume is accurately estimated and the parameters are reasonably adapted, reliable non-repeating timestamps for a single table can be generated to ensure the complete preservation of time-series data. In addition, the complete set of time-series data storage solutions provided by the embodiments of the present invention from timestamp generation to storage comprehensively solves the problems of single-point failure and timestamp repetition in the prior art, and is applicable to the storage requirements of time-series data of various scales, especially having significant advantages in distributed large-scale application systems.

[0049] The technical features of the above embodiments can be combined arbitrarily. For the sake of concise description, not all possible combinations of the technical features in the above embodiments are described. Only the preferred embodiments of the present invention are expressed. The description is relatively specific and detailed, but it should not be construed as a limitation on the scope of the present invention. As long as the combinations of these technical features do not conflict, they should be considered as the scope described in this specification.

[0050] It should be noted that for those of ordinary skill in the art, without departing from the concept of the present invention, several deformations and improvements can still be made, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the present invention should be subject to the appended claims.

Claims

1. A distributed timestamp generation method based on the number of tables, characterized in that, It includes the following steps: Step S1: Asynchronously receive data through a message queue, calculate the hash value for the key fields in each piece of data, and determine the target table to which each piece of data belongs based on the hash value; Step S2: Obtain the total number of machines in the cluster. Each machine competes for a unique shard number using a distributed lock, and calculates the distributed timestamp of each machine based on the current timestamp and the shard number; Step S3: Traverse within the time window of the target table to which the data belongs, check whether the distributed timestamp of the current machine is occupied. If it is occupied, execute Step S4; otherwise, mark the current distributed timestamp as occupied and execute Step S5; Step S4: Adjust the distributed timestamp according to the total number of machines in the current cluster, and return to Step S3; Step S5: Send the distributed timestamp corresponding to each piece of data to the storage queue and write it into the time series database according to the table name.

2. The distributed timestamp generation method based on the number of tables according to claim 1, wherein: The expression for calculating the distributed timestamp of each machine according to the shard number in Step S2 is: ； In the formula, is the distributed timestamp; is the shard number of the machine; is the current timestamp; represents the modulo operation; is the total number of machines in the current cluster.

3. A distributed timestamp generation method based on the number of tables according to claim 2, characterized in that: In step S3, centered on the distributed timestamp, expand forward and backward to obtain a dynamic array, and use the dynamic array as the timestamp window , where is the window size.

4. A distributed timestamp generation method based on the number of tables according to claim 3, characterized in that: In S3, according to the set precision of the distributed timestamp and the number of transactions per second TPS Adjust the size of the time window of the target table. The greater the precision of the distributed timestamp, the smaller the time window; the number of transactions per second TPS is smaller, the smaller the time window.

5. A distributed timestamp generation method based on the number of tables according to claim 1, characterized in that: Adjusting the distributed timestamp according to the total number of machines in the current cluster in step S4 includes the following steps: advancing the distributed timestamp forward by steps. If the distributed timestamp exceeds the left boundary of the time window when advancing it forward, then advance the distributed timestamp backward by steps, where is the total number of machines in the current cluster.

6. A distributed timestamp generation method based on the number of tables according to claim 1, characterized in that: The step where each machine in Step S2 competes for a unique shard number using a distributed lock includes the following steps: Step S21: Query the total number of machines in the current cluster , and determine that the value range of the shard number is ; Step S22: Each machine sends a lock request. Only one machine is allowed to obtain the lock at the same time. The machine that obtains the lock executes Step S23, and other machines that do not obtain the lock wait for the release of the distributed lock; Step S23: Traverse the numbers from 0 to -1, N concatenate the fixed prefix with the current number N to generate an identifier for the machine; Step S24: Query in the coordination center whether the current identifier has been occupied. If it has not been occupied, mark the shard number of the current machine as N , otherwise let N = N + 1, and repeat steps S22 to S23; Step S25: Release the distributed lock and return to Step S22 until shard numbers are assigned to all machines.

7. A distributed timestamp generation system based on the number of tables, implemented based on a distributed timestamp generation method based on the number of tables according to any one of claims 1 to 6, characterized in that It includes: A data reception module, a distributed coordination module, a timestamp generation module, and a data storage module; The data reception module: Receives data sent by an external system through a message queue, extracts the key fields in the data, determines the target table to which the data belongs after hashing the key fields, and sends the data bound to the target table to the timestamp generation module; The distributed coordination module: maintains the total number of cluster machines through the coordination center and the list of surviving nodes, registers nodes when new machines join, automatically removes faulty nodes, and uses a distributed lock to assign a unique shard number to each machine; The timestamp generation module: calculates the distributed timestamp of the machine based on the current timestamp of the system and the shard number, checks the occupancy of the distributed timestamp within the independent time window of each target table, and adjusts the distributed timestamp by a step if there is a conflict. Adjust the distributed timestamp; The data storage module: Receives the data processed by the timestamp generation module and writes the data into the corresponding time series database according to the target table corresponding to the data.

8. A distributed timestamp generation system based on the number of tables according to claim 7, characterized in that: The system further includes a fault tolerance and monitoring module. The fault tolerance and monitoring module: Monitors the health status of machine nodes, automatically isolates faulty nodes, and redistributes the shard numbers and data of faulty nodes to healthy nodes.

9. An electronic device, comprising: A memory, a processor, and a computer program, characterized in that: the computer program is stored in the memory and is configured to be executed by the processor to implement a method for generating a distributed timestamp based on the number of tables according to any one of claims 1 to 6.

10. A computer-readable storage medium, characterized in that: A computer program is stored in the computer-readable storage medium, and the computer program is executed by a processor to implement a method for generating a distributed timestamp based on the number of tables according to any one of claims 1 to 6.

Images