A distributed timestamp generation method and system based on table quantity

Abstract

The present invention discloses a method and system for generating distributed timestamps based on the number of tables. The method comprises the following steps: asynchronously receiving data through a message queue, performing hash calculation on key fields in each data, and determining the target table to which each data belongs according to the hash value; obtaining the total number of machines in a cluster, each machine competing for a unique shard number using a distributed lock, and calculating the distributed timestamp of each machine according to the current timestamp and the shard number; traversing within the time window of the target table to which the data belongs, checking whether the distributed timestamp of the current machine is occupied, and if so, adjusting the distributed timestamp according to the total number of machines in the current cluster; otherwise, marking the current distributed timestamp as occupied; and sending the distributed timestamp corresponding to each data to a storage queue and writing it into a time series database according to the table name.

Description

Technical Field

[0001] The present invention relates to the technical field of distributed timestamp generation, and in particular to a distributed timestamp generation method and system based on the number of tables. Background Art

[0002] In many application scenarios, the generation and storage of timestamps are crucial for data integrity and accuracy. Timestamps not only need to accurately record the time when data was generated, but also need to maintain consistency and reliability during subsequent data processing and storage.

[0003] Currently, timestamp generation mainly relies on single-point timestamp systems. These systems usually use data splicing to generate timestamps by combining multiple data fragments.

[0004] However, single-point timestamp systems present significant reliability issues. A single machine failure impacts the entire timestamp generation service, resulting in a loss of record keeping of the data generation time. The splicing method used by existing systems when generating timestamps can lead to significant deviations between the generated timestamps and the actual time of the original data, impacting data accuracy and reliability. Furthermore, in high-concurrency scenarios, existing systems cannot completely avoid the generation of duplicate timestamps, further increasing the complexity and risk of errors in data processing. Summary of the Invention

[0005] The present invention proposes a distributed timestamp generation method and system based on the number of tables, which solves the problem of low reliability of the existing timestamp generation method.

[0006] To solve the above technical problems, the present invention provides a distributed timestamp generation method based on the number of tables, comprising the following steps:

[0007] Step S1: asynchronously receive data through the message queue, perform hash calculation on the key fields in each data, and determine the target table to which each data belongs based on the hash value;

[0008] Step S2: Get the total number of machines in the cluster. Each machine competes for a unique shard number using a distributed lock. Calculate the distributed timestamp of each machine based on the current timestamp and the shard number.

[0009] Step S3: Traverse the time window of the target table to which the data belongs and check whether the distributed timestamp of the current machine is occupied. If so, execute step S4; otherwise, mark the current distributed timestamp as occupied and execute step S5;

[0010] Step S4: Adjust the distributed timestamp according to the total number of machines in the current cluster, and return to step S3;

[0011] Step S5: Send the distributed timestamp corresponding to each data to the storage queue and write it into the time series database according to the table name.

[0012] Preferably, in step S2, the expression for calculating the distributed timestamp of each machine according to the shard number is:

[0013] ;

[0014] Where, is a distributed timestamp; Number the shard of the machine; is the current timestamp; Represents the modulo operation; The total number of machines in the current cluster.

[0015] Preferably, in step S3, the distributed timestamp is used as the center, and a dynamic array is obtained by expanding forward and backward, and the dynamic array is used as the timestamp window. ,in is the window size.

[0016] Preferably, S3 is based on the accuracy of the distributed timestamp and the number of transactions per second. TPS Adjust the size of the time window of the target table. The greater the precision of the distributed timestamp, the smaller the time window. TPS The smaller it is, the smaller the time window is.

[0017] Preferably, the step S4 of adjusting the distributed timestamp according to the total number of machines in the current cluster includes the following steps: advancing the distributed timestamp If the distributed timestamp exceeds the left boundary of the time window when it is pushed forward, the distributed timestamp is pushed backward. step length, where The total number of machines in the current cluster.

[0018] Preferably, in step S2, each machine uses a distributed lock to compete for a unique shard number, including the following steps:

[0019] Step S21: Query the total number of machines in the current cluster , determine the value range of the fragment number ;

[0020] Step S22: Each machine sends a lock request. Only one machine is allowed to obtain the lock at a time. The machine that obtains the lock executes step S23, and the other machines that have not obtained the lock wait for the release of the distributed lock.

[0021] Step S23: From 0 to -1 Iterate over the numbersN , add a fixed prefix to the current number N Splicing, generating the machine identifier;

[0022] Step S24: Check in the coordination center whether the current identifier is occupied. If not, mark the slice number of the current machine as N , otherwise let N = N +1, repeat steps S22 to S23;

[0023] Step S25: Release the distributed lock and return to step S22 until shard numbers are assigned to all machines.

[0024] The present invention also provides a distributed timestamp generation system based on the number of tables, which is implemented based on the above-mentioned distributed timestamp generation method based on the number of tables and includes: a data receiving module, a distributed coordination module, a timestamp generation module and a data storage module;

[0025] The data receiving module receives data sent by an external system through a message queue, extracts key fields from the data, hashes the key fields to determine the target table to which the data belongs, binds the data to the target table, and sends the data to the timestamp generation module;

[0026] The distributed coordination module: maintains the total number of cluster machines through the coordination center And the list of surviving nodes, when a new machine joins, the node is registered, the faulty node is automatically removed, and a distributed lock is used to assign a unique shard number to each machine;

[0027] The timestamp generation module calculates the distributed timestamp of the machine according to the current timestamp of the system and the shard number, checks the occupancy of the distributed timestamp in the independent time window of each target table, and if there is a conflict, calculates the distributed timestamp according to the step size. S Adjust distributed timestamps;

[0028] The data storage module receives the data processed by the timestamp generation module and writes the data into the corresponding time series database according to the target table corresponding to the data.

[0029] Preferably, the system further comprises a fault tolerance and monitoring module, which monitors the health status of machine nodes, automatically isolates faulty nodes, and reallocates shard numbers and data of faulty nodes to healthy nodes.

[0030] The present invention also provides an electronic device, comprising: a memory, a processor and a computer program, wherein the computer program is stored in the memory and is configured to be executed by the processor to implement the above-mentioned distributed timestamp generation method based on the number of tables.

[0031] The present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and the computer program is executed by a processor to implement the above-mentioned distributed timestamp generation method based on the number of tables.

[0032] The benefits of the present invention include at least:

[0033] 1. Data is distributed to different target tables through hash calculation, achieving distributed data storage. This effectively avoids the problem of excessive storage pressure on a single table while ensuring even data distribution.

[0034] 2. The distributed lock competes for the shard number and generates a distributed timestamp based on the current timestamp and the shard number, ensuring the uniqueness of the timestamp and avoiding duplicate timestamps.

[0035] 3. Through the mechanism of distributed lock and shard numbering, the timestamp generation strategy can be dynamically adjusted to avoid timestamp duplication or conflict caused by high concurrency. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 Schematic diagram of a method flow in an embodiment of the present invention;

[0037] Figure 2 This is a schematic diagram of the implementation process architecture of an embodiment of the present invention;

[0038] Figure 3 A schematic diagram of the process of obtaining a shard number by a machine in an embodiment of the present invention;

[0039] Figure 4 This is a schematic diagram of a process for generating table dimension timestamps in a single machine according to an embodiment of the present invention;

[0040] Figure 5 Schematic diagram of the relationship between the timestamp window and the table in an embodiment of the present invention;

[0041] Figure 6 This is an operational logic diagram of the time window of an embodiment of the present invention. DETAILED DESCRIPTION

[0042] The following is a clear and complete description of the technical solutions in the embodiments of the present invention, in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts are within the scope of protection of the present invention.

[0043] like Figure 1 and Figure 2As shown, an embodiment of the present invention provides a distributed timestamp generation method based on the number of tables, comprising the following steps:

[0044] Step S1: asynchronously receive data through the message queue, perform hash calculation on the key fields in each data, and determine the target table to which each data belongs based on the hash value.

[0045] Specifically, before the system officially goes live, the business side evaluates the data requiring timestamp generation to determine the number of tables in advance. This determination is crucial because once the system is running and generating timestamps, the number of tables cannot be dynamically increased. If it is later discovered that the number of tables does not meet the requirements, the current machine must be stopped, the number of tables increased, and the system redeployed. The number of tables is not directly related to the shard number, but a reasonable setting of the number of tables will directly affect the size of the time window and the efficiency of timestamp generation.

[0046] The embodiment of the present invention adopts the method of subscribing to the message queue kafka to asynchronously receive the data that needs to generate a timestamp. After receiving the data, the system will perform a hash calculation based on the key fields in the data to determine which table the data should be stored in.

[0047] Step S2: Get the total number of machines in the cluster. Each machine uses a distributed lock to compete for a unique shard number. The distributed timestamp of each machine is calculated based on the current timestamp and the shard number.

[0048] Specifically, when the timestamp generation cluster is started, each machine first obtains the total number of timestamp generation modules in the cluster through the unified coordination center. After obtaining the total number, each machine uses the distributed lock to compete for the unique shard number. Figure 3 As shown, the following steps are included:

[0049] Step S21: Each machine instance competes for the same distributed lock at the same time. Only one machine can obtain the lock at a time. Other machines that have not obtained the lock enter a waiting state, waiting for the release of the distributed lock. In the embodiment of the present invention, the waiting time is set to 30 seconds.

[0050] Step S22: The machine instance that obtains the distributed lock is from 0 to -1 to loop, assuming the current loop reaches N , then the fixed prefix and the number N Combined into a new identifier key.

[0051] Step S23: Check whether the identifier key is occupied in the unified coordination center. If not, write the identifier to the unified coordination center and set the shard number corresponding to the current machine instance to N , then release the distributed lock; otherwise, execute step S24.

[0052] Step S24: Set N = N +1, regenerate a new identifier, and repeat steps S21 to S23 until all machines successfully write the shard number.

[0053] Since the total number of machines in the current cluster is known, the key written to the unified coordination center is in milliseconds. Distributed locks can ensure that only one machine instance successfully occupies the lock at the same time. Therefore, during the startup process, only one instance will go to the unified coordination center to write the key at the same time. Through the above mechanism, each machine can be guaranteed to obtain a unique shard number. N ,and N The range is from 0 to -1, providing guarantee for the generation of subsequent timestamps.

[0054] Step S3: Traverse the time window of the target table to which the data belongs, and check whether the distributed timestamp of the current machine is occupied. If it is occupied, execute step S4; otherwise, mark the current distributed timestamp as occupied and execute step S5.

[0055] Step S4: Adjust the distributed timestamp according to the total number of machines in the current cluster, and return to step S3.

[0056] Step S5: Send the distributed timestamp corresponding to each data to the storage queue and write it into the time series database according to the table name.

[0057] Specifically, if Figure 4 As shown, after obtaining the data that needs to generate a timestamp, first obtain the system's current long type timestamp , and then calculate the distributed timestamp :

[0058] ;

[0059] Where, Number the shard of the machine; Represents the modulo operation; The total number of machines in the current cluster.

[0060] This formula ensures that the timestamps generated by each machine are The modulo is equal to the corresponding machine shard number N , thus ensuring that timestamps generated by different machines will not be repeated.

[0061] like Figure 5 As shown, the distributed timestamp is obtained Afterwards, is the starting node, combined with the total number of machines in the cluster , get the final generated timestamp through the following timestamp window. According to the table information of the data allocation, get the timestamp window of the table. The timestamp window is a dynamic array with the current time as the center and extending several seconds before and after. ,in is the window size. Traverse the array and query the generated distributed timestamps Is it occupied? If it is already occupied, subtract the current timestamp from , get a new timestamp, and check the occupancy again. If the timestamp that is advanced reaches the left boundary of the time window, add the current timestamp to the left boundary of the time window. , continue checking until an unoccupied timestamp is found within the window.

[0062] The size of the left and right boundaries of the time window has nothing to do with the shard number, but is indirectly related to the number of tables, the timestamp accuracy required by the business, and the actual transaction volume per second of the timestamp data that needs to be generated. TPS Directly related. Because the data that actually needs to generate timestamps will eventually be stored in separate tables, and each table can only distinguish timestamps at the millisecond level at most, the actual insertion speed of each table cannot exceed 1000 records per second. Therefore, for a type of data, the more tables are divided, the smaller the insertion TPS of each table after hash distribution, and the smaller the time window. Therefore, the size of the time window is related to the number of tables and the TPS of the actual data. It needs to be formulated in advance according to the actual business needs before accessing the data. If the accuracy requirements for generating timestamps are high and the current transaction volume per second for timestamp data needs to be generated, TPS If the requirements are met, the left and right windows can be appropriately reduced. If the accuracy requirement is not high, they can be appropriately increased. This mechanism ensures that the generated timestamps are unique within a single table, avoiding conflicts.

[0063] Based on the acquired timestamp information, the time series data, the generated timestamp information, and the corresponding storage table information are sent to another message queue. After receiving this data, the storage module stores the data in the specified time series database based on the table name and timestamp.

[0064] To verify that the timestamp generation method in this embodiment of the present invention generates non-duplicate timestamps for time series data in a single table, the following assumptions are made:

[0065] Case 1: If Figure 6As shown in the figure, suppose there are two time series data D1 and D2 that need to be stored in table TableA, which are respectively distributed to machine A with shard number N1 and machine B with shard number N2 through the message queue. According to the timestamp generation algorithm of the embodiment of the present invention, the timestamp T1 generated by D1 is the total number of servers The modulo must be equal to N1, and the timestamp T2 generated by D2 is the total number of servers After modulo, it must be equal to N2. At the same time, D1 and D2 are all stored in TableA, so T1 and T2 must be different.

[0066] Scenario 2: Suppose there are two time series data items, D1 and D2, that need to be stored in table TableA. They are assigned to the same machine, shard numbered N1, through a message queue. Since messages in a message queue must be processed sequentially, suppose D1 is timestamped T1 after processing. According to the timestamp generation algorithm, when D2 obtains timestamp T2, T1 is already occupied in the time window. Therefore, the timestamp assigned to T2 must be different from T1.

[0067] An embodiment of the present invention also provides a distributed timestamp generation system based on the number of tables, which is implemented based on the above-mentioned distributed timestamp generation method based on the number of tables, and includes: a data receiving module, a distributed coordination module, a timestamp generation module, a data storage module and a fault tolerance and monitoring module.

[0068] Data receiving module: Receives data sent by the external system through the message queue and sends the data to the message queue in JSON format. The data contains key fields for unified hashing. The key fields in the data are extracted, hashed, and the target table to which the data belongs is determined. The data is bound to the target table and sent to the pending queue of the timestamp generation module.

[0069] Distributed coordination module: maintains the total number of cluster machines through the coordination center And the list of surviving nodes, register the node when a new machine joins, automatically remove the failed node, and use a distributed lock to assign a unique shard number to each machine.

[0070] Timestamp generation module: It is deployed in a cluster manner, and an appropriate number of server instances are deployed in the system according to the size of the business volume. At the same time, a collaborative center is deployed for communication such as quantity confirmation between server instances. An independent cache service is also deployed for the generation of distributed locks during the shard acquisition process. In the embodiment of the present invention, redis is used as a cache, and zookeeper is used as a collaborative center. Redis is used for distributed locks, and zookeeper is used to obtain the total number of servers. The distributed timestamp of the machine is calculated based on the current timestamp of the system and the shard number, and the occupancy of the distributed timestamp is checked in the independent time window of each target table. If there is a conflict, the time is calculated according to the step size. Adjust distributed timestamps.

[0071] The data storage module receives data processed by the timestamp generation module and writes it to the corresponding time series database based on the target table. This module deploys an independent multi-node time series database for data storage and configuration generation of table dimension time windows. In this embodiment of the present invention, the time series database TDengine is used as the storage module for time series data.

[0072] Fault tolerance and monitoring module: monitors the health status of machine nodes, automatically isolates faulty nodes, and reallocates the shard numbers and data of faulty nodes to healthy nodes. If a single-machine instance fails at a certain point in time, the system can automatically exclude the relevant nodes. When a machine instance node fails, it will no longer pull messages from Kafka, and the data originally assigned to the machine instance will be randomly assigned to other normal machines through Kafka's allocation mechanism, and timestamp generation will be completed on other machines. After the operation and maintenance personnel discover a machine failure, they can restart a new instance. The new instance will obtain the shard number corresponding to the faulty machine according to the algorithm rules. At the same time, Kafka will assign the original data to the new machine, thereby dynamically ensuring the availability of the entire system.

[0073] After the system is deployed and operated in a company's intranet, it receives all network switch data within the company, as well as other operation and maintenance time series data. TPS A 6-instance cluster was deployed in the timestamp generation module for 13,000 times. The table dimension was split into 10,000 tables, and 10,000 time windows were generated in the single-machine table structure. It can still run stably when the total amount of time series data stored in a week is about 1TB.

[0074] An embodiment of the present invention further provides an electronic device, comprising: a memory, a processor, and a computer program, wherein the computer program is stored in the memory and configured to be executed by the processor to implement the above-mentioned distributed timestamp generation method based on the number of tables.

[0075] An embodiment of the present invention further provides a computer-readable storage medium, in which a computer program is stored. The computer program is executed by a processor to implement the above-mentioned method for generating distributed timestamps based on the number of tables.

[0076] Most existing timestamp generation technologies utilize non-distributed architectures, making it difficult to meet the ever-increasing demand for time series data generation. Standalone systems present performance bottlenecks and single-point failure risks. Once a machine fails, the entire timestamp generation service will be impacted. Furthermore, existing timestamp generation algorithms often use a random number plus a specified bit approach. This theoretically creates the possibility of duplicate timestamps as data volumes increase indefinitely. Existing timestamp-related patents are primarily focused on embedded systems or electronic devices, with limited application in large-scale distributed application systems, making them difficult to adapt to the storage needs of large-scale time series data.

[0077] The embodiment of the present invention proposes a distributed timestamp generation method and system based on the number of tables. The method and system adopt a distributed architecture and can be infinitely horizontally expanded through the combination of multiple machines and multiple tables, effectively meeting the storage needs of large-scale time series data. By introducing a time window mechanism, as long as the business volume estimation is accurate and the parameters are reasonably adapted, reliable single-table non-duplicate timestamps can be generated to ensure the complete preservation of time series data. In addition, the embodiment of the present invention provides a complete set of time series data storage solutions from timestamp generation to storage, which comprehensively solves the single point failure and timestamp duplication problems in the prior art, and is suitable for time series data storage needs of various scales, especially in distributed large-scale application systems. It has significant advantages.

[0078] The technical features of the above embodiments may be combined in any manner. To simplify the description, not all possible combinations of the technical features in the above embodiments are described. Only preferred embodiments of the present invention are presented. While the description is relatively specific and detailed, it should not be construed as limiting the scope of the present invention. As long as there are no conflicts in the combination of these technical features, they should be considered to be within the scope of this specification.

[0079] It should be noted that, for those skilled in the art, various modifications and improvements can be made without departing from the scope of the present invention, and these modifications and improvements fall within the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be determined by the appended claims.

Claims

1. A distributed timestamp generation method based on the number of tables, characterized in that: The following steps are involved: Step S1: asynchronously receive data through the message queue, perform hash calculation on the key fields in each data, and determine the target table to which each data belongs based on the hash value; Step S2: Get the total number of machines in the cluster. Each machine competes for a unique shard number using a distributed lock. Calculate the distributed timestamp of each machine based on the current timestamp and the shard number. Step S3: With the distributed timestamp as the center, expand forward and backward to obtain a dynamic array, and use the dynamic array as the timestamp window [T2-Δ, T2+Δ], where Δ is the window size. The window size is related to the number of tables and the actual transaction volume (TPS) of the data. Traverse the timestamp window of the target table to which the data belongs, and check whether the distributed timestamp of the current machine is occupied. If so, execute step S4; otherwise, mark the current distributed timestamp as occupied, and execute step S5; Step S4: Adjust the distributed timestamp according to the total number of machines in the current cluster, and return to step S3; Step S5: Send the distributed timestamp corresponding to each data to the storage queue and write it into the time series database according to the table name.

2. The method for generating distributed timestamps based on the number of tables according to claim 1, wherein: The expression for calculating the distributed timestamp of each machine according to the shard number in step S2 is: T2 = T1 - T1% S + N; Where T2 is the distributed timestamp; N is the shard number of the machine; T1 is the current timestamp; % represents the modulo operation; S is the total number of machines in the current cluster.

3. The method for generating distributed timestamps based on the number of tables according to claim 1, wherein: In S3, the size of the time window of the target table is adjusted according to the set precision of the distributed timestamp and the transaction volume per second (TPS). The greater the precision of the distributed timestamp, the smaller the time window; the smaller the transaction volume per second (TPS), the smaller the time window.

4. The method for generating distributed timestamps based on the number of tables according to claim 1, wherein: Adjusting the distributed timestamp according to the total number of machines in the current cluster in step S4 includes the following steps: advancing the distributed timestamp forward by S steps; if the distributed timestamp exceeds the left boundary of the time window when advancing forward, advancing the distributed timestamp backward by S steps, where S is the total number of machines in the current cluster.

5. The method for generating distributed timestamps based on the number of tables according to claim 1, wherein: In step S2, each machine uses a distributed lock to compete for a unique shard number, including the following steps: Step S21: Query the total number of machines S in the current cluster and determine that the range of the shard number is (0, S-1); Step S22: Each machine sends a lock request. Only one machine is allowed to obtain the lock at a time. The machine that obtains the lock executes step S23, and the other machines that have not obtained the lock wait for the release of the distributed lock. Step S23: traverse the number N from 0 to S-1, concatenate the fixed prefix with the current number N, and generate the machine identifier; Step S24: Query the coordination center to see if the current identifier is occupied. If not, mark the slice number of the current machine as N. Otherwise, set N=N+1 and repeat steps S22 to S23. Step S25: Release the distributed lock and return to step S22 until shard numbers are assigned to all machines.

6. A distributed timestamp generation system based on the number of tables, implemented based on the distributed timestamp generation method based on the number of tables according to any one of claims 1 to 5, characterized in that: include: Data receiving module, distributed coordination module, timestamp generation module and data storage module; The data receiving module receives data sent by an external system through a message queue, extracts key fields from the data, hashes the key fields to determine the target table to which the data belongs, binds the data to the target table, and sends the data to the timestamp generation module; The distributed coordination module: maintains the total number of cluster machines S and the list of surviving nodes through the coordination center, registers the node when a new machine joins, automatically removes the faulty node, and uses a distributed lock to assign a unique shard number to each machine; The timestamp generation module calculates the distributed timestamp of the machine according to the current timestamp of the system and the shard number, checks the occupancy of the distributed timestamp in the independent time window of each target table, and adjusts the distributed timestamp by the step size S if there is a conflict; The data storage module receives the data processed by the timestamp generation module and writes the data into the corresponding time series database according to the target table corresponding to the data.

7. The distributed timestamp generation system based on the number of tables according to claim 6, characterized in that: The system also includes a fault tolerance and monitoring module, which monitors the health status of machine nodes, automatically isolates faulty nodes, and reallocates shard numbers and data of faulty nodes to healthy nodes.

8. An electronic device comprising: A memory, a processor and a computer program, characterized in that: the computer program is stored in the memory and is configured to be executed by the processor to implement the distributed timestamp generation method based on the number of tables as described in any one of claims 1 to 5.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is executed by a processor to implement the distributed timestamp generation method based on the number of tables according to any one of claims 1 to 5.

Images