Method and system for generating distributed id based on snowflake algorithm
By adding callback segments in the snowflake algorithm and using Redis cache counters, the ID duplication problem caused by clock callback in the snowflake algorithm is solved, efficient distributed ID generation and uniqueness in distributed systems are achieved, and the Internet of Things is improved The system's ID generation experience.
Patent Information
- Application Number
- CN202411316982.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-20
- Publication Date
- 2024-12-27
AI Technical Summary
The existing snowflake algorithm has defects such as ID length limitation, different machine clocks and clock callbacks in distributed systems, resulting in duplication of IDs and poor user experience.
By splitting long integer binary bits, adding a callback segment, and saving the last callback counter in the Redis cache, adding 1 when a clock callback occurs, regenerating the ID, ensuring ID uniqueness, and efficiently generating it in the Internet of Things system Distributed ID.
It effectively solves the problem of ID duplication caused by clock callback, ensures efficient generation and uniqueness of IDs in the Internet of Things system, and can handle callback problems even in abnormal restarts.
Smart Images
Figure CN119203925A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of generating unique IDs for distributed systems, and in particular to a method and system for generating distributed IDs based on a snowflake algorithm. Background Art
[0002] In the current era of big data, distributed systems are widely used in the fields of Internet, cloud computing, Internet of Things, and artificial intelligence. In these systems, generating unique and ordered identifiers is crucial for tasks such as data management, indexing, and tracking.
[0003] Traditional ID generation methods face some challenges. For example, in a distributed environment, there are concurrency issues with multiple machines generating IDs at the same time, and confusing timestamps may lead to ID conflicts. In addition, the formats of IDs generated on different machines may be inconsistent, which brings difficulties to data exchange and system collaborative development.
[0004] The snowflake algorithm is a commonly used distributed ID generation algorithm. By using a 64-bit integer as a unique identifier, it has sufficient capacity to ensure that unique IDs are generated in multiple machines or distributed environments. However, the existing snowflake algorithm divides the 64-bit binary into sign bit / reserved bit, time period, data center ID and machine identifier ID and serial number segment. The time period is obtained from the current machine clock. The machine clock is subject to the risk of dialback correction. Once the clock dialback occurs, the generated distributed ID will be repeated, affecting the business.
[0005] Patent document CN118394843A (application number: 202410606272.X) discloses a distributed ID generation optimization method, system, medium and equipment based on the snowflake algorithm, which belongs to the technical field of distributed system unique ID generation. The technical problem to be solved by this invention is how to optimize the defects of the snowflake algorithm such as ID length limitation, different machine clocks and callback, so as to improve the user experience in the actual production environment. The technical solution adopted is: initialization configuration: configure and allocate data center ID and machine identification ID to ensure uniqueness and distinguish different machines and data centers; generate ID, as follows: obtain current timestamp: use a high-precision clock source to obtain the current timestamp; determine time callback: compare with the previously generated timestamp: if an event callback occurs, wait or use compensation method to process; generate serial number: generate a unique serial number within the same millisecond; assemble ID: assemble the data center ID, machine identification ID, timestamp and serial number into a 64-bit integer ID in sequence. Summary of the invention
[0006] In view of the defects in the prior art, the purpose of the present invention is to provide a system and method for generating distributed IDs based on the snowflake algorithm.
[0007] A system for generating a distributed ID based on a snowflake algorithm according to the present invention includes:
[0008] A distributed ID interface module is used to obtain a request for generating a distributed ID and transmit the obtained request for generating a distributed ID to a distributed ID core module;
[0009] The distributed ID core module is used to receive requests for generating distributed IDs and generate distributed IDs through the snowflake algorithm;
[0010] A service information acquisition module is used to obtain the service signal of the current service;
[0011] A callback management module, used for maintaining a callback counter based on the service signal of the current service obtained;
[0012] The Redis cache module is used to receive and save the echo counter from the callback management module and provide the callback counter query interface to the callback management module.
[0013] Preferably, the distributed ID core module includes:
[0014] Module M2.1: Receives IP information from the service information query module and intercepts the last two segments of the IP to generate machine code;
[0015] Module M2.2: Receives machine time from the service information query module and saves it as a time period;
[0016] Module M2.3: Compare the acquired machine time with the local time, and add 1 to the serial number counter when they are in the same time period. When the time period changes, the serial number counter is reset to 0;
[0017] Module M2.4: receiving the callback counter of the callback management module;
[0018] Module M2.5: Generates a distributed ID based on the generated machine code, time period, callback counter, and sequence number counter according to binary shift operations.
[0019] Preferably, the distributed ID in the module M2.5 includes: the time period is moved to the 22nd to 62nd bit through a bit shift operation, the machine code is moved to the 12th to 21st bit through a bit shift operation, the callback segment is moved to the 10th to 11th bit, and the serial number counter is moved to the 0th to 9th bit.
[0020] Preferably, the callback management module includes:
[0021] Module M4.1: The distributed ID core module transmits the service name and IP received from the service information acquisition module to the callback management module. The callback management module queries the last callback counter of the corresponding service through the Redis cache module based on the obtained service name and IP, and updates it to the local callback counter;
[0022] Module M4.1: Receives the machine time from the service information acquisition module and compares it with the local time to determine whether a clock callback occurs. When a clock callback occurs, the callback counter is incremented by 1, and then a bitwise AND operation is performed with the maximum value of the callback counter to obtain a new callback counter and update the local time.
[0023] Preferably, the Redis cache module includes: caching the new callback counter into Redis through the Redis cache module.
[0024] A method for generating a distributed ID based on a snowflake algorithm according to the present invention includes:
[0025] Step S1: using the distributed ID interface module to obtain a distributed ID generation request, and transmitting the obtained distributed ID generation request to the distributed ID core module;
[0026] Step S2: Use the distributed ID core module to receive a distributed ID generation request and generate a distributed ID through the snowflake algorithm.
[0027] Preferably, step S2 comprises:
[0028] Step S2.1: Receive IP information from the service information query module, and intercept the last two segments of the IP to generate machine code;
[0029] Step S2.2: receiving the machine time from the service information query module and saving it as a time period;
[0030] Step S2.3: Compare the obtained machine time with the local time, and when they are in the same time period, the serial number counter is increased by 1. When the time period changes, the serial number counter is reset to 0;
[0031] Step S2.4: receiving a callback counter from the callback management module;
[0032] Step S2.5: Generate a distributed ID according to a binary shift operation based on the generated machine code, time period, callback counter, and sequence number counter;
[0033] The distributed ID in step S2.5 includes: the time period is moved to the 22nd to 62nd bit through the bit shift operation, the machine code is moved to the 12th to 21st bit through the bit shift operation, the callback segment is moved to the 10th to 11th bit, and the serial number counter is moved to the 0th to 9th bit.
[0034] Preferably, the method further comprises:
[0035] Step S3: using the service information acquisition module to acquire the service signal of the current service;
[0036] Step S4: maintaining a callback counter based on the service signal of the current service acquired by the callback management module;
[0037] Step S5: using the Redis cache module to receive and save the echo counter from the callback management module, and providing the callback counter query interface to the callback management module.
[0038] Preferably, step S4 comprises:
[0039] Step S4.1: The distributed id core module transmits the service name and IP received from the service information acquisition module to the callback management module. The callback management module queries the last callback counter of the corresponding service through the Redis cache module based on the obtained service name and IP, and updates it to the local callback counter;
[0040] Step S4.2: Receive the machine time from the service information acquisition module and compare it with the local time to determine whether the clock is dialed back. When the clock is dialed back, the dial-back counter is incremented by 1, and then a bitwise AND operation is performed with the maximum value of the dial-back counter to obtain a new dial-back counter, and the local time is updated.
[0041] Preferably, step S5 includes: caching the new callback counter into Redis through a Redis cache module.
[0042] Compared with the prior art, the present invention has the following beneficial effects:
[0043] 1. The present invention adopts splitting long integer binary bits to increase the callback segment. When the clock is dialed back, the callback segment is increased by 1, thereby solving the problem of ID duplication during clock dialing back.
[0044] 2. The present invention caches the last callback counter through Redis, thereby solving the callback problem within the restart time even if the IoT service restarts abnormally;
[0045] 3. The present invention comprehensively applies Redis and binary bits, and divides the 64 binary bits of the long integer into the highest reserved bit (1 bit), time period (41 bits), machine code segment (10 bits), callback segment (2 bits), and sequence number segment (10 bits). When the clock is dialed back, the callback segment is increased by 1, so that the clock is dialed back and no duplicate ID is generated, and distributed ID generation is efficiently realized in the Internet of Things system. BRIEF DESCRIPTION OF THE DRAWINGS
[0046] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:
[0047] Figure 1 A schematic diagram of a system for generating distributed IDs based on the snowflake algorithm.
[0048] Figure 2 Generate a flow chart for distributed id.
[0049] Figure 3 Schematic diagram of distributed ID binary bits. DETAILED DESCRIPTION
[0050] The present invention is described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those of ordinary skill in the art, several changes and improvements can also be made without departing from the concept of the present invention. These all belong to the protection scope of the present invention.
[0051] Example 1
[0052] According to the present invention, a system for generating distributed IDs based on a snowflake algorithm is provided, such as Figure 1-3 As shown, including:
[0053] A service information acquisition module, used to acquire service signals;
[0054] A callback management module, used for maintaining a callback counter based on the acquired service signal;
[0055] Redis cache module, used to receive and save the echo counter from the callback management module, and provide the callback counter query interface to the callback management module;
[0056] The distributed ID core module is used to receive a request to generate a distributed ID and generate a distributed ID through a snowflake algorithm; and transmit the generated distributed ID to the distributed ID interface module;
[0057] A distributed ID interface module is used to obtain a request for generating a distributed ID and transmit the obtained request for generating a distributed ID to a distributed ID core module;
[0058] The service information acquisition module
[0059] Provide service name and IP query, and provide query function for the callback management module.
[0060] Provide IP query and IP information query function for the distributed ID core module.
[0061] Provide machine time query function to the distributed ID core module and callback management module.
[0062] The call-back management module includes:
[0063] Maintain a callback counter.
[0064] Maintain a callback counter with a maximum value of 3.
[0065] The distributed id core module receives the service name and IP from the service information acquisition module, queries the last callback counter through the Redis cache module, and updates the local callback counter.
[0066] Receive the machine time from the service information acquisition module and compare it with the local time to determine whether the clock is dialed back. If the clock is dialed back, the dialback counter is incremented by 1, and then the new dialback counter is obtained by bitwise AND operation with the maximum value of the dialback counter. Then update the local time.
[0067] The new callback counter is cached in Redis through the Redis cache module.
[0068] The Redis cache module includes:
[0069] Receive the callback counter from the callback management module and save it to Redis.
[0070] Provide a callback counter query interface to the callback management module.
[0071] The distributed id core module includes:
[0072] Receive the IP information from the service information query module, and intercept the last two segments of the IP to generate machine code.
[0073] The time received from the service information acquisition module is saved as a time period.
[0074] Generation: Sequence number counter. The sequence number counter is incremented by 1 in the same time period. When the time period changes, the sequence number counter is reset to 0.
[0075] Receive the callback counter from the callback management module and save it locally.
[0076] The above machine code, time period, callback counter, and serial number counter are used to generate a long integer number according to the binary shift operation. The time period is moved to the 22nd to 62nd bit through the shift operation, the machine code is moved to the 12th to 21st bit through the shift operation, the callback segment is moved to the 10th to 11th bit, and the serial number counter is moved to the 0th to 9th bit. The above operation generates a distributed ID and outputs it to the distributed ID interface module.
[0077] The distributed id interface module includes:
[0078] Receive distributed ID generation request and call the distributed ID core module to generate distributed ID.
[0079] Receive the ID generated by the distributed ID core module and provide an ID generation interface for the IoT service business function module.
[0080] The present invention also provides a system for generating distributed IDs based on a snowflake algorithm. The system for generating distributed IDs based on a snowflake algorithm can be implemented by executing the process steps of the method for generating distributed IDs based on a snowflake algorithm, that is, those skilled in the art can understand the method for generating distributed IDs based on a snowflake algorithm as a preferred implementation of the system for generating distributed IDs based on a snowflake algorithm.
[0081] Those skilled in the art know that, in addition to realizing the system and its various devices, modules, and units provided by the present invention in a purely computer-readable program code, the system and its various devices, modules, and units provided by the present invention can be completely realized in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps to realize the same functions. Therefore, the system and its various devices, modules, and units provided by the present invention can be considered as a hardware component, and the devices, modules, and units included therein for realizing various functions can also be regarded as structures within the hardware component; the devices, modules, and units for realizing various functions can also be regarded as both software modules for realizing the method and structures within the hardware component.
[0082] The above describes the specific embodiments of the present invention. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art can make various changes or modifications within the scope of the claims, which does not affect the essence of the present invention. In the absence of conflict, the embodiments of the present application and the features in the embodiments can be combined with each other arbitrarily.
Claims
1. A system for generating distributed IDs based on a snowflake algorithm, characterized in that: include: A distributed ID interface module is used to obtain a request for generating a distributed ID and transmit the obtained request for generating a distributed ID to a distributed ID core module; The distributed ID core module is used to receive requests for generating distributed IDs and generate distributed IDs through the snowflake algorithm; A service information acquisition module is used to obtain the service signal of the current service; A callback management module, used for maintaining a callback counter based on the service signal of the current service obtained; The Redis cache module is used to receive and save the echo counter from the callback management module and provide the callback counter query interface to the callback management module.
2. The system for generating distributed IDs based on the snowflake algorithm according to claim 1, characterized in that: The distributed ID core module includes: Module M2.1: Receives IP information from the service information query module and intercepts the last two segments of the IP to generate machine code; Module M2.2: Receives machine time from the service information query module and saves it as a time period; Module M2.3: Compare the acquired machine time with the local time, and add 1 to the serial number counter when they are in the same time period. When the time period changes, the serial number counter is reset to 0; Module M2.4: receiving the callback counter of the callback management module; Module M2.5: Generates a distributed ID based on the generated machine code, time period, callback counter, and sequence number counter according to binary shift operations.
3. The system for generating distributed IDs based on the snowflake algorithm according to claim 2, characterized in that: The distributed ID in the module M2.5 includes: the time period is moved to the 22nd to 62nd bit through the bit shift operation, the machine code is moved to the 12th to 21st bit through the bit shift operation, the callback segment is moved to the 10th to 11th bit, and the serial number counter is moved to the 0th to 9th bit.
4. The system for generating distributed IDs based on the snowflake algorithm according to claim 1, characterized in that: The call-back management module includes: Module M4.1: The distributed ID core module transmits the service name and IP received from the service information acquisition module to the callback management module. The callback management module queries the last callback counter of the corresponding service through the Redis cache module based on the obtained service name and IP, and updates it to the local callback counter; Module M4.1: Receives the machine time from the service information acquisition module and compares it with the local time to determine whether a clock callback occurs. When a clock callback occurs, the callback counter is incremented by 1, and then a bitwise AND operation is performed with the maximum value of the callback counter to obtain a new callback counter and update the local time.
5. The system for generating distributed IDs based on the snowflake algorithm according to claim 1, characterized in that: The Redis cache module includes: caching the new callback counter into Redis through the Redis cache module.
6. A method for generating distributed IDs based on a snowflake algorithm, characterized in that: include: Step S1: using the distributed ID interface module to obtain a distributed ID generation request, and transmitting the obtained distributed ID generation request to the distributed ID core module; Step S2: Use the distributed ID core module to receive a distributed ID generation request and generate a distributed ID through the snowflake algorithm.
7. The method for generating distributed ID based on snowflake algorithm according to claim 6, characterized in that: The step S2 comprises: Step S2.1: Receive IP information from the service information query module, and intercept the last two segments of the IP to generate machine code; Step S2.2: receiving the machine time from the service information query module and saving it as a time period; Step S2.3: Compare the obtained machine time with the local time, and when they are in the same time period, the serial number counter is increased by 1. When the time period changes, the serial number counter is reset to 0; Step S2.4: receiving a callback counter from the callback management module; Step S2.5: Generate a distributed ID according to a binary shift operation based on the generated machine code, time period, callback counter, and sequence number counter; The distributed ID in step S2.5 includes: the time period is moved to the 22nd to 62nd bit through the bit shift operation, the machine code is moved to the 12th to 21st bit through the bit shift operation, the callback segment is moved to the 10th to 11th bit, and the serial number counter is moved to the 0th to 9th bit.
8. The method for generating distributed ID based on snowflake algorithm according to claim 6, characterized in that: The method further comprises: Step S3: using the service information acquisition module to acquire the service signal of the current service; Step S4: maintaining a callback counter based on the service signal of the current service acquired by the callback management module; Step S5: using the Redis cache module to receive and save the echo counter from the callback management module, and providing the callback counter query interface to the callback management module.
9. The method for generating distributed IDs based on the snowflake algorithm according to claim 8, characterized in that: The step S4 comprises: Step S4.1: The distributed id core module transmits the service name and IP received from the service information acquisition module to the callback management module. The callback management module queries the last callback counter of the corresponding service through the Redis cache module based on the obtained service name and IP, and updates it to the local callback counter; Step S4.2: Receive the machine time from the service information acquisition module and compare it with the local time to determine whether the clock is dialed back. When the clock is dialed back, the dial-back counter is incremented by 1, and then a bitwise AND operation is performed with the maximum value of the dial-back counter to obtain a new dial-back counter, and the local time is updated.
10. The method for generating distributed ID based on snowflake algorithm according to claim 8, characterized in that: The step S5 includes: caching the new callback counter into Redis through the Redis cache module.