Distributed ID generation method and device, computer equipment and storage medium

Abstract

The invention discloses a distributed ID generation method and device, computer equipment and a storage medium, and relates to the field of cloud deployment, and the method comprises the steps: deploying a plurality of Redis clusters in different machine rooms; when the client applies for the ID, sending the application to the load balancer and forwarding the application to a plurality of Redis clusters through the load balancer; acquiring a global unique latest timestamp and a machine identifier through a Redis cluster; generating an ID according to the globally unique latest timestamp and the machine identifier by using the INCR operation of Redis; and returning the ID to the client. According to the method, the global unique ID is generated in a distributed environment by using the atomic operation of Redis, and the generation of the ID does not need additional services, so that the architecture of the system is simplified, and the problem of single-point failure is avoided. By deploying the Redis cluster in different machine rooms, cross-machine-room distributed ID generation is realized, and the availability and expansibility of the system are improved. In addition, the length of the generated ID is short, and the efficiency in the data storage and transmission process is improved.

Description

Technical Field

[0001] The present invention relates to the field of cloud deployment, and in particular to a distributed ID generation method, device, computer equipment and storage medium. Background Art

[0002] With the rapid development of Internet technology, distributed systems have become an indispensable infrastructure for modern Internet applications and are widely used in many key areas such as distributed cache and distributed database. In the distributed system architecture, ensuring the correctness and consistency of data interaction between nodes is one of the core elements to maintain the stability and reliability of the entire system. Among them, the technology of generating a globally unique ID (GUID) plays a vital role. The globally unique ID is not only used to identify each entity or data record in the system, but also ensures the uniqueness and traceability of data in a distributed environment.

[0003] In response to this demand, the industry has proposed and implemented a variety of technical solutions. Twitter's Snowflake algorithm is one of the more well-known ones. By combining elements such as timestamps, work machine IDs, and serial numbers, it can efficiently generate globally unique 64-bit IDs. However, the Snowflake algorithm relies on a centralized ID generation service, which requires the service to maintain high availability and high performance. Once this independent node fails, the entire system will be unable to generate a new ID, seriously affecting the system's ability to continue operating. In addition, the existence of this centralized node may also become a bottleneck for system expansion, limiting the system's horizontal expansion capabilities.

[0004] Another common solution is to use UUID (Universally Unique Identifier), which is generated based on random numbers and / or specific algorithms to ensure global uniqueness. Although UUID performs well in terms of uniqueness, its long length (usually 128 bits) brings additional burden to data storage and transmission, reducing the overall efficiency of the system, especially in scenarios that require frequent reading and writing or transmission of large amounts of data.

[0005] As a traditional and simple way to generate IDs, database auto-increment IDs perform well in a single-machine environment, ensuring the uniqueness and increment of IDs through the internal mechanism of the database. However, in a distributed system, especially in a cross-computer room deployment scenario, auto-increment IDs cannot ensure global uniqueness, because each independent database instance will generate IDs according to its own sequence, which may lead to ID conflicts and destroy the consistency and integrity of the data. Summary of the invention

[0006] The purpose of the present invention is to provide a distributed ID generation method, apparatus, computer equipment and storage medium, aiming to solve the problems that existing ID generation methods either require an independent ID generation service, or the generated ID is long, or are only applicable to a single-machine environment.

[0007] In a first aspect, an embodiment of the present invention provides a distributed ID generation method, including:

[0008] Deploy multiple Redis clusters in different computer rooms;

[0009] When a client applies for an ID, the application is sent to a load balancer and the load balancer forwards the application to multiple Redis clusters;

[0010] Obtain the globally unique latest timestamp and machine ID through the Redis cluster;

[0011] Generate an ID based on the globally unique latest timestamp and machine ID using the INCR operation of Redis;

[0012] The ID is returned to the client.

[0013] In a second aspect, an embodiment of the present invention provides a distributed ID generation device, including:

[0014] Deployment unit, used to deploy multiple Redis clusters in different computer rooms;

[0015] A sending unit, configured to send the application to a load balancer when the client applies for an ID, and forward the application to multiple Redis clusters through the load balancer;

[0016] An acquisition unit, used to acquire the latest globally unique timestamp and machine identifier through the Redis cluster;

[0017] A generating unit, configured to generate an ID according to the globally unique latest timestamp and the machine identifier and using an INCR operation of Redis;

[0018] The returning unit is used to return the ID to the client.

[0019] In a third aspect, an embodiment of the present invention further provides a computer device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the distributed ID generation method described in the first aspect when executing the computer program.

[0020] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the distributed ID generation method described in the first aspect above is implemented.

[0021] The present invention discloses a distributed ID generation method, device, computer equipment and storage medium, the method comprising: deploying multiple Redis clusters in different computer rooms respectively; when a client applies for an ID, sending the application to a load balancer and forwarding the application to multiple Redis clusters through the load balancer; obtaining the globally unique latest timestamp and machine identifier through the Redis cluster; generating an ID based on the globally unique latest timestamp and machine identifier and using the INCR operation of Redis; and returning the ID to the client. The present invention uses Redis's atomic operation to realize the generation of a globally unique ID in a distributed environment, and the generation of the ID does not require additional services, which simplifies the system architecture and avoids the problem of single point failure. By deploying Redis clusters in different computer rooms, distributed ID generation across computer rooms is realized, which improves the availability and scalability of the system. In addition, the length of the generated ID is short, which improves the efficiency of data storage and transmission, and reduces the consumption of bandwidth and storage costs during data storage and transmission. The embodiment of the present invention also provides a distributed ID generation device, a computer-readable storage medium and a computer device, which have the above-mentioned beneficial effects and are not repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying any creative work.

[0023] Figure 1 A flowchart of a distributed ID generation method;

[0024] Figure 2 It is a schematic block diagram of a distributed ID generation device. DETAILED DESCRIPTION

[0025] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0026] It should be understood that when used in this specification and the appended claims, the terms "include" and "comprises" indicate the presence of described features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.

[0027] It should also be understood that the terms used in this specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the specification of the present invention and the appended claims, the singular forms "a", "an" and "the" are intended to include plural forms unless the context clearly indicates otherwise.

[0028] It should be further understood that the term "and / or" used in the present description and the appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0029] See also Figure 1 , this embodiment provides a distributed ID generation method, including:

[0030] S101: Deploy multiple Redis clusters in different computer rooms;

[0031] Specifically, multiple Redis clusters are deployed in different computer rooms, and multiple Redis nodes are deployed in each computer room. For example, three Redis nodes are deployed in each computer room, and each node is an independent Redis cluster.

[0032] Multiple Redis nodes are deployed in each computer room (each node constitutes an independent cluster). Even if a node in the computer room fails, other nodes in the cluster can continue to provide services through data replication and failover mechanisms, ensuring high availability within the computer room.

[0033] S102: When a client applies for an ID, the application is sent to a load balancer, and the load balancer forwards the application to multiple Redis clusters;

[0034] S103: Obtaining the latest globally unique timestamp and machine ID through the Redis cluster;

[0035] Wherein, obtaining the globally unique latest timestamp and machine ID through the Redis cluster is to obtain the globally unique latest timestamp and machine ID in each Redis node.

[0036] Since each Redis node is independent, it can process multiple requests at the same time, improving the concurrency capability of the system.

[0037] In this embodiment, obtaining the globally unique latest timestamp includes:

[0038] Get the latest local timestamp and the latest timestamp stored in the Redis cluster;

[0039] Compare the latest local timestamp with the latest timestamp stored in the Redis cluster, and use the latest timestamp after comparison as the globally unique latest timestamp.

[0040] By comparing the latest local timestamp with the latest timestamp stored in the Redis cluster, it is possible to ensure that the IDs of all distributed services are increasing, thereby preventing ID duplication caused by timestamp rollback.

[0041] In this embodiment, after obtaining the globally unique latest timestamp, the method includes: saving the globally unique latest timestamp to the Redis cluster, and obtaining the latest timestamp of the Redis cluster.

[0042] As a distributed storage system, Redis cluster can maintain data consistency among multiple nodes. By storing globally unique timestamps in Redis cluster, it can ensure that all nodes can access the latest timestamp information, thus avoiding data inconsistency problems that may occur in distributed systems.

[0043] In this embodiment, obtaining the latest timestamp of the Redis cluster includes: changing the latest timestamp of the Redis cluster to a local timestamp.

[0044] Changing the latest timestamp of the Redis cluster to the local timestamp can ensure the legitimacy of the local ID.

[0045] S104: Generate an ID using the INCR operation of Redis according to the globally unique latest timestamp and the machine identifier;

[0046] In this embodiment, the ID includes: a globally unique latest timestamp, a machine identifier, and a single-machine buffer timestamp.

[0047] Specifically, the globally unique latest timestamp and machine ID are concatenated as the prefix of each ID and the Redis INCR operation is used to generate a local 19-digit distributed ID; among them, the first 3 digits record the machine ID (i.e. machine ID), the middle 13 digits record the globally unique latest timestamp, and the last 3 digits are used as a single-machine buffer timestamp.

[0048] The globally unique latest timestamp can ensure that each ID is unique, and the machine ID can ensure that IDs generated by different machines will not be repeated.

[0049] In this embodiment, the following steps are performed to generate an ID based on the globally unique latest timestamp and the machine identifier and using the INCR operation of Redis:

[0050] Compare the ID with the latest ID recorded in the Redis cluster;

[0051] If the ID is greater than the latest ID recorded in the Redis cluster, the ID is returned to the client and updated to the latest ID recorded in the Redis cluster;

[0052] If the ID is less than or equal to the latest ID recorded in the Redis cluster, a new ID is generated.

[0053] Each time the latest ID is generated, it will be compared with the latest ID recorded in the Redis cluster to see if it is the latest. Because the ID is a long number, it can be used as long as the number is larger than the latest ID recorded in the Redis cluster. Otherwise, it needs to be regenerated. When comparing, the current latest ID (a long number) is compared with the latest ID (a long number) recorded in the Redis cluster.

[0054] By comparing the generated ID with the latest ID recorded in the Redis cluster, the uniqueness and order of the ID can be further ensured. If the generated ID is less than or equal to the latest ID recorded in the Redis cluster, it is regenerated to ensure that the ID is always incremented.

[0055] In this embodiment, after generating an ID according to the globally unique latest timestamp and the machine identifier using the INCR operation of Redis, the method further includes: setting an expiration time of the ID by using the EXPIRE operation of Redis.

[0056] Through the Redis EXPIRE operation, you can set a clear expiration time for each ID. This means that after the ID is generated, it will only be valid for a specific period of time. Once expired, the ID will be automatically deleted or invalidated, thus avoiding potential problems caused by the long-term existence of the ID, such as data pollution or resource occupation.

[0057] S105: Return the ID to the client.

[0058] This embodiment realizes distributed ID generation across computer rooms by deploying Redis clusters in different computer rooms, thereby avoiding the single point failure problem and significantly improving the availability of the system. In addition, the solution can realize distributed ID generation across computer rooms in a distributed environment, and can be easily expanded horizontally as business needs grow. In this solution, ID generation does not require additional services, simplifies the system architecture, and avoids the single point failure problem. This design makes the system more robust and reliable, because even if a node fails, other nodes can still continue to work to ensure the normal operation of the entire system. At the same time, due to the use of Redis atomic operations, the uniqueness and consistency of IDs are ensured in high concurrency situations. Another important advantage is that the ID length (i.e., the length of the JAVA language long data type) generated by this solution is short, which is conducive to improving efficiency during data storage and transmission.

[0059] In this embodiment, in a cross-computer room scenario, multiple Redis instances can be deployed, and IDs can be generated collaboratively through certain strategies. One strategy is to assign a specific ID range to each Redis instance. For example, suppose there are three Redis instances, namely Redis-A, Redis-B, and Redis-C. Redis-A can be assigned an ID range of 1-1000000, Redis-B can be assigned 1000001-2000000, and Redis-C can be assigned 2000001-3000000. When it is necessary to generate an ID, the system selects a suitable Redis instance to generate an ID according to a certain algorithm (such as based on business type, computer room load, etc.). In this way, the resources of multiple Redis instances can be fully utilized, the efficiency and reliability of ID generation can be improved, and the global uniqueness of the ID can also be guaranteed.

[0060] In this embodiment, a scheduled time can be set to perform an RDB backup, while AOF persistence is enabled, and the AOF file is rewritten at regular intervals to reduce the file size. The scheduled time can be set according to the actual usage scenario.

[0061] In some embodiments, the global uniqueness of the ID can be ensured by configuring different starting values ​​and step sizes. Suppose there are two computer rooms A and B, and each computer room has a Redis instance deployed. The starting value for the Redis instance in computer room A can be configured to be 1 and the step size can be 2; the starting value for the Redis instance in computer room B can be configured to be 2 and the step size can be 2. In this way, the IDs generated by the two instances will not be repeated, and can cover the ID requirements of the entire distributed system. When new business data needs to generate an ID, the system can select one of the Redis instances to generate the ID according to certain strategies (such as computer room load balancing, etc.).

[0062] This approach allows for flexible adjustment of the start value and step size based on actual application scenarios and requirements. As the system or application expands, the parameters can be easily adjusted to accommodate larger ID requirements without major modifications to the system. By configuring different start values ​​and step sizes for different systems, modules, or instances, the possibility of ID conflicts can be greatly reduced or eliminated.

[0063] In some embodiments, the master-slave relationship of the Redis instance is configured, and the master node and the slave node are specified. The Redis service is started to ensure that the master and slave nodes are connected normally. The status of the master and slave nodes is monitored to ensure the consistency of data synchronization.

[0064] Configure the master-slave relationship of Redis instances, specify master nodes and slave nodes, start Redis services, and ensure normal connection and data synchronization consistency between master and slave nodes. It has the flexibility to improve data availability, achieve data redundancy and backup, and configuration and management.

[0065] Furthermore, when the slave node sends the psyncreplDkoffset command, the master node responds and sends the fullresyncreplDoffset command, and then the master node executes the bgsave command to generate an RDB (RedisDatabase) file and sends this file to the slave node. After receiving the RDB file, the slave node will clear the memory data and load the data of the RDB file into the memory. In addition, the master node will also store the write instructions generated during this period in the buffer replication_buffer, and after the slave node loads the RDB file, it will send the content in replication_buffer to the slave node to ensure that the data of the slave node is completely consistent with that of the master node.

[0066] By synchronizing the master node and the slave node incrementally, the amount of data transmission can be reduced and the synchronization efficiency between the master node and the slave node can be improved.

[0067] Furthermore, when the slave node sends a psync command to the master node, along with the offset and replid of its own replication backlog buffer, the master node records all subsequent write operations in the replication_buffer in memory. Then, the master node sends the saved write operations to the slave node based on the information sent by the slave node.

[0068] By sending the PSYNC command and the corresponding offset and replid, the slave node can tell the master node its current data status. If the slave node has been synchronized with the master node before and the disconnection time is not long (that is, the offset is within the replication backlog buffer of the master node), the master node can directly send incremental data to the slave node starting from the offset, avoiding the overhead of full synchronization.

[0069] In some embodiments, a sentinel node is configured to specify the Redis master and slave nodes to be monitored. The sentinel service is started to ensure that the sentinel node normally monitors the status of the Redis master and slave nodes. When the master node fails, the sentinel node will automatically fail over to ensure the high availability of the Redis service.

[0070] Sentinel nodes can continuously monitor the status of Redis master and slave nodes. Once the master node fails, the sentinel node will quickly detect the change. Through the internal election mechanism, the sentinel node can automatically select a slave node as the new master node and perform failover. This process does not require manual intervention, greatly improving the high availability of the Redis service.

[0071] In some embodiments, configure the Redis cluster node and specify the node's IP address and port number. Use tools such as redis-cli to create a Redis cluster and allocate hash slots. Start the Redis cluster service to ensure that each node is connected and communicates normally. At the application level, configure the Redis client to connect to the Redis cluster to achieve distributed storage and access of data.

[0072] Redis cluster can improve system availability through data sharding and data replication. Even if some nodes fail, other nodes can still continue to provide services to ensure business continuity. Redis cluster uses hash slots to distribute data to ensure that data is evenly distributed among nodes.

[0073] See also Figure 2 This embodiment provides a distributed ID generation device 200, including:

[0074] A deployment unit 201 is used to deploy multiple Redis clusters in different computer rooms;

[0075] A sending unit 202 is used for sending the application to a load balancer when the client applies for an ID, and forwarding the application to multiple Redis clusters through the load balancer;

[0076] An acquisition unit 203 is used to acquire a globally unique latest timestamp and a machine identifier through the Redis cluster;

[0077] A generating unit 204, configured to generate an ID according to the globally unique latest timestamp and the machine identifier and using an INCR operation of Redis;

[0078] The returning unit 205 is configured to return the ID to the client.

[0079] Furthermore, the ID includes: a globally unique latest timestamp, a machine identifier, and a single-machine buffer timestamp.

[0080] Furthermore, the acquisition unit 203 includes:

[0081] The timestamp acquisition subunit is used to obtain the latest local timestamp and the latest timestamp stored in the Redis cluster;

[0082] The comparison subunit is used to compare the latest local timestamp with the latest timestamp stored in the Redis cluster to obtain a globally unique latest timestamp.

[0083] Furthermore, the acquisition unit 203 further includes:

[0084] The saving subunit is used to save the globally unique latest timestamp to the Redis cluster to obtain the latest timestamp of the Redis cluster.

[0085] Furthermore, the acquisition unit 203 further includes:

[0086] The update subunit is used to change the latest timestamp of the Redis cluster to a local timestamp.

[0087] Furthermore, the generating unit 204 includes:

[0088] An ID comparison subunit, used to compare the ID with the latest ID recorded in the Redis cluster;

[0089] An ID updating subunit is configured to return the ID to the client and update the ID to the latest ID recorded in the Redis cluster if the ID is greater than the latest ID recorded in the Redis cluster;

[0090] The regeneration subunit is used to regenerate the ID if the ID is less than or equal to the latest ID recorded in the Redis cluster.

[0091] Furthermore, the generating unit 204 further includes:

[0092] The setting subunit is used to set the expiration time of the ID by using the EXPIRE operation of Redis.

[0093] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-mentioned devices and units can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here.

[0094] The present invention also provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed, the method provided in the above embodiment can be implemented. The storage medium may include: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes.

[0095] The present invention also provides a computer device, which may include a memory and a processor, wherein a computer program is stored in the memory, and when the processor calls the computer program in the memory, the method provided in the above embodiment can be implemented. Of course, the computer device may also include various network interfaces, power supplies and other components.

[0096] The various embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same and similar parts between the various embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description. It should be pointed out that for ordinary technicians in this technical field, without departing from the principle of the present invention, several improvements and modifications can be made to the present invention, and these improvements and modifications also fall within the scope of protection of the claims of the present invention.

[0097] It should also be noted that, in this specification, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprises", or any other variations thereof are intended to cover non-exclusive.

[0098] Inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of more restrictions, the elements defined by the sentence "including a..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

Claims

1. A distributed ID generation method, characterized in that: include: Deploy multiple Redis clusters in different computer rooms; When a client applies for an ID, the application is sent to a load balancer and the load balancer forwards the application to multiple Redis clusters; Obtain the globally unique latest timestamp and machine ID through the Redis cluster; Generate an ID based on the globally unique latest timestamp and machine ID using the INCR operation of Redis; The ID is returned to the client.

2. The distributed ID generation method according to claim 1, characterized in that: The ID includes: a globally unique latest timestamp, a machine identifier, and a single-machine buffer timestamp.

3. The distributed ID generation method according to claim 1, characterized in that: Obtaining the globally unique latest timestamp includes: Get the latest local timestamp and the latest timestamp stored in the Redis cluster; The local latest timestamp is compared with the latest timestamp stored in the Redis cluster, and the latest timestamp after the comparison is used as the globally unique latest timestamp.

4. The distributed ID generation method according to claim 3, characterized in that: The method of obtaining the globally unique latest timestamp includes: saving the globally unique latest timestamp to a Redis cluster, and obtaining the latest timestamp of the Redis cluster.

5. The distributed ID generation method according to claim 4, characterized in that: The obtaining of the latest timestamp of the Redis cluster includes: changing the latest timestamp of the Redis cluster to a local timestamp.

6. The distributed ID generation method according to claim 1, characterized in that: The step of generating an ID according to the globally unique latest timestamp and the machine identifier and using the INCR operation of Redis includes: Compare the ID with the latest ID recorded in the Redis cluster; If the ID is greater than the latest ID recorded in the Redis cluster, the ID is returned to the client and the ID is updated to the latest ID recorded in the Redis cluster; If the ID is less than or equal to the latest ID recorded in the Redis cluster, a new ID is generated.

7. The distributed ID generation method according to claim 1, characterized in that: After generating the ID according to the globally unique latest timestamp and the machine identifier using the INCR operation of Redis, the method further includes: setting the expiration time of the ID by using the EXPIRE operation of Redis.

8. A distributed ID generation device, characterized in that: include: Deployment unit, used to deploy multiple Redis clusters in different computer rooms; A sending unit, configured to send the application to a load balancer when the client applies for an ID, and forward the application to multiple Redis clusters through the load balancer; An acquisition unit, used to acquire the latest globally unique timestamp and machine identifier through the Redis cluster; A generating unit, configured to generate an ID according to the globally unique latest timestamp and the machine identifier and using an INCR operation of Redis; The returning unit is used to return the ID to the client.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the distributed ID generation method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which, when executed by a processor, causes the processor to perform the distributed ID generation method according to any one of claims 1 to 7.

Images