Lightweight technology implementation scheme for generating distributed unique ID based on snowflake algorithm and Redis

Abstract

The invention relates to a lightweight technical implementation scheme for generating a distributed unique ID based on a snowflake algorithm and Redis, in particular to the field of online car-hailing, comprising the steps of initializing snowflake algorithm parameters as a basis for generating the unique ID, ensuring that the ID generated by calling for multiple times in the same timestamp has uniqueness, selecting a cloud to deploy a Redis instance, and starting RDB and AOF at the same time. The method comprises the following steps: constructing an ID generator class by utilizing a Spring Boot framework, adding Redis related dependency, defining a starting timestamp and judging an ID range through row, supporting high-concurrency requests and avoiding performance bottlenecks, adding a thread group and setting a thread number to simulate a concurrency user number, adding an HTTP (Hyper Text Transport Protocol) request, recording performance indexes of Redis by utilizing a redis-cli monitoring command, and checking response time of Redis and throughput of a system. IDs are stored by utilizing a BIGTNT type, slow queries exceeding execution time are checked through a SLOWLOG command, online car-hailing marketing business data which are not used any more are cleaned by utilizing redis-cli, a memory is released, efficient resource use is kept, and memory leakage or overuse is prevented.

Description

Technical Field

[0001] The present invention relates to the field of online car-hailing, and more specifically, to a lightweight technical implementation solution for generating distributed unique IDs based on a snowflake algorithm and Redis. Background Art

[0002] In the online car-hailing marketing business, with the rapid growth of data volume and the diversification of data structure, the challenges of managing and maintaining data are also increasing. In order to ensure the consistency and integrity of data, especially in the environment of sharding, a globally unique ID generator is particularly important. As the business expands, online car-hailing platforms need to process data from different sources, such as user information, order details, vehicle status, etc. Therefore, the globally unique ID generator has become a core component in modern data architecture.

[0003] The current unique ID generation technology usually needs to be deployed independently, which means that each service or microservice often requires an independent ID generation instance to ensure the uniqueness and order of the ID. In the case of low concurrency scenarios, this separate deployment strategy will lead to a huge waste of server resources. For a long time, the ID generator may be in a low-load state, but it still needs to occupy a certain amount of memory and computing resources, resulting in a decrease in overall resource utilization. The current ID generation technology often has relatively independent functions, forming a certain degree of information isolation in actual business operations. It is difficult to directly understand the specific business entity or activity represented by the ID, resulting in difficulty in quickly associating the ID with specific user activities or business operations during subsequent data analysis and troubleshooting, thereby increasing the complexity of data tracking and monitoring. If the existing ID generator fails to provide an intuitive description of business attributes, it is impossible to quickly obtain clear business insights. Summary of the invention

[0004] In view of the technical problems existing in the prior art, the present invention provides a lightweight technical implementation solution for generating a distributed unique ID based on a snowflake algorithm and Redis to solve the problems raised in the above background technology.

[0005] The technical solution of the present invention to solve the above technical problems is as follows: a lightweight technical implementation solution for generating a distributed unique ID based on a snowflake algorithm and Redis, comprising the following steps:

[0006] S101. Initialize the snowflake algorithm parameters as the basis for generating a unique ID, set the data center ID and pair it with the machine ID, select the cloud-based Redis instance, and enable RDB and AOF at the same time;

[0007] S102. Use the Spring Boot framework to build an ID generator class, add Redis related dependencies, define the start timestamp, and determine the ID range through throw;

[0008] S103. Create a new test plan in JMeter, add a thread group and set the number of threads to simulate the number of concurrent users, add HTTP requests, use the redis-cli monitoring command to record Redis performance indicators, and check Redis response time and system throughput;

[0009] S104. Integrate ID generation into order management, user management and vehicle management of online car-hailing marketing business, use BIGTNT type to store ID, use SLOWLOG command to check slow queries that exceed execution time, and use redis-cli to clean up unused online car-hailing marketing business data and release memory;

[0010] In a preferred embodiment, in S101, the snowflake algorithm parameters are initialized as the basis for generating a unique ID, the fixed decimal number is set to 15 bits, converted to 50 bits in binary, the 50-bit binary number is cut to include a 31-bit second-level timestamp, a 9-bit business code, a 2-bit machine code and an 8-bit random code, the data center ID is set to represent different data centers, the machine ID is set to pair with the data center ID, and an auto-increment calculator is added to generate a serial number.

[0011] Furthermore, we chose to deploy Redis instances in the cloud, use the Sentinel mode to enhance high availability and load balancing capabilities, adjust the optimal number of connections, memory limits, and persistence configurations according to the amount of read and write requests, enable RDB and AOF at the same time, use RDB persistence to take snapshots of the online car-hailing marketing business data at specified time intervals and save them to disk, configure RDB through the save rule option, keep the snapshot every 900 seconds if there is at least one write operation, save the snapshot every 300 seconds if there are at least 10 write operations, and save the snapshot every 60 seconds if there is at least one write operation. Use AOF persistence to append each write operation to the log file of the online car-hailing marketing business data and serialize the records for replay, use the appendfsync always command to synchronize immediately after each write operation, and use the appendfsync everysec command to synchronize once per second, and compress the online car-hailing marketing business data through the AOF rewrite mechanism.

[0012] In a preferred embodiment, in S102, the Spring Boot framework is used to build an ID generator class, the Spring Data Redis framework is used to add Redis related dependencies, the starting timestamp is defined, the ID range is judged by throw, and when the current timestamp is less than the timestamp of the last ID generation, it means that the system clock is rolled back and an exception is thrown. The longredisSequence command is used to obtain the Redis auto-increment sequence number, and the ID is generated by the return command. The synchronized keyword is added to the generateId method to ensure the security of the multi-threaded environment, the increment method is used to obtain the sequence number, the IdGenerator class is injected and the generateId method is called.

[0013] In a preferred embodiment, in S103, a new test plan is created in JMeter, a thread group is added and the number of threads is set to simulate the number of concurrent users, an HTTP request is added, the URL is set to the API address for generating the ID, a listener is added for aggregate reporting, the result tree is viewed, the redis-cli monitoring command is used to record the performance indicators of Redis, the report generated by JMeter is viewed, the response time of Redis and the throughput of the system are checked, and when the size of the ID set is the same as the number of requests, it means that the ID is unique, otherwise, there are duplicate IDs.

[0014] In a preferred embodiment, in S104, ID generation is integrated into the order management, user management and vehicle management of the online car-hailing marketing business. The generateId() method is called when creating a new order, registering a user and adding a new vehicle. The encapsulation service provides a unified ID generation interface, and the ID is stored using the BIGTNT type. The ID generation is integrated into the main branch of the online car-hailing marketing business through the version control tool. The generation performance indicators are defined as the response time, success rate, failure rate and request volume of the generated ID. The performance indicators of Redis are monitored, and monitoring is added to the ID generation through the Prometheus monitoring tool. The performance indicators related to ID generation are sent regularly, the monitoring documents are updated and the usage methods and performance indicators of ID generation are recorded. The Redis INFO command is used to monitor Redis performance indicators, and the slow queries that exceed the execution time are viewed through the SLOWLOG command. The redis-cli is used to clean up the online car-hailing marketing business data that is no longer used and release memory.

[0015] The beneficial effects of the present invention are as follows: the uniqueness of the ID generated by the snowflake algorithm is also provided with a time sorting feature, which is applicable to data tables and data structures in distributed systems, the high availability and load balancing capability of Redis are improved through the Redis Sentinel mode, and it is ensured that the system can automatically switch to the standby node when the main node fails, and the service is ensured to run continuously, and a globally unique ID is generated by combining the timestamp, the work machine ID and the serial number, so as to avoid duplication and support high-concurrency scenarios, and the number of digits can be limited for the ID, so as to reduce the use of server resources and reduce costs, and avoid deploying a separate server for ID generation, and combine the RDB and AOF persistence mechanisms of Redis to ensure that data can be recovered after the system crashes. RDB saves data through regular snapshots, and AOF records all write operations so as to be reapplied during recovery, and using Redis as an intermediary for ID generation can effectively reduce resource consumption, and through its efficient memory storage and operation capabilities, a unique ID can be quickly generated, and high-concurrency access can be supported, and combined with the cluster mode of Redis, the load can be shared on multiple nodes, and the scalability and stability of the system can be improved. At the same time, through the persistence function of Redis, it is ensured that the generated ID data will not be lost. Specific business information about the timestamp is embedded in the ID to reflect the generation time of the generated ID, which not only improves the business relevance of the ID, but also provides valuable information in the subsequent data analysis and processing process. The ID generation service is integrated into the order, user and vehicle management system of the online car-hailing marketing business, providing a unified ID generation interface. The performance indicators and usage methods of ID generation are recorded through version control and monitoring documents to ensure the stability and maintainability of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 is a flow chart of the method of the present invention; DETAILED DESCRIPTION

[0017] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0018] In the description of this application, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of this application, "plurality" means two or more, unless otherwise clearly and specifically defined.

[0019] In the description of the present application, the term "for example" is used to mean "used as an example, illustration or description". Any embodiment described as "for example" in the present application is not necessarily to be construed as being more preferred or advantageous than other embodiments. The following description is given to enable any technician in the field to implement and use the present invention. In the following description, details are listed for the purpose of explanation. It should be understood that a person of ordinary skill in the art can recognize that the present invention can be implemented without using these specific details. In other examples, well-known structures and processes will not be elaborated in detail to avoid unnecessary details to obscure the description of the present invention. Therefore, the present invention is not intended to be limited to the embodiments shown, but is consistent with the widest scope consistent with the principles and features disclosed in the present application.

[0020] This embodiment provides Figure 1 A lightweight technical implementation scheme for generating a distributed unique ID based on the snowflake algorithm and Redis is shown, which specifically includes the following steps:

[0021] S101. Initialize the snowflake algorithm parameters as the basis for generating a unique ID, set the data center ID and pair it with the machine ID, select the cloud-based Redis instance, and enable RDB and AOF at the same time;

[0022] Furthermore, the snowflake algorithm parameters are initialized as the basis for generating a unique ID, the fixed number of decimal digits is set to 15, converted to 50 binary digits, and the 50-bit binary number is cut to include a 31-bit second-level timestamp, a 9-bit business code, a 2-bit machine code, and an 8-bit random code, to ensure that the generated ID is unique and ordered in the distributed online car-hailing marketing business, which helps to avoid ID conflicts. The data center ID is set to represent different data centers. Services deployed in multiple data centers can ensure that IDs generated in different geographic locations will not conflict. The machine ID is set to pair with the data center ID to identify different servers in the same data center. An auto-increment calculator is added to generate a serial number to ensure that multiple calls to generate IDs within the same timestamp are unique.

[0023] Furthermore, we chose to deploy Redis instances in the cloud, and used the Sentinel mode to enhance high availability and load balancing capabilities to ensure normal operation when a node fails. We adjusted the optimal number of connections, memory limits, and persistence configurations based on the volume of read and write requests to support high concurrent requests and avoid performance bottlenecks, which helps improve performance and stability. At the same time, we enabled RDB and AOF, and combined the RDB and AOF persistence methods to provide reliable backup and rapid recovery capabilities for online car-hailing marketing business data, improve fault tolerance, and use RDB persistence to perform backups of online car-hailing marketing business data at specified time intervals. Take a snapshot and save it to disk. Configure RDB through the save rule option. Every 900 seconds, if there is at least one write operation, keep the snapshot. Every 300 seconds, if there are at least 10 write operations, save the snapshot. Every 60 seconds, if there is at least one write operation, save the snapshot. The process of starting RDB to generate a snapshot is asynchronous and will not block the main process of Redis. The recovery efficiency of the online car-hailing marketing business data is high in a short time. AOF persistence is used to append each write operation to the log file of the online car-hailing marketing business data and serialize the records for replay. The appendfsync always command is used to synchronize immediately after each write operation, and the appendfsync everysec command is used to synchronize once per second. The online car-hailing marketing business data is compressed through the AOF rewrite mechanism to reduce the occupied space.

[0024] S102. Use the Spring Boot framework to build an ID generator class, add Redis related dependencies, define the start timestamp, and determine the ID range through throw;

[0025] Furthermore, the Spring Boot framework is used to build an ID generator class, and the Spring Data Redis framework is used to add Redis related dependencies, define the starting timestamp, and judge the ID range through throw. When the current timestamp is less than the timestamp of the last ID generation, it means that the system clock has rolled back and an exception is thrown, which can effectively prevent ID conflicts caused by clock rollbacks and increase robustness. The long redisSequence command is used to obtain the Redis auto-increment sequence number to avoid ID conflicts or generation errors caused by multi-threaded access, and new IDs can be generated quickly and reliably. The return command is used to generate the ID, and the synchronized keyword is added to the generateId method to ensure the security of the multi-threaded environment. The increment method is used to obtain the sequence number, which is injected into the IdGenerator class and the generateId method is called.

[0026] S103. Create a new test plan in JMeter, add a thread group and set the number of threads to simulate the number of concurrent users, add HTTP requests, use the redis-cli monitoring command to record Redis performance indicators, and check Redis response time and system throughput;

[0027] Furthermore, create a new test plan in JMeter, add a thread group and set the number of threads to simulate the number of concurrent users, add an HTTP request, configure the request method to GET, set the URL to the API address for generating the ID, add a listener for aggregate reporting, view the result tree, use the redis-cl i monitoring command to record Redis performance indicators, view the report generated by JMeter to evaluate whether the system's performance under high concurrency meets the needs of the online car-hailing marketing business, check the response time of Redis and the throughput of the system to ensure that it can work stably under high concurrency, and check the size of the generated ID set. When the size of the ID set is the same as the number of requests, it means that the ID is unique, otherwise, there are duplicate IDs.

[0028] S104. Integrate ID generation into order management, user management and vehicle management of online car-hailing marketing business, use BIGTNT type to store ID, use SLOWLOG command to check slow queries that exceed execution time, and use redis-cli to clean up unused online car-hailing marketing business data and release memory;

[0029] Furthermore, ID generation is integrated into the order management, user management, and vehicle management of the online car-hailing marketing business, making ID generation more unified and consistent, which helps to improve the collaborative work efficiency of the online car-hailing marketing business. The generateId() method is called when creating a new order, registering a user, and adding a new vehicle. The encapsulation service provides a unified ID generation interface, simplifies the calls to ID generation by different management, reduces code duplication and maintenance costs, and uses the BIGTNT type to store IDs to ensure that the ID can handle large amounts of data and avoid storage overflow problems. The ID generation is integrated into the main branch of the online car-hailing marketing business through the version control tool, ensuring version control and management of ID generation, allowing for better version updates and tracking. , define the generation performance indicators as the response time, success rate, failure rate and request volume of the generated ID, which helps to quantify and evaluate the performance of the ID generator to ensure that it meets business needs, monitor the performance indicators of Redis, add monitoring to ID generation through the Prometheus monitoring tool, and regularly send performance indicators related to ID generation, update the monitoring document and record the usage and performance indicators of ID generation, use Redis's INFO command to monitor Redis performance indicators, use the SLOWLOG command to view slow queries that exceed the execution time, and use redis-cli to clean up the unused online car-hailing marketing business data and release memory to keep resources efficient and prevent memory leaks or overuse.

[0030] It should be noted that in the above embodiments, the description of each embodiment has its own emphasis, and for parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0031] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0032] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded computer, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0033] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0034] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0035] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0036] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A lightweight technical solution for generating distributed unique IDs based on the snowflake algorithm and Redis, characterized in that: The specific steps include: S101. Initialize the snowflake algorithm parameters as the basis for generating a unique ID, set the data center ID and pair it with the machine ID, select the cloud-based Redis instance, and enable RDB and AOF at the same time; S102. Use the Spring Boot framework to build an ID generator class, add Redis related dependencies, define the start timestamp, and determine the ID range through throw; S103. Create a new test plan in JMeter, add a thread group and set the number of threads to simulate the number of concurrent users, add HTTP requests, use the redis-cli monitoring command to record Redis performance indicators, and check Redis response time and system throughput; S104. Integrate ID generation into the order management, user management, and vehicle management of the online car-hailing marketing business, use the BIGTNT type to store IDs, use the SLOWLOG command to view slow queries that exceed the execution time, and use redis-cli to clean up unused online car-hailing marketing business data and release memory.

2. According to claim 1, a lightweight technical implementation scheme for generating a distributed unique ID based on a snowflake algorithm and Redis, characterized in that: In S101, initialize the snowflake algorithm parameters as the basis for generating a unique ID, set the decimal fixed number to 15 bits, convert it to 50 bits in binary, cut the 50-bit binary number to include a 31-bit second-level timestamp, a 9-bit business code, a 2-bit machine code and an 8-bit random code, set the data center ID to represent different data centers, set the machine ID to pair with the data center ID, and add an auto-increment calculator to generate a serial number.

3. According to claim 1, a lightweight technical implementation scheme for generating a distributed unique ID based on a snowflake algorithm and Redis, characterized in that: In S101, a Redis instance is selected to be deployed in the cloud, and the Sentinel mode is used to enhance high availability and load balancing capabilities. The optimal number of connections, memory limits, and persistence configurations are adjusted according to the amount of read and write requests. RDB and AOF are enabled at the same time. RDB persistence is used to snapshot the online car-hailing marketing business data within a specified time interval and save it to disk. RDB is configured through the save rule option. AOF persistence is used to append each write operation to the log file of the online car-hailing marketing business data and serialize the records for replay. The appendfsync always command is used to synchronize immediately after each write operation, and the appendfsync everysec command is used to synchronize once per second. The online car-hailing marketing business data is compressed through the AOF rewrite mechanism.

4. According to claim 3, a lightweight technical implementation scheme for generating a distributed unique ID based on a snowflake algorithm and Redis, characterized in that: The save rule option implements configuration of RDB specifically as follows: every 900 seconds, if there is at least one write operation, keep a snapshot; every 300 seconds, if there are at least 10 write operations, save a snapshot; every 60 seconds, if there is at least one write operation, save a snapshot.

5. According to claim 1, a lightweight technical implementation scheme for generating a distributed unique ID based on a snowflake algorithm and Redis, characterized in that: In S102, the Spring Boot framework is used to build an ID generator class, the SpringData Redis framework is used to add Redis related dependencies, the starting timestamp is defined, the ID range is judged by throw, and when the current timestamp is less than the timestamp of the last ID generation, it means that the system clock is rolled back and an exception is thrown. The longredisSequence command is used to obtain the Redis auto-increment sequence number, and the return command is used to generate the ID. The synchronized keyword is added to the generateId method to ensure the security of the multi-threaded environment, the increment method is used to obtain the sequence number, the IdGenerator class is injected, and the generateId method is called.

6. According to claim 1, a lightweight technical implementation scheme for generating a distributed unique ID based on a snowflake algorithm and Redis, characterized in that: In S103, a new test plan is created in JMeter, a thread group is added and the number of threads is set to simulate the number of concurrent users, an HTTP request is added, the URL is set to the API address for generating the ID, a listener is added for aggregate reporting, the result tree is viewed, the redis-cli monitoring command is used to record the performance indicators of Redis, the report generated by JMeter is viewed, and the response time of Redis and the throughput of the system are checked.

7. A lightweight technical implementation scheme for generating a distributed unique ID based on a snowflake algorithm and Redis according to claim 6, characterized in that: When the size of the ID set is the same as the number of requests, the IDs are unique, otherwise, there are duplicate IDs.

8. According to claim 1, a lightweight technical implementation scheme for generating a distributed unique ID based on a snowflake algorithm and Redis, characterized in that: In the S104, ID generation is integrated into the order management, user management and vehicle management of the online car-hailing marketing business. The generateId() method is called when creating a new order, registering a user and adding a new vehicle. The encapsulation service provides a unified ID generation interface, and the ID is stored using the BIGTNT type. The ID generation is integrated into the main branch of the online car-hailing marketing business through the version control tool. The generation performance indicators are defined as the response time, success rate, failure rate and request volume of the generated ID. The performance indicators of Redis are monitored, and monitoring is added to the ID generation through the Prometheus monitoring tool. The performance indicators related to ID generation are sent regularly, the monitoring documents are updated and the usage methods and performance indicators of ID generation are recorded. The Redis INFO command is used to monitor Redis performance indicators, and the slow queries that exceed the execution time are viewed through the SLOWLOG command. The redis-cli is used to clean up the online car-hailing marketing business data that is no longer used and release memory.

Images