A high-concurrency short link generation and analysis method

Abstract

The application discloses a high-concurrency short link generation and analysis method, which combines the advantages of short link generation technology and uses the Redis INCRBY command and a random number code to generate a sequence number with an increasing trend. The sequence number is mapped by using a 62-base hash algorithm to generate a short link. This method not only realizes the uniqueness of the short link, but also has multiple advantages such as high performance, high security and low storage space. The method can effectively manage the short link, improve the performance and security of the system, and save storage space.

Description

Technical Field

[0001] This invention relates to the marketing field's need for converting long links to short links. Specifically, it relates to a method for generating and parsing short links in a high-concurrency environment. This method enables high-concurrency generation and access to short links in a distributed architecture. Background Technology

[0002] Currently, there are three common short link generation techniques: hash algorithm-based, auto-incrementing sequence-based, and random code-based.

[0003] 1) Based on auto-incrementing sequences: Short links are generated by mapping long links to auto-incrementing ID sequences. This method can guarantee the uniqueness of links, but it has security risks and the risk of being enumerated.

[0004] 2) Hash Algorithm-Based: This method uses a hash algorithm to convert long links into short strings or numbers, thus generating shorter links. This method is simple and fast. However, due to the nature of hash algorithms, different long links may generate the same short link, leading to link collisions and potentially causing duplicate or overwritten links.

[0005] 3) Based on random codes: A short link is generated using a random code, such as a UUID or a random string. To ensure the uniqueness and security of the link, the random code needs to be of sufficient length and complexity, which results in problems such as long length, large storage space consumption, and a relatively time-consuming generation process.

[0006] As mentioned above, these short link generation techniques all have some drawbacks. Short links based on auto-incrementing sequences have security vulnerabilities and the risk of being enumerated; short links based on hash algorithms may experience link collisions, leading to data overwriting; short links based on random codes, while offering higher security, incur significant overhead in generation and storage. Summary of the Invention

[0007] This invention provides a high-concurrency short link generation and parsing method to solve the aforementioned problems. This method combines the advantages of the aforementioned short link generation techniques and uses the Redis INCRBY command and random numeric codes to generate trending increasing sequence numbers. Short links are generated by mapping these sequence numbers using a base-62 hash algorithm. This method not only ensures the uniqueness of short links but also possesses advantages such as high performance, high security, and low storage space. Using this method can effectively manage short links, improve system performance and security, and save storage space.

[0008] Definitions:

[0009] 1. MySQL: An open-source relational database management system that supports multiple operating systems and programming languages, and features high efficiency, stability, and security.

[0010] 2. Redis: A high-performance in-memory data storage system that supports various data structures and a rich set of operation commands. It can be used as a database, cache, and message broker.

[0011] 3. RabbitMQ: An open-source message queue middleware developed using the Erlang language and implementing the Advanced Message Queuing Protocol (AMQP). It has advantages such as being free, open-source, secure, high-performance, and persistent.

[0012] 4. High Availability: This refers to improving the availability of systems and applications by minimizing downtime caused by routine maintenance operations (planned) and sudden system crashes (unplanned).

[0013] 5. Bloom filter: A data structure based on bit arrays and hash functions, used to quickly determine whether an element exists in a set. It has the characteristics of high efficiency and low storage space, but it has a certain false positive rate.

[0014] 6. Cuckoo Filter: A highly efficient probabilistic data structure used to quickly determine whether an element exists in a set. It features high efficiency and a low false positive rate. Compared to the Bloom filter, it supports deletion operations and has an even lower false positive rate.

[0015] 7. Hash: A hash function is a function that compresses a message of arbitrary length into a message digest of a fixed length. It is often used in scenarios such as data encryption and unique identification.

[0016] 8. Long links: Long links are links with longer URLs that contain multiple parameters. They are typically used to convey large amounts of information or to implement website functions, but due to their length, they are not convenient for users to remember or share.

[0017] 9. Shortened Links: Shortened links are links with shorter URLs, usually consisting of letters and numbers. They are easier for users to share and remember, and also help save storage space and improve website loading speed.

[0018] 10. Base-62: Base-62 is a way of encoding numbers and letters in a mixed manner, with a range of [a-zA-Z0-9]. It can convert long strings into short strings.

[0019] 11. Redis INCRBY: Redis INCRBY is an atomic operation command provided by Redis for performing addition operations on specified keys. It supports integers and floating-point numbers and allows specifying the amount to be added.

[0020] To achieve the above technical objectives and effects, this invention provides a high-concurrency short link generation and parsing method, which can generate a massive number of short links in real time and persistently store them. The specific steps are as follows:

[0021] Step 1: Set up a highly available RabbitMQ as a distributed message queue;

[0022] Step 2: Build a highly available MySQL cluster as a data storage service;

[0023] Step 3: Build a highly available Redis Stack service cluster as a short-connection cache and an increasing sequence generator;

[0024] Step 4: Call the short link generation interface, pass in the long link and business number, and complete the signature verification;

[0025] Step 5: Generate random numbers and use Redis's INCRBY command to obtain an increasing sequence based on the generated random numbers;

[0026] Step 6: Convert the incrementing sequence value generated in Step 5 into a base-62 short link code, and use the Cuckoo Filter to determine if the short link code exists. If the short link code already exists, return to Step 5 to regenerate the incrementing sequence until a new short link code that does not exist in the Cuckoo Filter is generated; if the short link code does not exist, proceed to Step 7.

[0027] Step 7: Save the short link code generated in Step 6 to the Cuckoo Filter for subsequent deduplication.

[0028] Step 8: Map the short link code generated in Step 6 to the long link passed in Step 4, store it in Redis using a hash structure, and set an expiration time.

[0029] Step 9: Send the short link code and long link mapping information from Step 8 to the RabbitMQ message queue. In the RabbitMQ message queue, the short link code and long link can be encapsulated into a message for subsequent asynchronous processing.

[0030] Step 10: Subscribe to RabbitMQ messages, parse the messages to extract the short link code and long link, and store them in MySQL. Also, store the long and short link mapping information in MySQL.

[0031] Furthermore, the specific steps for high-concurrency short link access are as follows:

[0032] Step 1: When accessing a long link using a shortened link, the user can access it by entering the shortened link address in the browser, clicking on a page containing the shortened link address, or via SMS.

[0033] Step 2: When using a short link to access a long link, the server needs to extract the short link code from the short link. A whitelist + regular expression matching mechanism is used. If the short link address is not in the path whitelist and meets the requirements of the regular expression, the short link code will be extracted.

[0034] Step 3: Use the Cuckoo Filter to determine if the short link code exists. If the query result shows that the short link exists, it means that the short link is valid, and proceed to Step 4. If the query result shows that the short link does not exist, it means that the short link is invalid, and the server will return the corresponding error message.

[0035] Step 4: Determine if the short link code exists by querying the Hash structure in Redis; if it exists, proceed to step 6; if it does not exist, proceed to step 5.

[0036] Step 5: Query the MySQL database to determine if the short link code exists; if it exists, proceed to step 6; if it does not exist, the short link is invalid, and the server will return the corresponding error message.

[0037] Step 6: Extract the long-lived connection and redirect to it; store the user access logs asynchronously via RabbitMQ;

[0038] Step 7: If the link does not exist, redirect the user to the system 404 page and inform the user that the link has expired or does not exist.

[0039] Furthermore, the specific steps for deleting short links in high-concurrency scenarios are as follows:

[0040] Step (1) When using the delete short link interface, pass in the short link and related verification information;

[0041] Step (2) The server needs to extract the short link code from the short link. It uses a whitelist + regular expression matching mechanism. If the short link address is not in the path whitelist and meets the requirements of the regular expression, the short link code will be extracted.

[0042] Step (3) Use the Cuckoo Filter to determine if the short link code exists; if the query result is that it exists, it means that the short link is valid, and proceed to step (4); if the query result is that it does not exist, it means that the short link is invalid, and the server will return the corresponding error message.

[0043] Step (4) Delete the short link code in the cuckoo filter;

[0044] Step (5) Delete the short link code in Redis;

[0045] Step (6) Change the status of the short link key in MySQL to "deleted";

[0046] Step (7) will indicate that the deletion was successful for existing links and will provide a corresponding prompt for non-existent links.

[0047] Furthermore, the Cuckoo Filter allows for the real-time deletion or disabling of short links, enabling real-time control over them. Compared to the Bloom Filter, it has a lower false positive rate and supports deletion operations.

[0048] Furthermore, by employing RabbitMQ message queues, the generation and persistence of short links can be decoupled, thereby achieving a loosely coupled system architecture and improving system concurrency. Through message queues, functions such as short link generation, access, and deletion can be broken down into independent services, each running on different processes or servers, thus improving system scalability and reliability.

[0049] Furthermore, in step eight, the short link code generated in step six is ​​mapped to the long link passed in step four. When storing the mapping in Redis using a hash structure, the short link code is used as the key and the long link is used as the value.

[0050] Compared with the prior art, the beneficial effects of the present invention are:

[0051] Firstly, uniqueness: by generating trending increasing sequence numbers and combining them with a cuckoo filter, the short link code is unique, avoiding the repeated generation of the same short link.

[0052] Secondly, regarding security, by generating random numbers and combining them with disordered base-62 characters, the risk of short links being enumerated is greatly reduced, ensuring the security of short links. When accessing short links, the existence of the link is determined by the Cuckoo Filter, which can avoid subsequent caching and database queries, thereby reducing invalid access and avoiding the risk of cache penetration.

[0053] Thirdly, it boasts high performance. Compared to traditional database-generated serial numbers, this invention uses the in-memory database Redis for generation, resulting in faster generation speeds and significantly improved efficiency and performance in short link generation. The short links are stored in a distributed cache, providing high concurrency capabilities, with a measured single-machine throughput of up to 10K / s.

[0054] Fourthly, it has low storage space. Compared with generating short links based on random code methods, this invention uses a base-62 hash algorithm to map the trending increasing sequence number. Only 6 characters are needed to represent billions of links, saving storage space.

[0055] Fifthly, scalability: This invention employs asynchronous processing technology, placing data storage requests into a message queue for asynchronous processing by multiple nodes, thereby achieving parallel processing of requests and improving system scalability. Furthermore, the short link generation and access services of this invention are stateless; by adding new nodes, horizontal scaling of the system can be achieved, thus improving system scalability. Attached Figure Description

[0056] Figure 1 This is a detailed flowchart of the short link generation process of the high-concurrency short link generation and parsing method of the present invention.

[0057] Figure 2 This is a detailed flowchart of the short link access method of the high-concurrency short link generation and parsing method of the present invention.

[0058] Figure 3 This is a detailed flowchart of the short link deletion process in the high-concurrency short link generation and parsing method of the present invention. Detailed Implementation

[0059] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments, so that those skilled in the art can better understand the present invention and implement it. However, the embodiments are not intended to limit the present invention.

[0060] This invention provides a detailed process design for generating high-concurrency short links. The specific steps are as follows, see details. Figure 1 .

[0061] Step 1: Call the short link generation interface, passing in the long link, business number, and signature verification information.

[0062] Step 2: Generate random numbers, with a value range of 1-10. The range of random numbers can be adjusted according to the actual situation.

[0063] Step 3: In order to obtain an increasing sequence and prevent short links from being too short, this invention uses the Redis INCRBY command in combination with the random number generated in step 2, and sets an auto-incrementing starting value, for example, 62 to the power of 4 equals 14,776,336.

[0064] Step 4: Convert the sequence value to a base-62 short link code. Converting the increasing sequence value generated in Step 3 to a base-62 short link code converts a long link into a short link; the length of the short link code is the length of the short link. During the conversion, a base-62 encoding table can be used to convert decimal values ​​to base-62 values ​​and map them to corresponding characters according to the encoding table, thus generating the short link code.

[0065] Step 5: Use the Cuckoo Filter to determine if the short link code exists. If the short link code already exists, return to Step 2 to regenerate the increasing sequence until a new short link code that does not exist in the Cuckoo Filter is generated; if the short link code does not exist, proceed to Step 6. The Cuckoo Filter is an efficient deduplication algorithm that can quickly determine whether an element exists in a set, and has advantages such as small memory footprint and low false positive rate.

[0066] Step 6: Save the generated short link code to the Cuckoo Filter for subsequent deduplication.

[0067] Step 7: Map the generated short link codes to long links and store them in Redis using a hash structure. When storing, the short link code can be used as the key and the long link as the value in Redis, making it easy to query the corresponding long link based on the short link code. To prevent excessive memory usage due to too much data in Redis, an expiration time can be set, allowing Redis to automatically clean up expired data. A reasonable expiration time can be set according to business needs to ensure a balance between system performance and storage space.

[0068] Step 8: Map the generated short link codes to long links and send them to RabbitMQ. In the message queue, the short link codes and long links can be encapsulated into a single message for asynchronous processing later. Asynchronous processing separates the generation of short links from the storage of long links, improving system performance and scalability. Simultaneously, the load balancing and failover mechanisms of the message queue enhance system availability and security.

[0069] Step 9: Consume RabbitMQ messages and store the long and short link mapping information in MySQL. When consuming messages, the short link code and long link can be extracted from the message and stored in MySQL. MySQL is a commonly used relational database with advantages such as clear data structure and high data consistency, making it suitable for storing structured data. By storing the long and short link codes in MySQL, the mapping relationship between long and short links can be established, facilitating subsequent queries, access, and statistics.

[0070] This invention provides a detailed process design for high-concurrency short-connection access, with specific steps as follows (see details). Figure 2 .

[0071] Step 1: When accessing a long link using a shortened link, users can access it by entering the shortened link address in their browser, clicking on a page containing the shortened link address, or via SMS.

[0072] Step Two: When accessing a long link using a shortened link, the server needs to extract the shortened link code from the shortened link. To ensure the correctness and security of the extracted shortened link code, this invention employs a whitelist + regular expression matching mechanism. Specifically, if the shortened link address is not in the path whitelist and meets the requirements of the regular expression, the shortened link code will be extracted. This method prevents unauthorized access to shortened links while ensuring the uniqueness and correctness of the shortened links. In practical applications, reasonable whitelists and regular expressions can be set according to business needs to improve system security and scalability.

[0073] Step 3: Use a cuckoo filter to determine if the short link code exists. In this invention, to improve system performance and security, a cuckoo filter is used to determine the existence of the short link code. If the query result shows that the short link exists, it means that the short link is valid, and proceed to step 4; if the query result shows that the short link does not exist, it means that the short link is invalid, and the server will return the corresponding error message.

[0074] Step 4: Determine if the short link code exists by querying the Hash structure in Redis. Using the RedisHash structure allows for a quick and accurate determination of whether the short link exists. If it exists, proceed to Step 6; otherwise, proceed to Step 5.

[0075] Step 5: Query the MySQL database to determine if the shortened link code exists. If it exists, proceed to Step 6; otherwise, the shortened link is invalid, and the server will return an appropriate error message.

[0076] Step Six: Extract the long-lived connection and redirect to it. Store user access logs asynchronously via RabbitMQ for statistical analysis of short-connection usage.

[0077] Step 7: If the link does not exist, redirect the user to the system 404 page, indicating that the link has expired or does not exist.

[0078] This invention provides a detailed process design for high-concurrency short link deletion, with specific steps as follows (see details). Figure 2 .

[0079] Step 1: When using the delete short link API, pass in the short link and related verification information.

[0080] Step 2: The server needs to extract the short link code from the short links. To ensure the correctness and security of the extracted short link code, this invention employs a whitelist + regular expression matching mechanism. Specifically, if the short link address is not in the path whitelist and meets the requirements of the regular expression, the short link code will be extracted. This method prevents unauthorized access to short links while ensuring the uniqueness and correctness of the short links. In practical applications, reasonable whitelists and regular expressions can be set according to business needs to improve system security and scalability.

[0081] Step 3: Use a cuckoo filter to determine if the short link code exists. In this invention, to improve system performance and security, a cuckoo filter is used to determine the existence of the short link code. If the query result shows that the short link exists, it means that the short link is valid, and proceed to step 4; if the query result shows that the short link does not exist, it means that the short link is invalid, and the server will return the corresponding error message.

[0082] Step 4: Delete short link codes from the Cuckoo Filter. The Cuckoo Filter is the entry point for querying and generating short links, and its accuracy is crucial for the normal operation of the entire system. Therefore, before performing any other operations, it is necessary to delete short link codes from the Cuckoo Filter first.

[0083] Step 5: Delete short connection keys from Redis. Timely deletion of unused short connection keys from Redis ensures data accuracy and reduces storage pressure.

[0084] Step 6: Change the status of the short link code in MySQL to "deleted". To avoid affecting the statistics and analysis of short link data, the short link code in the MySQL database will not be physically deleted, but its status will only be marked as "deleted".

[0085] Step 7: For existing links, a message will be displayed indicating successful deletion; for non-existent links, a corresponding message will be displayed.

[0086] The above are merely preferred embodiments of the present invention and do not limit the patent scope of the present invention. Any equivalent structural or procedural transformations made based on the content of the present invention's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. A method for generating and parsing high-concurrency short links, characterized in that: The method can generate a massive number of short links in real time and store them persistently. The specific steps are as follows: Step 1: Set up a highly available RabbitMQ as a distributed message queue; Step 2: Build a highly available MySQL cluster as a data storage service; Step 3: Build a highly available Redis Stack service cluster as a short-connection cache and an increasing sequence generator; Step 4: Call the short link generation interface, pass in the long link and business number, and complete the signature verification; Step 5: Generate random numbers and use Redis's INCRBY command to obtain an increasing sequence based on the generated random numbers; Step 6: Convert the incrementing sequence value generated in Step 5 into a base-62 short link code, and use the Cuckoo Filter to determine if the short link code exists. If the short link code already exists, return to Step 5 to regenerate the incrementing sequence until a new short link code that does not exist in the Cuckoo Filter is generated; if the short link code does not exist, proceed to Step 7. Step 7: Save the short link code generated in Step 6 to the Cuckoo Filter for subsequent deduplication. Step 8: Map the short link code generated in Step 6 to the long link passed in Step 4, store it in Redis using a hash structure, and set an expiration time. Step 9: Send the short link code and long link mapping information from Step 8 to the RabbitMQ message queue. In the RabbitMQ message queue, the short link code and long link can be encapsulated into a message for subsequent asynchronous processing. Step 10: Subscribe to RabbitMQ messages, parse the messages to extract the short link code and long link, and store them in MySQL. Also, store the long and short link mapping information in MySQL.

2. The high-concurrency short link generation and parsing method according to claim 1, characterized in that: The specific steps for high-concurrency short link access are as follows: Step 1: When accessing a long link using a shortened link, the user can access it by entering the shortened link address in the browser, clicking on a page containing the shortened link address, or via SMS. Step 2: When using a short link to access a long link, the server needs to extract the short link code from the short link. A whitelist + regular expression matching mechanism is used. If the short link address is not in the path whitelist and meets the requirements of the regular expression, the short link code will be extracted. Step 3: Use the Cuckoo Filter to determine if the short link code exists; if the query result shows that it exists, it means that the short link is valid, and proceed to Step 4. If the query result is not found, it means that the short link is invalid, and the server will return the corresponding error message; Step 4: Determine if the short link code exists by querying the Hash structure in Redis; if it exists, proceed to step 6; if it does not exist, proceed to step 5. Step 5: Query the MySQL database to determine if the short link code exists; if it exists, proceed to step 6; if it does not exist, the short link is invalid, and the server will return the corresponding error message. Step 6: Extract the long-lived connection and redirect to it; store the user access logs asynchronously via RabbitMQ; Step 7: If the link does not exist, redirect the user to the system 404 page and inform the user that the link has expired or does not exist.

3. The high-concurrency short link generation and parsing method according to claim 1, characterized in that: The specific steps for deleting short links during high concurrency are as follows: Step (1) When using the delete short link interface, pass in the short link and related verification information; Step (2) The server needs to extract the short link code from the short link. It uses a whitelist + regular expression matching mechanism. If the short link address is not in the path whitelist and meets the requirements of the regular expression, the short link code will be extracted. Step (3) Use the Cuckoo Filter to determine if the short link code exists; if the query result is that it exists, it means that the short link is valid, and proceed to step (4); if the query result is that it does not exist, it means that the short link is invalid, and the server will return the corresponding error message. Step (4) Delete the short link code in the cuckoo filter; Step (5) Delete the short link code in Redis; Step (6) Change the status of the short link key in MySQL to "deleted"; Step (7) will indicate that the deletion was successful for existing links and will provide a corresponding prompt for non-existent links.

4. The high-concurrency short link generation and parsing method according to claim 1, characterized in that: The Cuckoo Filter allows for the real-time deletion or disabling of short links, enabling real-time control over them.

5. The high-concurrency short link generation and parsing method according to claim 1, characterized in that: By using RabbitMQ message queues, the generation of short connections and persistent operations can be decoupled, thereby improving the system's concurrency capabilities.

6. The high-concurrency short link generation and parsing method according to claim 1, characterized in that: In step eight, the short link code generated in step six is ​​mapped to the long link passed in step four. When storing the mapping in Redis using a hash structure, the short link code is used as the key and the long link is used as the value.

Images