Redis-based hundred million-level order number duplicate removal method, electronic equipment and storage medium
By dynamically selecting Set and BitMap structures in Redis for storage and deduplication verification of logistics order numbers, the problems of slow single number deduplication verification and high memory usage in the existing technology are solved, and an efficient and low memory usage order deduplication solution is realized.
Patent Information
- Application Number
- CN202411762681.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-03
- Publication Date
- 2025-04-25
AI Technical Summary
When handling the deduplication of the logistics order number of billions of logistics, the inspection speed is too slow and the memory is used too high, which affects the system stability and service efficiency.
Using Redis-based method, dynamically select Set structure and BitMap structure for storage. For low-volume order numbers, use Set structure; when the number exceeds the threshold, it is converted to a BitMap structure to reduce memory usage and improve weight checking efficiency.
It realizes efficient deduplication verification of billions of logistics order numbers under low memory usage, improving the weight check efficiency and system stability.
Smart Images
Figure CN119884087A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of logistics, and in particular relates to a method for deduplicating billions of order numbers based on Redis, an electronic device and a storage medium. Background Art
[0002] With the rapid development of express delivery business and the access of e-commerce business systems in various channels, order number deduplication verification is required in various business scenarios. However, the current order number verification is too slow and the memory usage is too high. In order to ensure the stability of the order number deduplication verification system and improve service efficiency, enterprises need a practical solution for deduplication of hundreds of millions of order numbers. Based on this, this solution is proposed. Summary of the invention
[0003] Purpose of the invention: In order to overcome the deficiencies in the prior art, the present invention provides a method, electronic device and storage medium for deduplicating billions of order numbers based on Redis, which can realize the storage of extremely large amounts of data, reduce memory usage, and improve the efficiency of duplicate checking for logistics order numbers.
[0004] Technical solution: To achieve the above purpose, the technical solution of the present invention is as follows:
[0005] First, a method for deduplicating hundreds of millions of order numbers based on Redis:
[0006] The Redis data structure includes a Set structure and a BitMap structure;
[0007] Count the number of imported logistics order numbers and dynamically select the storage structure based on the number of logistics order numbers;
[0008] If the number of logistics order numbers does not exceed the set threshold, the logistics order number data is stored through the Set structure;
[0009] If the number of logistics order numbers exceeds the set threshold, the logistics order number data is stored through the BitMap structure;
[0010] When there is a new logistics order number that needs to be deduplicated, the logistics order number is added as a member to the corresponding Set structure or BitMap structure collection, and it is verified whether the logistics order number already exists in the Set structure or BitMap structure collection.
[0011] Furthermore, when the number of logistics order numbers exceeds a set threshold, the Set structure is converted into a BitMap structure.
[0012] Furthermore, in step 1, the condition for converting the Set structure into the BitMap structure is:
[0013] When the logistics order number is within a predetermined quantity range and the memory required by the Set structure is close to or equal to the memory required by the BitMap structure.
[0014] Furthermore, the logistics order number is split into two parts, the first part and the second part. The last 8 digits of the logistics order number are taken and converted into a numerical type to form the first part; the content after removing the last 8 digits is taken to form the second part, and the second part is used as one of the elements of the Redis Key.
[0015] Furthermore, the logistics order number is written into the Set structure and the BitMap structure at the same time;
[0016] Or, when the logistics order number is written into the Set structure, it is synchronously written into the BitMap structure.
[0017] Furthermore, in the process of converting the Set structure into the BitMap structure, the Set structure is queried first.
[0018] Furthermore, in step 1, when checking for duplicates, first determine whether the message already exists in the Set structure. If it does, it indicates that the logistics order number already exists; if it does not exist, it indicates that it is a new logistics order number.
[0019] Furthermore, in step 1, the logistics order number is mapped to a specific position on the BitMap structure, and then the SETBIT command is used to set the value of the bit to 1; when checking for duplicates, the GETBIT command is first used to check the value of the bit. If the value is 1, it indicates that the logistics order number exists; if the value is 0, it indicates that it is a new logistics order number.
[0020] In a second aspect, an electronic device is provided, comprising a memory and a processor, wherein the memory stores program instructions, and when the processor runs the program instructions, the steps of any one of the methods described in the first aspect are executed.
[0021] According to a third aspect, a storage medium stores a computer program, wherein the computer program, when executed by a processor, implements the steps of any one of the methods described in the first aspect.
[0022] Beneficial effects: The present invention dynamically selects a storage structure according to the number of logistics order numbers, and can dynamically select a Set structure and a BitMap structure. When the number is low, the Set structure is used for storage, which occupies little memory and can accurately search and check for duplicates of logistics order numbers with high accuracy and efficiency. When the number of logistics order numbers is large, the BitMap structure can be used to store very large amounts of data, reduce memory usage, and improve the efficiency of checking for duplicates of logistics order numbers. BRIEF DESCRIPTION OF THE DRAWINGS
[0023] AttachedFigure 1 It is a schematic diagram of the overall process of the present invention. DETAILED DESCRIPTION
[0024] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution in the embodiment of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiment of the present invention. Obviously, the described embodiment is only a part of the embodiment of the present invention, not all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0025] The terms used in the present disclosure are for the purpose of describing specific embodiments only and are not intended to limit the present disclosure. The singular forms "a", "said" and "the" used in the present disclosure and the appended claims are also intended to include the plural forms, unless the context clearly indicates otherwise.
[0026] The specific implementation of the present invention is further described in detail below in conjunction with the drawings and examples.
[0027] As attached Figure 1 As shown in the figure, firstly, a method for deduplication of hundreds of millions of order numbers based on Redis:
[0028] The Redis data structure includes a Set structure and a BitMap structure;
[0029] Count the number of imported logistics order numbers and dynamically select the storage structure based on the number of logistics order numbers;
[0030] If the number of logistics order numbers does not exceed the set threshold, the logistics order number data is stored through the Set structure;
[0031] If the number of logistics order numbers exceeds the set threshold, the logistics order number data is stored through the BitMap structure;
[0032] When there is a new logistics order number that needs to be deduplicated, the logistics order number is added as a member to the corresponding Set structure or BitMap structure collection, and it is verified whether the logistics order number already exists in the Set structure or BitMap structure collection.
[0033] The present invention dynamically selects a storage structure according to the number of logistics order numbers, and can dynamically select a Set structure and a BitMap structure. When the number is low, the Set structure is used for storage, which occupies little memory, and can accurately search and check for duplicates of logistics order numbers, with high accuracy and high efficiency. When the number of logistics order numbers is large, the BitMap structure can be used to store ultra-large amounts of data, reduce memory usage, and improve the efficiency of checking for duplicates of logistics order numbers.
[0034] Specifically, when the number of order numbers is small, Redis's Set structure is used for storage. The Set structure supports fast add, delete, and search operations, and is suitable for scenarios with small amounts of data. When the number of order numbers increases to a certain extent and the memory required by the Set structure approaches the memory required by the BitMap, the Set is upgraded to the BitMap structure. The BitMap structure has higher memory utilization and is suitable for large-scale data storage.
[0035] By upgrading from Set to BitMap, memory usage is significantly reduced. When processing large amounts of data, BitMap can save memory resources to the maximum extent while maintaining efficient duplicate checking. The system can dynamically adjust the storage structure according to the actual number of order numbers, thereby achieving memory optimization under any scale of data volume. Both Set and BitMap can implement complex search operations to ensure high efficiency in large-scale data scenarios. The system can receive and process order number data in real time to ensure the timeliness and accuracy of duplicate checking operations.
[0036] When the number of logistics order numbers exceeds the set threshold, the Set structure is converted into a BitMap structure, thereby changing the data storage structure to ensure low occupancy and high efficiency of duplicate checking.
[0037] Specifically, the condition for converting the Set structure into the BitMap structure is: the logistics order number is within a predetermined quantity range, and the memory required by the Set structure is close to or equal to the memory required by the BitMap structure.
[0038] In step S1, the set threshold value may be the upper limit of the memory size occupied by the set structure, or the total quantity value of the logistics order number data. Preferably, the memory required by the Set structure is close to or equal to the memory required by the BitMap. For example, when the memory occupied by the set structure exceeds 1GB, the Set structure begins to be converted into the BitMap structure.
[0039] In some specific implementations, the logistics order number is split into two parts, the first part and the second part. The last 8 digits of the logistics order number are taken and converted into a numeric type to form the first part; the content after the last 8 digits are removed is taken to form the second part, and the second part is used as one of the elements of the Redis Key to reduce the size of a single Key. This further reduces memory usage and improves duplicate checking efficiency.
[0040] Write the logistics order number into the Set structure and the BitMap structure at the same time; or when writing the logistics order number into the Set structure, write it into the BitMap structure at the same time. When upgrading the Set to the BitMap, in order to ensure availability, the order number needs to be written into both the Set and the BitMap at the same time.
[0041] In the process of converting the Set structure into the BitMap structure, the Set structure is queried first to ensure that the query is always in an available state.
[0042] In step 1, when checking for duplicates, first determine whether the message already exists in the Set structure. If it does, it indicates that the logistics order number already exists; if it does not exist, it indicates that it is a new logistics order number, and subsequent corresponding operations are performed, such as storing a new logistics order number.
[0043] In step 1, the logistics order number is mapped to a specific position on the BitMap structure, and then the SETBIT command is used to set the value of the bit to 1; when checking for duplicates, first use the GETBIT command to check the value of the bit. If the value is 1, it indicates that the logistics order number exists; if the value is 0, it indicates that it is a new logistics order number.
[0044] In a second aspect, an electronic device includes a memory and a processor, wherein the memory stores program instructions, and when the processor runs the program instructions, the steps in any one of the methods in the first aspect are executed. The electronic device in the second aspect also has all the beneficial effects in the first aspect.
[0045] In a third aspect, a storage medium stores a computer program, wherein when the computer program is executed by a processor, the steps of any one of the methods described in the first aspect are implemented. When the storage medium of the third aspect performs an operation, it also has all the beneficial effects of the first aspect.
[0046] It should be understood that although the terms first, second, third, etc. may be used in the present disclosure to describe various information, such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present disclosure, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information.
[0047] In the description of the present invention, it is necessary to understand that the terms "upper", "lower", "front", "back", "left", "right", "top", "bottom", "inside", "outside", etc., indicating the orientation or position relationship are based on the orientation or position relationship shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention.
[0048] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.
Claims
1. A method for deduplicating hundreds of millions of order numbers based on Redis, characterized in that: The Redis data structure includes a Set structure and a BitMap structure; Count the number of imported logistics order numbers and dynamically select the storage structure based on the number of logistics order numbers; If the number of logistics order numbers does not exceed the set threshold, the logistics order number data is stored through the Set structure; If the number of logistics order numbers exceeds the set threshold, the logistics order number data is stored through the BitMap structure; When there is a new logistics order number that needs to be deduplicated, the logistics order number is added as a member to the corresponding Set structure or BitMap structure collection, and it is verified whether the logistics order number already exists in the Set structure or BitMap structure collection.
2. According to the method of claim 1, wherein the method is characterized in that: When the number of logistics order numbers exceeds a set threshold, the Set structure is converted into a BitMap structure.
3. According to claim 2, a method for deduplicating hundreds of millions of order numbers based on Redis, characterized in that: In step 1, the condition for converting the Set structure to the BitMap structure is: When the logistics order number is within a predetermined quantity range and the memory required by the Set structure is close to or equal to the memory required by the BitMap structure.
4. According to claim 2, a method for deduplicating hundreds of millions of order numbers based on Redis, characterized in that: The logistics order number is split into two parts, the first part and the second part. The last 8 digits of the logistics order number are taken and converted into a numeric type to form the first part; the content after removing the last 8 digits is taken to form the second part, and the second part is used as one of the elements of the Redis Key.
5. According to a method for deduplicating hundreds of millions of order numbers based on Redis as claimed in claim 2, it is characterized in that: At the same time, the logistics order number is written into the Set structure and the BitMap structure; Or, when the logistics order number is written into the Set structure, it is synchronously written into the BitMap structure.
6. According to claim 5, a method for deduplicating hundreds of millions of order numbers based on Redis, characterized in that: In the process of converting the Set structure into the BitMap structure, the Set structure is queried first.
7. The method for deduplicating hundreds of millions of order numbers based on Redis according to claim 1, characterized in that: In step 1, when checking for duplicates, first determine whether the message already exists in the Set structure. If it does, it indicates that the logistics order number already exists; if it does not exist, it indicates that it is a new logistics order number.
8. The method for deduplicating hundreds of millions of order numbers based on Redis according to claim 1, characterized in that: In step 1, the logistics order number is mapped to a specific position on the BitMap structure, and then the SETBIT command is used to set the value of the bit to 1; when checking for duplicates, first use the GETBIT command to check the value of the bit. If the value is 1, it indicates that the logistics order number exists; if the value is 0, it indicates that it is a new logistics order number.
9. An electronic device, characterized in that: The electronic device comprises a memory and a processor, wherein the memory stores program instructions, and when the processor runs the program instructions, the steps in the method according to any one of claims 1 to 8 are executed.
10. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.