A binary library bit information compression storage method based on dynamic multi-level mapping

Abstract

This application relates to a method for compressing and storing binary storage location information based on dynamic multi-level mapping. The method comprises the following steps: collecting storage location information and establishing an original dataset, wherein the storage location information includes a storage location number, binary attributes, and non-binary attributes; mapping the information to a specified bit position in a first-level bit storage area using a preset first mapping algorithm; detecting whether the mapped bit position in the first-level bit storage area is valid; if not, completing the first-level bit storage of the storage location number; if valid, initiating second-level mapping processing, using a second mapping algorithm different from the first-level mapping algorithm to map the storage location number to a specified bit position in the second-level bit storage area and marking it as valid. This application can significantly compress storage space for tens of millions of storage location information entries, and, based on direct bit-level location, eliminates the need to traverse all records, resulting in high query efficiency.

Description

Technical Field

[0001] This application relates to the field of warehouse management technology, and in particular to a method for compressing and storing binary warehouse location information based on dynamic multi-level mapping. Background Technology

[0002] In existing warehouse management systems, inventory information is typically stored in a structured manner in a database, with each storage location corresponding to one record containing multiple fields such as storage location number, inventory status, and material information. As inventory scales to tens of millions, traditional storage methods face the problems of enormous storage space and low query efficiency.

[0003] Taking a MySQL database as an example, if the varchar(50) field is used to store the location number in UTF-8 encoding, a single record will occupy about 150 bytes. For ten million records, this field alone would require about 1.4GB of storage space. If it contains more than ten necessary fields, the total storage will reach tens of GB or even higher, resulting in high system storage pressure and slow retrieval response.

[0004] Therefore, there is an urgent need for a technical solution that can significantly reduce the storage space of inventory information while supporting efficient retrieval. Summary of the Invention

[0005] The purpose of this application is to provide a binary library location information compression and storage method based on dynamic multi-level mapping, which solves the problems of excessively large data storage and long execution time in the prior art.

[0006] The technical solution of this application is: a method for compressing and storing binary library location information based on dynamic multi-level mapping, comprising the following steps: Collect storage location information and establish an original dataset, wherein the storage location information includes storage location number, binary attributes and non-binary attributes; For each storage location number, a preset first mapping algorithm is used to map it to a specified bit position in the first-level bit storage area, and the bit is marked as valid to indicate the existence of the corresponding storage location; Check whether the bit position mapped in the first-level bit storage area is in a valid state. If it is not in a valid state, complete the first-level bit storage of the storage location number. If it is in a valid state, start the second-level mapping process. Use a second mapping algorithm different from the first-level mapping algorithm to map the storage location number to the specified bit position in the second-level bit storage area and mark it as valid. At the same time, set a repeat mark in the corresponding bit position in the first-level bit storage area. If the mapped bit position in the secondary bit storage area is still valid, the secondary mapping process is restarted repeatedly, and the subsequent levels of bit storage are started in sequence, until the bit number finds an unoccupied bit position in a certain level of bit storage area and is marked.

[0007] Preferably, for binary attribute information, a separate bit storage area is used for bit marking and storage; for non-binary attribute information, it is converted into multiple binary attributes and then stored separately in separate bit storage areas.

[0008] Preferably, the first mapping algorithm and the second mapping algorithm are hash algorithms, and the hash algorithms used for different levels of bit storage are different.

[0009] Preferably, the duplicate marker is implemented using bits to indicate that there is a conflict between the storage location number stored in the corresponding first-level bit and that the storage has been completed in the second-level bit storage area.

[0010] Preferably, the storage location information is on the scale of tens of millions, and the total storage space occupied by the bits corresponding to the storage location number is greater than or equal to 1.192MB and less than or equal to 2.384MB.

[0011] Preferably, the repetition marker is implemented using a 2-bit status code, which is defined as follows: 00 = no conflict, 01 = second-level mapping conflict, 10 = third-level or higher mapping conflict, 11 = conflict chain to be optimized, used to accurately identify the conflict level and processing status.

[0012] Preferably, when the 2-bit status code is "11", the conflict chain merging mechanism is activated, and the multi-level conflict storage location numbers triggered by the same initial mapping position are uniformly migrated to a dedicated compressed storage block. The bits are reallocated using a prefix tree encoding method, so that the storage space corresponding to the conflict chain is compressed by 30%-50%.

[0013] Preferably, for non-binary attribute information, the method of converting it into multiple binary attributes and storing them separately using independent bit storage areas includes: if it is a continuous non-binary attribute, first use the K-means clustering algorithm to divide the attribute value into multiple discrete intervals, and then convert each interval into a binary attribute of "whether it belongs to the interval" for bit storage.

[0014] Compared with the prior art, the advantages of this application are: (1) This application achieves exponential compression of storage space in the scenario of tens of millions of inventory through the core logic of "multi-level bit mapping + binary conversion". At the same time, it takes into account query efficiency, scalability and implementation cost. It is especially suitable for inventory management systems that are sensitive to storage costs, have large data volume and require high-frequency access, and provides a lightweight solution for large-scale data storage.

[0015] (2) For traditional storage of tens of millions of warehouse location information, this algorithm can realize the storage and retrieval of warehouse location information with only MB of storage, which greatly improves the calculation and response latency of the entire system. Attached Figure Description

[0017] The present application will be further described below with reference to the accompanying drawings and embodiments: Figure 1 This is a flowchart illustrating a binary library location information compression and storage method based on dynamic multi-level mapping according to this application. Figure 2 This is a schematic diagram of binary information mapping, illustrating a binary library location information compression and storage method based on dynamic multi-level mapping proposed in this application. Detailed Implementation

[0018] The content of this application will be further described in detail below with reference to specific embodiments.

[0019] Example 1:

[0020] like Figure 1 As shown, this application provides a binary warehouse location information compression and storage method based on dynamic multi-level mapping, which can compress inventory information storage space and support efficient retrieval in a warehouse management system. The solution of this application includes the following steps; S1: Collect storage location information and establish an original dataset, wherein the storage location information includes storage location number, binary attributes, and non-binary attributes; Specifically, this method can be embedded into existing WMS or WCS systems as a core module for inventory information storage and retrieval. First, a warehouse inventory information set is collected, containing a sufficiently large amount of storage location information. In this embodiment, the storage location information is calculated based on ten million storage locations. The collected storage location information includes binary and non-binary information. For binary attribute information, it is directly stored using a dedicated bit storage area for bit marking. For non-binary attribute information, it is converted into multiple binary attributes and then stored using dedicated bit storage areas for bit marking. Storage space for binary information, such as "yes / no" fields, can also be significantly compressed. Non-binary information can also be converted into binary information for storage; for example, if the storage location material type is A, B, or C, it can be converted into "yes / no material type A," "yes / no material type B," and "yes / no material type C."

[0021] For continuous non-binary attributes such as inventory quantity and remaining shelf life, the K-means clustering algorithm is first used to divide the attribute value into multiple discrete intervals, and then each interval is converted into a binary attribute of "whether it belongs to the interval" and stored in bits.

[0022] For continuous attributes, such as inventory quantity (0-1000 pieces) and remaining shelf life (0-365 days), traditional storage methods, such as MySQL's int field, require 4 bytes (32 bits) per record. However, this solution divides the data into N intervals using K-means clustering, requiring only N bits to cover all attribute values, resulting in an exponential decrease in storage requirements.

[0023] The K-means clustering algorithm automatically divides intervals by calculating the density distribution of attribute values, avoiding the problem of "overly coarse high-density intervals and overly fine low-density intervals" caused by traditional fixed intervals such as "0-100, 101-200", thus ensuring that the search results are consistent with the actual business scenario.

[0024] When continuous attributes are combined with other attributes, such as material type and storage location status, this solution directly achieves multi-dimensional filtering through bit mask operations, avoiding the "numerical comparison + field association" operation of traditional solutions. Example: Search for "Inventory quantity 100-200 units + Shelf life 30-90 days + Category A materials" Traditional approach: Requires performing multi-condition numerical matching of "Inventory quantity ≥ 100 AND Inventory quantity ≤ 200 AND Expiry date ≥ 30 AND Expiry date ≤ 90 AND Material type = A", traversing a large number of records; In this solution, the result is converted to a bit mask "00100 (inventory range) + 0100 (expiration period range) + 1000 (material type)". A single AND operation is performed to filter the results, improving efficiency by more than 1000 times.

[0025] S2: For each storage location number, a preset first mapping algorithm is used to map it to a specified bit position in the first-level bit storage area, and the bit is marked as valid to indicate the existence of the corresponding storage location; S3: Check whether the mapped bit position in the first-level bit storage area is in a valid state. If it is not in a valid state, complete the first-level bit storage of the storage location number. If it is in a valid state, start the second-level mapping process. Use a second mapping algorithm different from the first-level mapping algorithm to map the storage location number to the specified bit position in the second-level bit storage area and mark it as valid. At the same time, set a duplicate mark in the corresponding bit position in the first-level bit storage area. Specifically, taking storage location number A001 as an example, the first-level mapping calculation can map it to the 3rd bit, which will be marked as 100 in binary. If the first-level bit marked by binary 100 has a record in the record-keeping checkpoint, then a second-level mapping calculation is performed. The duplicate mark is implemented through a bit and is used to indicate that the storage location number stored in the corresponding first-level bit has a conflict and has already been stored in the secondary bit storage area. In this embodiment, the first mapping algorithm and the second mapping algorithm are hash algorithms, and the hash algorithms used for different levels of bit storage are different.

[0026] S4: If the mapped bit position in the secondary bit storage area is still valid, the secondary mapping process is restarted again, and the subsequent levels of bit storage are started in sequence until the bit number finds an unoccupied bit position in a certain level of bit storage area and is marked.

[0027] Specifically, taking storage location number A001 as an example, the first-level mapping calculation can map to the corresponding 3rd bit, which will be marked as 100 in binary. If the first-level bit marked by binary 100 is stored in the record whether it is a duplicate, the second-level mapping calculation is performed. If there is still a duplicate mark, the second-level mapping process is restarted again, and the subsequent levels of bit storage are started in sequence until the storage location number finds an unoccupied bit position in a certain level of bit storage area and the marking is completed.

[0028] The storage location information is on the order of tens of millions of bits. The total storage space occupied by the bits corresponding to the storage location number alone is greater than or equal to 1.192MB and less than or equal to 2.384MB. The entire record of the storage number requires 10 million bits, which is approximately 1.192MB. Even in the worst case scenario with multiple repetitions, it will still be less than 2.384MB. The storage size of just one field is reduced from 1.4GB in the original method to less than 2.384MB through compression using this algorithm, a reduction of 587.24 times.

[0029] This embodiment achieves exponential compression of storage space in a scenario with tens of millions of inventory records through the core logic of "multi-level bit mapping + binary conversion". At the same time, it takes into account query efficiency, scalability and implementation cost. It is especially suitable for inventory management systems that are sensitive to storage costs, have large data volumes and require high-frequency access, and provides a lightweight solution for large-scale data storage.

[0030] For traditional storage of tens of millions of warehouse location information, this algorithm can achieve storage and retrieval of warehouse location information with only MB of storage, which greatly improves the computation and response latency of the entire system.

[0031] Example 2:

[0032] This embodiment takes a basic scenario with tens of millions of storage spaces as an example: A certain warehouse center has 10 million storage locations. The storage location information includes: storage location number (such as "A001-001" "B123-456"), binary attributes (whether it is an empty storage location, whether it is a cold chain storage location), and non-binary attributes (material type: A / B / C / D, inventory quantity range: 0 / 1-10 / 11-50 / 50+, remaining days of expiration: 0-30 / 31-90 / 91-180 / 180+).

[0033] Traditional storage solutions are shown in the table below:

[0034] The compressed storage scheme of this invention is as follows:

[0035] As can be seen from the above comparison, the storage compression ratio of this application is approximately 80 times (if the conflict rate is less than 5%, the actual compression ratio can reach more than 100 times); and the traditional solution requires more than 2GB of hard disk storage, while this solution only requires 20MB, reducing the storage hardware cost of a single warehouse center by 99%; the amount of data during data transmission backup / synchronization is reduced from 1.53GB to 19MB, and the transmission time is shortened by 80 times.

[0036] The above embodiments are merely illustrative of the technical concept and features of this application, intended to enable those skilled in the art to understand the content of this application and implement it accordingly, and should not be construed as limiting the scope of protection of this application. It is obvious to those skilled in the art that this application is not limited to the details of the above exemplary embodiments, and that this application can be implemented in other specific forms without departing from the spirit or essential characteristics of this application. Therefore, the embodiments should be considered exemplary and non-limiting in all respects. The scope of this application is defined by the appended claims rather than the foregoing description, and thus all variations falling within the meaning and scope of the equivalents of the claims are intended to be included within this application.

Claims

1. A binary library bit information compression storage method based on dynamic multi-level mapping, characterized by, Includes the following steps: Collect storage location information and establish an original dataset, wherein the storage location information includes storage location number, binary attributes and non-binary attributes; For each storage location number, a preset first mapping algorithm is used to map it to a specified bit position in the first-level bit storage area, and the bit is marked as valid to indicate the existence of the corresponding storage location; Check whether the mapped bit position in the first-level bit storage area is in a valid state. If it is not in a valid state, complete the first-level bit storage of the storage location number. If it is in a valid state, start the second-level mapping process. Use a second mapping algorithm, which is different from the first-level mapping algorithm, to map the storage location number to the specified bit position in the second-level bit storage area and mark it as valid. At the same time, set a repeat mark at the corresponding bit position in the first-level bit storage area. If the mapped bit position in the secondary bit storage area is still valid, the secondary mapping process is restarted repeatedly, and the subsequent levels of bit storage are started in sequence, until the bit number finds an unoccupied bit position in a certain level of bit storage area and is marked.

2. The binary library bit information compression storage method based on dynamic multi-level mapping according to claim 1, characterized in that, For binary attribute information, a separate bit storage area is used for bit marking and storage; for non-binary attribute information, it is converted into multiple binary attributes and then stored separately in separate bit storage areas.

3. The binary library bit information compression storage method based on dynamic multi-level mapping according to claim 1, characterized in that, The first mapping algorithm and the second mapping algorithm are hash algorithms, and the hash algorithms used for different levels of bit storage are different.

4. The binary library bit information compression storage method based on dynamic multi-level mapping of claim 1, wherein, The duplicate marker is implemented using bits to indicate that there is a conflict between the storage location number stored in the corresponding first-level bit and that the storage has been completed in the second-level bit storage area.

5. The binary library bit information compression storage method based on dynamic multi-level mapping of claim 1, wherein, The storage location information is in the tens of millions, and the total storage space occupied by the bits corresponding to the storage location number is greater than or equal to 1.192MB and less than or equal to 2.384MB.

6. The binary library location information compression and storage method based on dynamic multi-level mapping according to claim 1, characterized in that, The repetition marker is implemented using a 2-bit status code, which is defined as follows: 00 = no conflict, 01 = second-level mapping conflict, 10 = third-level or higher mapping conflict, and 11 = conflict chain to be optimized. This is used to accurately identify the conflict level and processing status.

7. The binary library location information compression and storage method based on dynamic multi-level mapping according to claim 6, characterized in that, When the 2-bit status code is "11", the conflict chain merging mechanism is activated. The multi-level conflict library numbers triggered by the same initial mapping position are uniformly migrated to a dedicated compressed storage block. The bits are reallocated using a prefix tree encoding method, so that the storage space corresponding to the conflict chain is compressed by 30%-50%.

8. The binary library location information compression and storage method based on dynamic multi-level mapping according to claim 1, characterized in that, For non-binary attribute information, the method of converting it into multiple binary attributes and storing them separately in independent bit storage areas includes: if it is a continuous non-binary attribute, first use the K-means clustering algorithm to divide the attribute value into multiple discrete intervals, and then convert each interval into a binary attribute of "whether it belongs to the interval" for bit storage.

Images