Hash table creation method, device and equipment, and storage medium

Abstract

Embodiments of the present application disclose a hash table creation method, device and equipment and a storage medium. The method is executed by a data storage system, at least one hash table is stored in the data storage system, each hash table comprises a plurality of original hash sub-tables, the method comprises: determining a load factor of an original hash sub-table for each original hash sub-table; when the load factor reaches a set proportion threshold, creating a new hash sub-table as an updated hash sub-table according to pre-set sub-table creation parameters of the hash sub-table; inserting a stored key-value pair in the original hash sub-table into the updated hash sub-table, and releasing the memory space occupied by the original hash sub-table. The method provided by the embodiments of the present application uses a plurality of original hash sub-tables for data storage, so that when a new hash sub-table is created, the new hash sub-table is created based on the hash sub-table with less memory occupation, thereby reducing the memory occupation required for creating a new hash sub-table when data storage is frequent.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method, apparatus, device and storage medium for creating hash tables. Background Technology

[0002] A hash table is a commonly used data structure, often referred to as a key-value pair, for storing key-value pairs. Hash tables access data records by calculating a function about the key, mapping the desired key to a location in the table, thus speeding up lookups. This mapping function is called the hash function, the array storing the data is called the hash table, and the storage location corresponding to each hash value is called a bucket or slot. As the number of key-value pairs inserted into the hash table increases, free spaces decrease, and the probability of hash collisions becomes higher, leading to a decrease in query efficiency.

[0003] In the process of developing this invention, the inventors discovered at least the following technical problems in the prior art: To ensure the operational efficiency of the hash table as much as possible, a new hash table needs to be created when the hash table is under high load. When creating a new hash table, there may be a situation where both the new and old hash tables exist simultaneously, resulting in high peak server memory usage and low long-term server memory utilization. Summary of the Invention

[0004] This invention provides a method, apparatus, device, and storage medium for creating hash tables, thereby reducing the peak memory usage of hash tables.

[0005] In a first aspect, embodiments of the present invention provide a hash table creation method, executed by a data storage system, wherein the data storage system stores at least one hash table, each hash table including multiple original hash sub-tables, and the method includes:

[0006] For at least one original hash subtable, determine the load factor of the original hash subtable;

[0007] When the load factor reaches the set ratio threshold, a new hash sub-table is created based on the pre-set sub-table creation parameters of the hash sub-table as the updated hash sub-table;

[0008] Insert the stored key-value pairs from the original hash subtable into the updated hash subtable, and release the memory space occupied by the original hash subtable.

[0009] Optionally, based on the above scheme, the sub-table creation parameters are determined according to the capacity of the original hash sub-table.

[0010] Optionally, based on the above scheme, the load factor of the original hash sub-table is determined, including:

[0011] The load factor of the original hash subtable is determined based on its capacity and the number of key-value pairs already stored in it.

[0012] Optionally, based on the above scheme, the load factor of the original hash sub-table is determined according to the capacity of the original hash sub-table and the number of key-value pairs already stored in the original hash sub-table, including:

[0013] The load factor of the original hash subtable is determined using the following formula:

[0014] K = M / N;

[0015] Where K is the load factor of the original hash subtable, M is the number of key-value pairs already stored in the original hash subtable, and N is the capacity of the original hash subtable.

[0016] Optionally, in addition to the above solutions, the following also applies:

[0017] Obtain the data to be stored and its storage identifier;

[0018] Determine the target hash sub-table index corresponding to the data to be stored based on the storage identifier;

[0019] The data to be stored is inserted into the target hash subtable based on the target hash subtable index.

[0020] Optionally, based on the above scheme, the target hash sub-table index corresponding to the data to be stored is determined according to the storage identifier, including:

[0021] The target hash subtable index corresponding to the data to be stored is determined based on the storage identifier and the number of hash subtables.

[0022] Optionally, based on the above scheme, the data to be stored is inserted into the target hash subtable based on the target hash subtable index, including:

[0023] The bucket index for the data to be stored is determined based on the storage identifier and the current capacity of the target hash subtable.

[0024] Insert the data to be stored into the target hash subtable based on the bucket index of the data to be stored.

[0025] Optionally, based on the above scheme, the bucket index for the data to be stored is determined according to the storage identifier and the current capacity of the target hash sub-table, including:

[0026] The bucket index for the data to be stored is determined using the following formula:

[0027] bucket_index=hash_value%subtable_capacity;

[0028] Where bucket_index is the bucket index of the data to be stored, hash_value is the storage identifier, and subtable_capacity is the current subtable capacity of the target hash subtable.

[0029] Secondly, embodiments of the present invention also provide a hash table creation apparatus, configured in a data storage system, wherein the data storage system stores at least one hash table, each hash table including multiple original hash sub-tables, and the apparatus includes:

[0030] The load factor determination module is used to determine the load factor of at least one original hash sub-table.

[0031] The hash sub-table creation module is used to create a new hash sub-table as an update hash sub-table when the load factor reaches a set ratio threshold.

[0032] The key-value pair processing module is used to insert the key-value pairs already stored in the original hash subtable into the updated hash subtable, thereby releasing the memory space occupied by the original hash subtable.

[0033] Thirdly, embodiments of the present invention also provide a computer device, the device comprising:

[0034] One or more processors;

[0035] Storage device for storing one or more programs;

[0036] When one or more programs are executed by one or more processors, the one or more processors implement the hash table creation method provided in any embodiment of the present invention.

[0037] Fourthly, embodiments of the present invention also provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the hash table creation method as provided in any embodiment of the present invention.

[0038] The hash table creation method provided in this invention determines the load factor of at least one original hash sub-table. When the load factor reaches a set threshold, a new hash sub-table is created as an updated hash sub-table according to pre-set sub-table creation parameters. The stored key-value pairs from the original hash sub-table are inserted into the updated hash sub-table, releasing the memory space occupied by the original hash sub-table. By setting multiple original hash sub-tables for data storage, when the load on a hash sub-table is high and a new hash sub-table needs to be created, a new hash sub-table is created based on a hash sub-table with lower memory usage, reducing the memory usage required to create a new hash sub-table and thus reducing the peak memory usage of the hash table. Attached Figure Description

[0039] Figure 1a This is a flowchart illustrating a hash table creation method provided in Embodiment 1 of the present invention;

[0040] Figure 1b This is a schematic diagram of the structure of a hash table provided in Embodiment 1 of the present invention;

[0041] Figure 2 This is a schematic diagram of the structure of a hash table creation device provided in Embodiment 2 of the present invention;

[0042] Figure 3 This is a schematic diagram of the structure of a computer device provided in Embodiment 3 of the present invention. Detailed Implementation

[0043] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it. Furthermore, it should be noted that, for ease of description, the accompanying drawings show only the parts relevant to the present invention, and not all of the structures.

[0044] Example 1

[0045] Figure 1a This is a flowchart illustrating a hash table creation method provided in Embodiment 1 of the present invention. This embodiment is applicable to situations where data is stored using a hash table data structure. The method can be executed by a hash table creation device, which can be implemented in software and / or hardware. For example, the hash table creation device can be configured in a computer device of a data storage system. Figure 1a As shown, the method includes:

[0046] S110. For at least one original hash subtable, determine the load factor of the original hash subtable.

[0047] The hash table creation method provided in this embodiment is executed by a data storage system. The data storage system stores at least one hash table, and each hash table includes multiple original hash sub-tables. It can be applied in different scenarios, such as machine learning scenarios and business scenarios. The implementation of hash tables differs in different application scenarios. Taking the C++ programming language as an example, there are implementations such as `std::unordered_map` from the standard template library, `boost::unordered_map` from the Boost open-source component, and `flat_hash_map` and `node_hash_map` from Google's Abseil open-source component. When using the above hash tables for large-scale storage of a single table, all of them suffer from excessively high peak memory usage during rehashing.

[0048] In this embodiment, to avoid rehashing the hash table, creating a new hash table consumes a significant amount of memory. Therefore, the hash table is divided into multiple original hash sub-tables. That is, the hash table is a one-dimensional array composed of hash sub-tables with contiguous addresses. Each hash sub-table has its corresponding memory address range.

[0049] During data storage, the capacity of any original hash sub-table will dynamically adjust according to the change of its load factor. The load factor of the original hash sub-table represents the size of the occupied capacity of the original hash sub-table. The larger the load factor, the larger the occupied capacity of the original hash sub-table, and the smaller the load factor, the smaller the occupied capacity of the original hash sub-table.

[0050] Optionally, determining the load factor of the original hash subtable includes: determining the load factor of the original hash subtable based on its capacity and the number of key-value pairs already stored in it. As described above, the load factor of the original hash subtable characterizes the occupancy of the key-value pairs stored in the hash subtable. The number of key-value pairs already stored in the original hash subtable is strongly correlated with its occupied capacity. Therefore, the load factor of the original hash subtable can be calculated based on the number of key-value pairs already stored in it and its capacity.

[0051] Specifically, the load factor of the original hash subtable is determined based on its capacity and the number of key-value pairs already stored. This is achieved using the following formula: K = M / N; where K is the load factor, M is the number of key-value pairs already stored, and N is the capacity of the original hash subtable. In other words, the ratio of the number of key-value pairs already stored to the capacity of the original hash subtable can be used as its load factor. For example, if the number of key-value pairs already stored in the original hash subtable is 4 and its capacity is 8, then the load factor is 4 / 8 = 0.5.

[0052] S120. When the load factor reaches the set ratio threshold, a new hash sub-table is created as the updated hash sub-table according to the pre-set hash sub-table creation parameters.

[0053] When the load factor of the original hash subtable reaches a set threshold, a new hash subtable is created according to pre-set subtable creation parameters. These subtable creation parameters can be the capacity of the new hash subtable. It is understood that the subtable creation parameters are determined based on the capacity of the original hash subtable. Optionally, the subtable creation parameters can be K times the capacity of the original hash subtable, where K is an integer greater than 1. For example, the subtable creation parameters can be twice the capacity of the original hash subtable. Determining the subtable creation parameters based on the capacity of the original hash subtable, which has lower memory consumption, results in lower memory usage when creating new hash subtables, avoiding excessive memory consumption during peak periods.

[0054] For example, assuming the original hash sub-table has a capacity of 953MB, the new hash sub-table can be set to have a capacity of 1906MB, meaning the sub-table creation parameter is set to the new hash sub-table's capacity of 1906MB. As the number of key-value pairs inserted into the original hash sub-table increases, the number of free spaces decreases, leading to a higher probability of hash collisions and reduced query efficiency. To ensure the operational efficiency of the original hash sub-table as much as possible, it is generally recommended to maintain a certain number of free buckets within the original hash sub-table. In this embodiment, a load factor can be used to represent the number of empty buckets. When the load factor reaches a set threshold, a rehashing operation is required: a new hash sub-table with a capacity twice that of the original hash sub-table is created; the key-value pairs stored in the original hash sub-table are recalculated and inserted into the new hash sub-table; after migration, the memory space occupied by the original hash sub-table is released. The set threshold can be adjusted according to actual conditions.

[0055] In this embodiment, the original hash sub-table has a small memory footprint and is only a part of the hash table. Because each original hash sub-table carries a portion of the entire hash table, when a single original hash sub-table undergoes rehashing, the peak memory usage is smaller compared to the memory required for rehashing the entire hash table and creating a new hash table in existing technologies, thus avoiding peak memory consumption. Furthermore, for editing operations on the original hash table, such as CRUD operations, the hash value of the key is only calculated once when determining the storage identifier. During the operation, only bitwise AND operations of the original hash sub-table are added, resulting in minimal increased system overhead.

[0056] Figure 1b This is a schematic diagram of the structure of a hash table provided in Embodiment 1 of the present invention. Figure 1bTaking a hash table implemented using the zipper method as an example, this embodiment illustrates the hash table creation method provided. Figure 1b As shown, during initialization, the hash table specifies the size of the original hash sub-table array as 8. That is, the hash table includes 8 original hash sub-tables: hash sub-table 0, hash sub-table 1, hash sub-table 2, hash sub-table 3, hash sub-table 4, hash sub-table 5, hash sub-table 6, and hash sub-table 7. When data needs to be stored in the hash table, the hash value of the data to be stored is first calculated, and then the hash sub-table index is calculated based on the hash value and the number of original hash sub-tables. After determining the hash sub-table index, the data to be stored is inserted into the original hash sub-table corresponding to the hash sub-table index. Each bucket in the hash table stores pointers to key-value pair nodes; when the computing system is a 64-bit computer system, each pointer is 8 bytes. Assuming the current hash table stores 1 billion key-value pairs, these key-value pairs are evenly distributed across these 8 sub-tables, and the memory occupied by each hash sub-table is: 1,000,000,000*8 / 1024 / 1024 / 8≈953MB.

[0057] As stored data increases, when a sub-table of the original hash table needs to be rehashed, the memory occupied by the new hash table created from the original sub-table is 953 * 2 = 1906 MB. At this point, the peak memory usage of the original sub-table is 1,000,000,000 * 8 / 1024 / 1024 + 1906 = 9593 MB. In current technology, on a 64-bit computer system, each pointer is 8 bytes. If the current hash table has a capacity of 1 billion, then the memory occupied by the current hash table itself is approximately 1,000,000,000 * 8 / 1024 / 1024 ≈ 7629 MB. When rehashing is performed, the created hash table is twice the original size, reaching 2 billion hashes. Therefore, the total memory occupied by the new hash table is approximately 2,000,000,000 * 8 / 1024 / 1024 ≈ 15258 MB. Therefore, it can be seen that the hash sub-table creation technology provided by the embodiments of the present invention greatly reduces peak memory usage and improves server resource utilization compared with the prior art.

[0058] S130. Insert the stored key-value pairs from the original hash subtable into the updated hash subtable and release the memory space occupied by the original hash subtable.

[0059] After creating a new hash subtable, the key-value pairs already stored in the original hash subtable are recalculated for hash values. Based on the recalculated hash values, the key-value pairs already stored in the original hash subtable are inserted into the new hash subtable. After the migration is complete, the memory space occupied by the original hash subtable is released, and the creation of the hash subtable is completed.

[0060] The hash table creation method provided in this invention determines the load factor of at least one original hash sub-table. When the load factor reaches a set threshold, a new hash sub-table is created as an updated hash sub-table according to pre-set sub-table creation parameters. The stored key-value pairs from the original hash sub-table are inserted into the updated hash sub-table, releasing the memory space occupied by the original hash sub-table. By setting multiple original hash sub-tables for data storage, when the load on a hash sub-table is high and a new hash sub-table needs to be created, a new hash sub-table is created based on a hash sub-table with lower memory usage. This reduces the memory usage required to create new hash sub-tables when data storage is frequent, thereby reducing the peak memory usage of the hash table.

[0061] Based on the above scheme, data storage needs to be performed using a hash sub-table. Optionally, the data storage method includes: obtaining the data to be stored and its storage identifier; determining the target hash sub-table index corresponding to the data to be stored based on the storage identifier; and inserting the data to be stored into the target hash sub-table based on the target hash sub-table index.

[0062] It is understood that the data to be stored is data that needs to be stored in a hash table data structure. The specific data is determined based on the application scenario of the data storage and is not limited here. For example, in a machine learning scenario, it can be used to store embedding vectors of images and videos, or model parameters for large-scale sparse deep learning. Therefore, when the data storage scenario is a machine learning scenario, the data to be stored can be embedding vectors of text, images, and videos, model parameters, etc. Here, the embedding vector can be understood as a floating-point vector representing the features of images, text, videos, etc., used to represent the features of the content such as images, text, and videos. The storage identifier of the data to be stored can be the location identifier for storing the data in the hash table. Generally, the storage identifier of the data to be stored can be determined by the hash value of the data to be stored. The hash value of the data to be stored can be calculated using a pre-set hash function and used as the storage identifier of the data to be stored, or the feature value of the hash value can be used as the storage identifier of the data to be stored. The hash function can be set according to actual needs; for example, the hash value of the data to be stored can be calculated using the hash function hasher().

[0063] When storing data, the data to be stored is stored in the original hash subtable. That is, when storing data, it is necessary to determine the original hash subtable where the data to be stored will be stored, i.e., determine the index of the hash subtable where the data to be stored will be stored as the target hash subtable index. The number of original hash subtables can be set according to actual needs and is not limited here. Preferably, it can be set to a power of 2 to reduce the overhead of hash calculation. Optionally, the target hash subtable index corresponding to the data to be stored is determined based on the storage identifier, including: determining the target hash subtable index corresponding to the data to be stored based on the storage identifier and the number of hash subtables. When the storage identifier is the hash value of the data to be stored, the target hash subtable index is calculated based on the hash value. Optionally, a bitwise AND operation can be used to calculate the target hash subtable index. Assuming the hash table includes m hash subtables, the target hash subtable index can be calculated using subtable_index = hash_value & 0x(m-1), where subtable_index is the target hash subtable index and hash_value is the hash value of the data to be stored.

[0064] After determining the target hash subtable index, the data to be stored is inserted into the target hash subtable corresponding to the target hash subtable index. Optionally, inserting the data to be stored into the target hash subtable based on the target hash subtable index includes: determining the bucket index of the data to be stored based on the storage identifier and the current subtable capacity of the target hash subtable; and inserting the data to be stored into the target hash subtable based on the bucket index of the data to be stored. It can be understood that the hash value of the data to be stored has already been calculated when calculating the storage identifier. When inserting the key-value pairs of the data to be stored into the target hash subtable, it is not necessary to recalculate it. It is only necessary to calculate the bucket index according to the current subtable capacity of the target hash subtable and insert the data to be stored into the position corresponding to the bucket index of the target hash subtable. The process of determining the bucket index of the data to be stored, based on the storage identifier and the current capacity of the target hash subtable, includes: The bucket index of the data to be stored is determined using the following formula: bucket_index = hash_value % subtable_capacity; where bucket_index is the bucket index of the data to be stored, hash_value is the storage identifier, and subtable_capacity is the current capacity of the target hash subtable. Specifically, when the storage identifier is a hash value, taking the remainder of the storage identifier divided by the current capacity of the target hash subtable yields the bucket index of the data to be stored within the target hash subtable.

[0065] It is understandable that the creation of hash sub-tables and the storage of data can be performed simultaneously or sequentially. For example, assuming the data storage system includes original hash sub-table 1 and original hash sub-table 2, the creation of a new hash sub-table from original hash sub-table 1 and the storage of the data to be stored in original hash sub-table 2 can be performed simultaneously. Alternatively, after data storage, the load factor of the original hash sub-table can be calculated, and the creation of a new hash sub-table can be performed if the load factor is greater than a threshold.

[0066] Storing data in a hash sub-table reduces the memory usage required to create new hash sub-tables when data storage is frequent, thereby reducing the peak memory usage of the hash table.

[0067] Example 2

[0068] Figure 2 This is a schematic diagram of a hash table creation device provided in Embodiment 2 of the present invention. The hash table creation device can be implemented in software and / or hardware. For example, the hash table creation device can be configured in a computer device of a data storage system, wherein the data storage system stores at least one hash table, and each hash table includes multiple original hash sub-tables. Figure 2 As shown, the device includes a load factor determination module 210, a hash sub-table creation module 220, and a key-value pair processing module 230, wherein:

[0069] The load factor determination module 210 is used to determine the load factor of at least one original hash sub-table.

[0070] The hash sub-table creation module 220 is used to create a new hash sub-table as an update hash sub-table when the load factor reaches a set ratio threshold.

[0071] The key-value pair processing module 230 is used to insert the stored key-value pairs in the original hash subtable into the updated hash subtable, thereby releasing the memory space occupied by the original hash subtable.

[0072] The hash table creation apparatus provided in this embodiment of the invention determines the load factor of at least one original hash sub-table. When the load factor reaches a set percentage threshold, a new hash sub-table is created as an updated hash sub-table according to pre-set sub-table creation parameters. The stored key-value pairs from the original hash sub-table are inserted into the updated hash sub-table, releasing the memory space occupied by the original hash sub-table. By setting multiple original hash sub-tables for data storage, when the load on a hash sub-table is high and a new hash sub-table needs to be created, a new hash sub-table is created based on a hash sub-table with lower memory usage. This reduces the memory usage required to create new hash sub-tables when data storage is frequent, thereby reducing the peak memory usage of the hash table.

[0073] Optionally, based on the above scheme, the sub-table creation parameters are determined according to the capacity of the original hash sub-table.

[0074] Optionally, based on the above scheme, the load factor determination module 210 is specifically used for:

[0075] The load factor of the original hash subtable is determined based on its capacity and the number of key-value pairs already stored in it.

[0076] Optionally, based on the above scheme, the load factor determination module 210 is specifically used for:

[0077] The load factor of the original hash subtable is determined using the following formula:

[0078] K = M / N;

[0079] Where K is the load factor of the original hash subtable, M is the number of key-value pairs already stored in the original hash subtable, and N is the capacity of the original hash subtable.

[0080] Optionally, based on the above scheme, the device further includes a data storage module, comprising:

[0081] A storage identifier determination unit is used to obtain the data to be stored and the storage identifier of the data to be stored.

[0082] The sub-table index determination unit is used to determine the target hash sub-table index corresponding to the data to be stored based on the storage identifier.

[0083] The data storage unit is used to insert the data to be stored into the target hash subtable based on the target hash subtable index.

[0084] Optionally, based on the above scheme, the sub-table index determination unit is specifically used for:

[0085] The target hash subtable index corresponding to the data to be stored is determined based on the storage identifier and the number of hash subtables.

[0086] Optionally, based on the above scheme, the data storage unit is specifically used for:

[0087] The bucket index for the data to be stored is determined based on the storage identifier and the current capacity of the target hash sub-table;

[0088] Insert the data to be stored into the target hash subtable based on the bucket index of the data to be stored.

[0089] Optionally, based on the above scheme, the data storage unit is specifically used for:

[0090] The bucket index for the data to be stored is determined using the following formula:

[0091] bucket_index=hash_value%subtable_capacity;

[0092] Where bucket_index is the bucket index of the data to be stored, hash_value is the storage identifier, and subtable_capacity is the current subtable capacity of the target hash subtable.

[0093] The hash table creation apparatus provided in the embodiments of the present invention can execute the hash table creation method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of executing the method.

[0094] Example 3

[0095] Figure 3 This is a schematic diagram of the structure of a computer device provided in Embodiment 3 of the present invention. Figure 3 A block diagram of an exemplary computer device 312 suitable for implementing embodiments of the present invention is shown. Figure 3 The computer device 312 shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of the present invention.

[0096] like Figure 3 As shown, the computer device 312 is presented in the form of a general-purpose computing device. The components of the computer device 312 may include, but are not limited to: one or more processors 316, system memory 328, and bus 318 connecting different system components (including system memory 328 and processor 316).

[0097] Bus 318 represents one or more of several bus architectures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration port, a processor 316, or a local bus using any of the various bus architectures. For example, these architectures include, but are not limited to, the Industry Standard Architecture (ISA) bus, the Micro Channel Architecture (MAC) bus, the Enhanced ISA bus, the Video Electronics Standards Association (VESA) local bus, and the Peripheral Component Interconnect (PCI) bus.

[0098] Computer device 312 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by computer device 312, including volatile and non-volatile media, removable and non-removable media.

[0099] System memory 328 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 330 and / or cache memory 332. Computer device 312 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, storage device 334 may be used to read and write non-removable, non-volatile magnetic media (…). Figure 3 Not shown; usually referred to as a "hard drive"). Although Figure 3 Not shown, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk") and an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to bus 318 via one or more data media interfaces. Memory 328 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of the embodiments of the present invention.

[0100] A program / utility 340 having a set (at least one) of program modules 342 may be stored, for example, in memory 328. Such program modules 342 include, but are not limited to, an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include an implementation of a network environment. Program modules 342 typically perform the functions and / or methods described in the embodiments of the present invention.

[0101] Computer device 312 can also communicate with one or more external devices 314 (e.g., keyboard, pointing device, display 324, etc.), and with one or more devices that enable a user to interact with the computer device 312, and / or with any device that enables the computer device 312 to communicate with one or more other computing devices (e.g., network card, modem, etc.). This communication can be performed via input / output (I / O) interface 322. Furthermore, computer device 312 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 320. As shown, network adapter 320 communicates with other modules of computer device 312 via bus 318. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with computer device 312, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0102] The processor 316 executes various functional applications and data processing by running programs stored in the system memory 328, such as implementing the hash table creation method provided in this embodiment of the invention, which includes:

[0103] For at least one original hash subtable, determine the load factor of the original hash subtable;

[0104] When the load factor reaches the set ratio threshold, a new hash sub-table is created based on the pre-set sub-table creation parameters of the hash sub-table as the updated hash sub-table;

[0105] Insert the stored key-value pairs from the original hash subtable into the updated hash subtable, and release the memory space occupied by the original hash subtable.

[0106] Of course, those skilled in the art will understand that the processor can also implement the technical solution of the hash table creation method provided in any embodiment of the present invention.

[0107] Example 4

[0108] Embodiment 4 of the present invention also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the hash table creation method provided in the embodiments of the present invention, the method comprising:

[0109] For at least one original hash subtable, determine the load factor of the original hash subtable;

[0110] When the load factor reaches the set ratio threshold, a new hash sub-table is created based on the pre-set sub-table creation parameters of the hash sub-table as the updated hash sub-table;

[0111] Insert the stored key-value pairs from the original hash subtable into the updated hash subtable, and release the memory space occupied by the original hash subtable.

[0112] Of course, the computer program stored on the computer-readable storage medium provided in the embodiments of the present invention is not limited to the above-described method operations, but can also perform related operations of the hash table creation method provided in any embodiment of the present invention.

[0113] The computer storage medium of this invention can be any combination of one or more computer-readable media. A computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0114] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, capable of sending, propagating, or transmitting programs for use by or in connection with an instruction execution system, apparatus, or device.

[0115] Program code contained on a computer-readable medium may be transmitted using any suitable medium, including—but not limited to—wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0116] Computer program code for performing the operations of this invention can be written in one or more programming languages ​​or a combination thereof. Programming languages ​​include object-oriented programming languages—such as Java, Smalltalk, and C++—as well as conventional procedural programming languages—such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0117] Note that the above description is merely a preferred embodiment of the present invention and the technical principles employed. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and various obvious changes, readjustments, and substitutions can be made without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in detail through the above embodiments, the present invention is not limited to the above embodiments, and may include many other equivalent embodiments without departing from the concept of the present invention, the scope of which is determined by the scope of the appended claims.

Claims

1. A method for creating a hash table, characterized in that, The method is executed by a data storage system that stores at least one hash table, each hash table comprising multiple original hash sub-tables. Each hash table is a one-dimensional array of hash sub-tables with contiguous addresses, and each hash sub-table has a corresponding memory address range. The method includes: For at least one of the original hash sub-tables, determine the load factor of the original hash sub-table; When the load factor reaches the set ratio threshold, a new hash sub-table is created according to the pre-set hash sub-table creation parameters as the updated hash sub-table; Insert the stored key-value pairs from the original hash subtable into the updated hash subtable, and release the memory space occupied by the original hash subtable; The hash table creation method further includes: Obtain the data to be stored and the storage identifier of the data to be stored; The target hash sub-table index corresponding to the data to be stored is determined based on the storage identifier; The data to be stored is inserted into the target hash subtable based on the target hash subtable index; The step of determining the target hash sub-table index corresponding to the data to be stored based on the storage identifier includes: Based on the storage identifier and the number of hash sub-tables, a bitwise AND operation is used to determine the index of the target hash sub-table corresponding to the data to be stored; The step of inserting the data to be stored into the target hash sub-table based on the target hash sub-table index includes: The bucket index of the data to be stored is determined based on the storage identifier and the current sub-table capacity of the target hash sub-table; Insert the data to be stored into the target hash sub-table based on the bucket index of the data to be stored; Determining the bucket index of the data to be stored based on the storage identifier and the current sub-table capacity of the target hash sub-table includes: The bucket index of the data to be stored is determined using the following formula: bucket_index = hash_value % subtable_capacity; Wherein, bucket_index is the bucket index of the data to be stored, hash_value is the storage identifier, and subtable_capacity is the current subtable capacity of the target hash subtable; The creation of a new hash subtable from an original hash subtable and the storage of the data to be stored in another original hash subtable are performed simultaneously.

2. The method according to claim 1, characterized in that, The sub-table creation parameters are determined based on the capacity of the original hash sub-table.

3. The method according to claim 1, characterized in that, Determining the load factor of the original hash sub-table includes: The load factor of the original hash subtable is determined based on the capacity of the original hash subtable and the number of key-value pairs already stored in the original hash subtable.

4. The method according to claim 3, characterized in that, The step of determining the load factor of the original hash sub-table based on its capacity and the number of key-value pairs already stored in it includes: The load factor of the original hash subtable is determined using the following formula: K=M / N; Where K is the load factor of the original hash sub-table, M is the number of key-value pairs already stored in the original hash sub-table, and N is the capacity of the original hash sub-table.

5. A hash table creation apparatus, characterized in that, Configured in a data storage system, the data storage system stores at least one hash table, each hash table including multiple original hash sub-tables, wherein the hash table is a one-dimensional array composed of hash sub-tables with contiguous addresses, each hash sub-table having its corresponding memory address range, the device comprising: A load factor determination module is used to determine the load factor of at least one of the original hash sub-tables. The hash sub-table creation module is used to create a new hash sub-table as an update hash sub-table when the load factor reaches a set ratio threshold. The key-value pair processing module is used to insert the stored key-value pairs in the original hash sub-table into the updated hash sub-table, thereby releasing the memory space occupied by the original hash sub-table. The hash table creation device further includes a data storage module, comprising: A storage identifier determination unit is used to obtain the data to be stored and the storage identifier of the data to be stored. The sub-table index determination unit is used to determine the target hash sub-table index corresponding to the data to be stored based on the storage identifier; A data storage unit is used to insert the data to be stored into the target hash sub-table based on the target hash sub-table index; The sub-table index determination unit is specifically used for: Based on the storage identifier and the number of hash sub-tables, a bitwise AND operation is used to determine the index of the target hash sub-table corresponding to the data to be stored; The data storage unit is specifically used for: The bucket index of the data to be stored is determined based on the storage identifier and the current sub-table capacity of the target hash sub-table; Insert the data to be stored into the target hash sub-table based on the bucket index of the data to be stored; The data storage unit is specifically used for: The bucket index of the data to be stored is determined using the following formula: bucket_index = hash_value % subtable_capacity; Wherein, bucket_index is the bucket index of the data to be stored, hash_value is the storage identifier, and subtable_capacity is the current subtable capacity of the target hash subtable; The creation of a new hash subtable from an original hash subtable and the storage of the data to be stored in another original hash subtable are performed simultaneously.

6. A computer device, characterized in that, The device includes: One or more processors; Storage device for storing one or more programs; When one or more programs are executed by one or more processors, the one or more processors implement the hash table creation method as described in any one of claims 1-4.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the hash table creation method as described in any one of claims 1-4.

Images