Data storage method and device based on distributed system, equipment and storage medium

Abstract

The application relates to the computer field and discloses a data storage method and device based on a distributed system, equipment and a storage medium. The method comprises the following steps: creating a data table and a hash chain table, performing partition processing and bucket processing on the data table, and establishing an association mapping relationship between the data table and the hash chain table according to the partition processing result and the bucket processing result; receiving a data storage request and obtaining corresponding to-be-processed data and the primary key identifiers of the to-be-processed data; in response to the data storage request, obtaining the number of buckets pre-stored in the hash chain table, taking the number of buckets as a benchmark, performing hash bucketing on the to-be-processed data according to the primary key identifiers, and storing the to-be-processed data according to the hash bucketing result; and counting the data files of each partition in the data table, adjusting the bucketing strategy according to the counting result, and updating the hash chain table. Through the above method, the data bucketing strategy can be dynamically adjusted, the data processing efficiency is improved, the storage management pressure is reduced, and the data maintenance cost is saved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a data storage method, device, equipment and storage medium based on a distributed system. Background Art

[0002] With the rapid development of big data and cloud technologies, data volumes are growing rapidly. While traditional storage technologies are mature, offer good performance, and offer high availability, they still suffer from poor scalability, high costs, single points of failure, and performance bottlenecks when faced with massive amounts of data. To meet this demand, distributed storage technologies have emerged.

[0003] In distributed storage systems, data bucketing is a further division of data based on the established data partitions in the big data field. Currently, data bucketing in the big data field is a very difficult problem to handle. The number of buckets for data must consider both the continuous growth of data due to business development, avoiding a situation where too few buckets could lead to data skew within a single bucket, and the small amount of data in a single partition, avoiding a situation where too many buckets could lead to an excessive number of small files within a single partition.

[0004] The above problems are particularly evident in batch processing scenarios for massive data, especially in scenarios where business data is dynamically changing. If the number of buckets is too small, the improvement in data reading, writing, and processing efficiency will not be significant enough. However, if the number of buckets is too large, it will cause great storage pressure and management costs on the file system. Regularly rearranging the buckets of the entire data based on the latest data distribution status is an extremely computationally expensive operation. Summary of the Invention

[0005] The present invention provides a data storage method, device, equipment and storage medium based on a distributed system, which can dynamically adjust the data bucketing strategy, improve data processing efficiency, reduce storage management pressure, and save data maintenance costs.

[0006] To solve the above technical problems, the present invention adopts a technical solution: providing a data storage method based on a distributed system, comprising:

[0007] Creating a data table and a hash chain table in the distributed system, performing partitioning and bucketing on the data table, and establishing an association mapping relationship between the data table and the hash chain table according to the partitioning and bucketing results;

[0008] Receive a data storage request and obtain the to-be-processed data corresponding to the data storage request and a primary key identifier of each of the to-be-processed data;

[0009] In response to the data storage request, obtaining the number of buckets pre-stored in the hash chain table, performing hash bucketing on the data to be processed based on the primary key identifier based on the number of buckets, and storing the data to be processed in the data table based on the hash bucketing result;

[0010] Count the data files of each partition in the data table, compare the statistical result with a preset threshold, adjust the bucketing strategy of the data table according to the comparison result, and update the adjustment result to the hash chain table.

[0011] According to one embodiment of the present invention, creating a data table and a hash table in a distributed system, performing partitioning and bucketing on the data table, and establishing an association mapping relationship between the data table and the hash table based on the partitioning and bucketing results includes:

[0012] Creating a data table in the distributed system and obtaining a storage root path of the data table, creating the hash linked table under the storage root path, and associating and binding the hash linked table with the data table;

[0013] Initialize partition processing on the data table to obtain the partition name and partition quantity of each partition;

[0014] Initialize bucketing for each partition to obtain the number of buckets for each partition;

[0015] The partition name and the number of buckets corresponding to each partition are stored in the hash chain table to form an associated mapping relationship between the data table and the hash chain table.

[0016] According to one embodiment of the present invention, in response to the data storage request, obtaining the number of buckets pre-stored in the hash linked table, performing hash bucketing on the to-be-processed data based on the primary key identifier based on the number of buckets, and storing the to-be-processed data in the data table based on the hash bucketing result includes:

[0017] In response to the data storage request, obtaining the number of buckets pre-stored in the hash table;

[0018] Based on the number of buckets, the primary key identifiers are hashed and converted to obtain hash values ​​and storage paths corresponding to the primary key identifiers.

[0019] The data to be processed is stored in the data table according to the hash value and the corresponding storage path.

[0020] According to one embodiment of the present invention, counting the data files of each partition in the data table, comparing the statistical result with a preset threshold, adjusting the bucketing strategy of the data table according to the comparison result, and updating the adjustment result to the hash linked table includes:

[0021] Obtaining attribute information of data files in each partition, and calculating cumulative values ​​of sizes of data files contained in the same partition based on the attribute information;

[0022] The accumulated value is compared with a preset threshold, the bucketing strategy of the data table is adjusted according to the comparison result, and the adjustment result is updated to the hash chain table.

[0023] According to one embodiment of the present invention, the preset threshold includes a bucket lower limit threshold, and comparing the accumulated value with the preset threshold, adjusting the bucket strategy of the data table according to the comparison result, and updating the adjustment result to the hash linked table includes:

[0024] Comparing the accumulated value with the bucket lower threshold;

[0025] If the accumulated value is less than the bucket lower limit threshold, a new number of buckets is calculated according to the first bucketing strategy, and the new number of buckets is updated to the hash chain table.

[0026] According to one embodiment of the present invention, the preset threshold includes a bucket upper threshold, and comparing the accumulated value with the preset threshold, adjusting the bucketing strategy of the data table according to the comparison result, and updating the adjustment result to the hash linked table includes:

[0027] Comparing the accumulated value with the bucket upper threshold;

[0028] If the accumulated value is greater than the bucket upper limit threshold, a new number of buckets is calculated according to the second bucketing strategy, and the new number of buckets is updated to the hash chain table.

[0029] According to one embodiment of the present invention, the first bucketing strategy is to calculate the ratio of the cumulative value to the bucket lower limit threshold, and determine the new number of buckets based on the ratio calculation result; the second bucketing strategy is to calculate the ratio of the cumulative value to the bucket upper limit threshold, and determine the new number of buckets based on the ratio calculation result.

[0030] To solve the above technical problems, another technical solution adopted by the present invention is to provide a data storage device based on a distributed system, comprising:

[0031] A creation module is used to create a data table and a hash table in the distributed system, perform partitioning and bucketing on the data table, and establish an association mapping relationship between the data table and the hash table based on the partitioning and bucketing results;

[0032] A receiving module, configured to receive a data storage request and obtain the to-be-processed data corresponding to the data storage request and a primary key identifier of each of the to-be-processed data;

[0033] a response module, configured to, in response to the data storage request, obtain the number of buckets pre-stored in the hash linked table, perform hash bucketing on the data to be processed based on the primary key identifier based on the number of buckets, and store the data to be processed in the data table based on the hash bucketing result;

[0034] The adjustment module is used to count the data files of each partition in the data table, compare the statistical results with a preset threshold, adjust the bucketing strategy of the data table according to the comparison result, and update the adjustment result to the hash chain table.

[0035] In order to solve the above technical problems, another technical solution adopted by the present invention is: to provide a computer device, including: a memory, a processor and a computer program stored in the memory and capable of running on the processor, and when the processor executes the computer program, the data storage method based on the distributed system is implemented.

[0036] In order to solve the above technical problems, another technical solution adopted by the present invention is: providing a computer storage medium on which a computer program is stored, and when the computer program is executed by a processor, the above data storage method based on a distributed system is implemented.

[0037] The beneficial effects of the present invention are: on the one hand, by establishing an associative mapping relationship between a data table and a hash table, and using an extensible hash table to store dynamically changing bucket information, data peaks can be effectively reduced, data skew can be avoided, and thus data processing efficiency can be improved; on the other hand, by storing the data to be processed in the data table, the data files of each partition in the data table are counted, the statistical results are compared with the preset threshold, and the bucketing strategy of the data table is adjusted according to the comparison result, thereby realizing dynamic adjustment of the data bucketing strategy, which can reduce storage management pressure and save data maintenance costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 This is a flow chart of a data storage method based on a distributed system according to an embodiment of the present invention;

[0039] Figure 2 1 is a flow chart of step S101 in the data storage method based on a distributed system according to an embodiment of the present invention;

[0040] Figure 3 is a schematic diagram of the architecture of a distributed system according to an embodiment of the present invention;

[0041] Figure 4 1 is a flow chart of step S104 in the data storage method based on a distributed system according to an embodiment of the present invention;

[0042] Figure 5 2 is a schematic diagram of the structure of a data storage device based on a distributed system according to an embodiment of the present invention;

[0043] Figure 6 is a schematic structural diagram of a computer device according to an embodiment of the present invention;

[0044] Figure 7 It is a schematic diagram of the structure of a computer storage medium according to an embodiment of the present invention. DETAILED DESCRIPTION

[0045] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0046] The terms "first", "second", and "third" in the present invention are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, a feature defined as "first", "second", and "third" may explicitly or implicitly include at least one of the features. In the description of the present invention, "multiple" means at least two, for example, two, three, etc., unless otherwise clearly and specifically defined. All directional indications in the embodiments of the present invention (such as up, down, left, right, front, back, etc.) are only used to explain the relative positional relationship, movement, etc. between the components under a specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication will also change accordingly. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units that are not listed, or may optionally include other steps or units that are inherent to these processes, methods, products, or devices.

[0047] Reference herein to an "embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment may be included in at least one embodiment of the present invention. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute a separate or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0048] Figure 1 It is a flow chart of a data storage method based on a distributed system according to an embodiment of the present invention. It should be noted that if there is substantially the same result, the method of the present invention is not based on Figure 1 The process sequence shown is limited. Figure 1 As shown, the method includes the steps of:

[0049] Step S101: creating a data table and a hash linked list in a distributed system, performing partition processing and bucket processing on the data table, and establishing an association mapping relationship between the data table and the hash linked list according to the partition processing results and the bucket processing results.

[0050] In step S101, the data table can store massive amounts of data on the order of tens of billions to hundreds of billions. The hash table is used to store the bucket information of the data table, supporting elastic expansion and data broadcasting. For a data table storing massive amounts of data on the order of hundreds of billions, the maximum size of the bucket information of the data table is only in the range of 1E4 to 1E5. Therefore, storing these bucket information will not bring storage burden to the hash table, nor will it affect the computing performance of the hash table.

[0051] In one possible embodiment, see Figure 2 , step S101 further includes the following steps:

[0052] Step S201: create a data table in a distributed system and obtain a storage root path of the data table, create a hash chain table under the storage root path, and associate and bind the hash chain table with the data table.

[0053] Specifically, the data table created by initialization is an empty table, and the hash table created by initialization is also an empty table. The hash table is associated and bound with the data table to facilitate the hash table to store dynamically changing bucket information.

[0054] Step S202: Initialize the partitioning process for the data table to obtain the partition name and the number of partitions for each partition.

[0055] Specifically, by initializing the partition processing of the data table, the partition name and number of partitions can be obtained. For example, Figure 3For example, a data table is initialized for partitioning to obtain three partitions, each of which is named Partition 1, Partition 2, and Partition 3. The partition names and number of the partitions in the initialization partitioning process of this embodiment can be set by the user or developer.

[0056] Step S203: Initialize bucketing processing for each partition to obtain the number of buckets for each partition.

[0057] Specifically, the number of buckets for the initialization bucketing process of this embodiment can be set by the user or developer. Figure 3 For example, Partition 1 is initialized and bucketed to obtain 3 buckets, Partition 2 is initialized and bucketed to obtain 1 bucket, and Partition 3 is initialized and bucketed to obtain 2 buckets.

[0058] Step S204: storing the partition name and the number of buckets corresponding to each partition in a hash table, forming an association mapping relationship between the data table and the hash table.

[0059] Specifically, the hash table stores the partition name and bucket number corresponding to each partition in the format of "partition name: bucket number." This means that the partitions and buckets in the data table correspond one-to-one to the bucket information stored in the hash table. This embodiment establishes an associative mapping relationship between the data table and the hash table. A partition and bucket in the hash table uniquely identify the data to be processed by a distributed node task in the data table. By utilizing an extensible hash table to store dynamically changing bucket information, data spikes can be effectively reduced, data skew can be avoided, and data processing efficiency can be improved. Furthermore, the low-level bucket information stored in the hash table can be quickly broadcast to any distributed node without incurring additional storage and computing costs.

[0060] Step S102: receiving a data storage request and obtaining the to-be-processed data corresponding to the data storage request and the primary key identifier of each to-be-processed data.

[0061] In step S102, the data to be processed is an object to be stored in a data table, which may be batch data, and the data volume may reach tens to hundreds of billions. The primary key identifier is a unique identifier of the data to ensure the uniqueness of the data.

[0062] Step S103: In response to the data storage request, obtain the number of buckets pre-stored in the hash chain table, use the number of buckets as a benchmark, hash bucket the data to be processed according to the primary key identifier, and store the data to be processed in the data table according to the hash bucket result.

[0063] In step S103, if the data storage request is the first request, the number of buckets is the initialized default number of buckets; if the data storage request is not the first request, the number of buckets is the updated number of buckets from the last data storage. In response to the data storage request, this embodiment obtains the number of buckets pre-stored in the hash table; based on the number of buckets, a hash value conversion process is performed on the primary key identifier based on a hash algorithm to obtain a hash value and a corresponding storage path corresponding to each primary key identifier; and the data to be processed is stored in the data table based on the hash value and the corresponding storage path. The data to be processed is stored in the data table to form a data file.

[0064] Step S104: Count the data files of each partition in the data table, compare the statistical result with a preset threshold, adjust the bucketing strategy of the data table according to the comparison result, and update the adjustment result to the hash chain table.

[0065] In step S104, each partition includes one or more buckets, each bucket stores one or more data files, and different data files carry different attribute information, such as the size and number of data files. Figure 4 , step S104 further includes the following steps:

[0066] Step S401: Acquire attribute information of data files of each partition, and calculate the cumulative value of the size of data files contained in the same partition based on the attribute information.

[0067] Specifically, the attribute information includes the size and number of data files. The cumulative value is the cumulative result of the sizes of the data files contained in each bucket in the same partition.

[0068] Step S402: Compare the accumulated value with a preset threshold, adjust the bucketing strategy of the data table according to the comparison result, and update the adjustment result into the hash linked table.

[0069] This embodiment compares the cumulative value of the size of each data file in the same partition with a preset threshold value to check whether the bucketing is reasonable. If the bucketing is unreasonable, the number of buckets is readjusted according to the comparison result to obtain the optimal bucketing result. The preset threshold value of this embodiment can be understood as the basis for bucketing, such as the upper and lower threshold values ​​of buckets. In this way, the bucket status can be dynamically checked and the number of buckets can be dynamically adjusted. The details of the underlying storage can be shielded from upper-level users. Users do not need to make accurate estimates of the number of buckets for the data, nor do they need to consider the possibility that the rapid growth of business data may lead to data skew, which greatly improves the user experience.

[0070] Furthermore, in an achievable embodiment, the preset thresholds include a bucket lower threshold and a bucket upper threshold, and step S402 specifically compares the cumulative value with the bucket lower threshold; if the cumulative value is less than the bucket lower threshold, the new number of buckets is calculated according to the first bucketing strategy, and the new number of buckets is updated to the hash list. In this embodiment, if the cumulative value is less than the bucket lower threshold, it means that there are too many small data files and too many buckets, which may bring storage pressure and management costs to the system, and the number of buckets needs to be reduced. The first bucketing strategy is to calculate the ratio of the cumulative value to the bucket lower threshold, and determine the new number of buckets based on the ratio calculation result. Specifically, the ratio calculation result is rounded back to the new number of buckets after the partition is re-planned.

[0071] Furthermore, in another feasible embodiment, step S402 specifically compares the cumulative value with the bucket upper limit threshold; if the cumulative value is greater than the bucket upper limit threshold, the new number of buckets is calculated according to the second bucketing strategy, and the new number of buckets is updated to the hash list. In this embodiment, if the cumulative value is greater than the bucket upper limit threshold, it means that there are too many big data files and the number of buckets is too small, which may affect data reading and writing and processing efficiency, and the number of buckets needs to be increased. The second bucketing strategy is to calculate the ratio of the cumulative value to the bucket upper limit threshold, and determine the new number of buckets based on the ratio calculation result. Specifically, the ratio calculation result is rounded back to the new number of buckets after the partition is re-planned.

[0072] The data storage method based on a distributed system in an embodiment of the present invention, on the one hand, establishes an associative mapping relationship between a data table and a hash table, and uses an extensible hash table to store dynamically changing bucket information, which can effectively reduce data peaks and avoid data skew, thereby improving data processing efficiency; on the other hand, after storing the data to be processed in the data table, the data files of each partition in the data table are counted, the statistical results are compared with the preset threshold, and the bucketing strategy of the data table is adjusted according to the comparison results, thereby realizing dynamic adjustment of the data bucketing strategy, which can reduce storage management pressure and save data maintenance costs.

[0073] Figure 5 Schematic diagram of the structure of a data storage device based on a distributed system according to an embodiment of the present invention. Figure 5 As shown, the device 50 includes a creation module 51 , a receiving module 52 , a response module 53 and an adjustment module 54 .

[0074] The creation module 51 is used to create a data table and a hash table in a distributed system, perform partitioning and bucketing on the data table, and establish an association mapping relationship between the data table and the hash table based on the partitioning and bucketing results;

[0075] The receiving module 52 is used to receive a data storage request and obtain the to-be-processed data corresponding to the data storage request and the primary key identifier of each to-be-processed data;

[0076] The response module 53 is used to respond to the data storage request, obtain the number of buckets pre-stored in the hash chain table, use the number of buckets as a basis, hash bucket the data to be processed according to the primary key identifier, and store the data to be processed in the data table according to the hash bucket result;

[0077] The adjustment module 54 is used to count the data files of each partition in the data table, compare the statistical results with the preset threshold, adjust the bucketing strategy of the data table according to the comparison result, and update the adjustment result to the hash chain table.

[0078] Furthermore, the creation module 51 also includes:

[0079] A creation unit is used to create a data table in a distributed system and obtain the storage root path of the data table, create a hash table under the storage root path, and associate and bind the hash table with the data table;

[0080] The partition unit is used to initialize the partition processing of the data table and obtain the partition name and number of partitions;

[0081] Bucketing unit, used to initialize bucketing of each partition and obtain the number of buckets for each partition;

[0082] The association unit is used to store the partition name and the number of buckets corresponding to each partition in the hash table, forming an association mapping relationship between the data table and the hash table.

[0083] Furthermore, the adjustment module 54 further includes:

[0084] A statistical unit, configured to obtain attribute information of data files in each partition and calculate the cumulative size of data files contained in the same partition based on the attribute information;

[0085] The adjustment unit is used to compare the accumulated value with a preset threshold, adjust the bucketing strategy of the data table according to the comparison result, and update the adjustment result to the hash chain table.

[0086] See also Figure 6 , Figure 6 FIG. 1 is a schematic diagram of the structure of a computer device according to an embodiment of the present invention. Figure 6 As shown, the computer device 60 includes a processor 61 and a memory 62 coupled to the processor 61 .

[0087] The memory 62 stores program instructions for implementing the data storage method based on a distributed system described in any of the above embodiments.

[0088] The processor 61 is configured to execute program instructions stored in the memory 62 to store data.

[0089] The processor 61 may also be referred to as a CPU (Central Processing Unit). The processor 61 may be an integrated circuit chip having signal processing capabilities. The processor 61 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component. The general-purpose processor may be a microprocessor or any conventional processor.

[0090] See Figure 7 , Figure 7 Schematic diagram of the structure of the computer storage medium of the embodiment of the present invention. The computer storage medium of the embodiment of the present invention stores a program file 71 that can implement all the above methods, wherein the program file 71 can be stored in the above computer storage medium in the form of a software product, including a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) or a processor to execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned computer storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, or terminal devices such as a computer, a server, a mobile phone, and a tablet.

[0091] In the several embodiments provided by the present invention, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be through some interface, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0092] In addition, the functional units in the various embodiments of the present invention may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0093] The above are merely embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention description and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A data storage method based on a distributed system, characterized in that: include: Creating a data table and a hash chain table in the distributed system, performing partitioning and bucketing on the data table, and establishing an association mapping relationship between the data table and the hash chain table according to the partitioning and bucketing results; Receive a data storage request and obtain the to-be-processed data corresponding to the data storage request and a primary key identifier of each of the to-be-processed data; In response to the data storage request, obtaining the number of buckets pre-stored in the hash chain table, performing hash bucketing on the data to be processed based on the primary key identifier based on the number of buckets, and storing the data to be processed in the data table based on the hash bucketing result; Count the data files of each partition in the data table, compare the statistical result with a preset threshold, adjust the bucketing strategy of the data table according to the comparison result, and update the adjustment result to the hash chain table.

2. The data storage method according to claim 1, wherein: The step of creating a data table and a hash linked table in a distributed system, performing partitioning and bucketing on the data table, and establishing an association mapping relationship between the data table and the hash linked table according to the partitioning and bucketing results includes: Creating a data table in the distributed system and obtaining a storage root path of the data table, creating the hash linked table under the storage root path, and associating and binding the hash linked table with the data table; Initialize partition processing on the data table to obtain the partition name and partition quantity of each partition; Initialize bucketing for each partition to obtain the number of buckets for each partition; The partition name and the number of buckets corresponding to each partition are stored in the hash chain table to form an associated mapping relationship between the data table and the hash chain table.

3. The data storage method according to claim 1, wherein: The step of responding to the data storage request, obtaining the number of buckets pre-stored in the hash linked table, performing hash bucketing on the data to be processed based on the primary key identifier based on the number of buckets, and storing the data to be processed in the data table based on the hash bucketing result includes: In response to the data storage request, obtaining the number of buckets pre-stored in the hash table; Based on the number of buckets, the primary key identifiers are hashed and converted to obtain hash values ​​and storage paths corresponding to the primary key identifiers. The data to be processed is stored in the data table according to the hash value and the corresponding storage path.

4. The data storage method according to claim 1, wherein: The counting of data files in each partition of the data table, comparing the counting result with a preset threshold, adjusting the bucketing strategy of the data table according to the comparison result, and updating the adjustment result to the hash linked table includes: Obtaining attribute information of data files in each partition, and calculating cumulative values ​​of sizes of data files contained in the same partition based on the attribute information; The accumulated value is compared with a preset threshold, the bucketing strategy of the data table is adjusted according to the comparison result, and the adjustment result is updated to the hash chain table.

5. The data storage method according to claim 4, characterized in that: The preset threshold includes a bucket lower threshold, and the comparing the accumulated value with the preset threshold, adjusting the bucket strategy of the data table according to the comparison result, and updating the adjustment result to the hash chain table includes: Comparing the accumulated value with the bucket lower threshold; If the accumulated value is less than the bucket lower limit threshold, a new number of buckets is calculated according to the first bucketing strategy, and the new number of buckets is updated to the hash chain table.

6. The data storage method according to claim 5, characterized in that: The preset threshold includes a bucket upper threshold, and the comparing the accumulated value with the preset threshold, adjusting the bucketing strategy of the data table according to the comparison result, and updating the adjustment result to the hash chain table includes: Comparing the accumulated value with the bucket upper threshold; If the accumulated value is greater than the bucket upper limit threshold, a new number of buckets is calculated according to the second bucketing strategy, and the new number of buckets is updated to the hash chain table.

7. The data storage method according to claim 6, characterized in that: The first bucketing strategy is to calculate the ratio of the cumulative value to the bucket lower limit threshold, and determine the new number of buckets based on the ratio calculation result; the second bucketing strategy is to calculate the ratio of the cumulative value to the bucket upper limit threshold, and determine the new number of buckets based on the ratio calculation result.

8. A data storage device based on a distributed system, characterized in that: include: A creation module is used to create a data table and a hash table in the distributed system, perform partitioning and bucketing on the data table, and establish an association mapping relationship between the data table and the hash table based on the partitioning and bucketing results; A receiving module, configured to receive a data storage request and obtain the to-be-processed data corresponding to the data storage request and a primary key identifier of each of the to-be-processed data; a response module, configured to, in response to the data storage request, obtain the number of buckets pre-stored in the hash linked table, perform hash bucketing on the data to be processed based on the primary key identifier based on the number of buckets, and store the data to be processed in the data table based on the hash bucketing result; The adjustment module is used to count the data files of each partition in the data table, compare the statistical results with a preset threshold, adjust the bucketing strategy of the data table according to the comparison result, and update the adjustment result to the hash chain table.

9. A computer device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the data storage method based on a distributed system as described in any one of claims 1 to 7 is implemented.

10. A computer storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the data storage method based on a distributed system is implemented as described in any one of claims 1 to 7.

Images