Mass data processing method and system for multiple nodes
By distributing massive data segmentation, compression and hash sharding technologies and establishing an index structure, it solves the problem that a single server is difficult to process massive data, improves storage and processing efficiency, and reduces costs and failure risks.
Patent Information
- Application Number
- CN202411115471.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-14
- Publication Date
- 2024-11-26
AI Technical Summary
In a cloud computing environment, the storage capacity and performance of a single server are difficult to meet the increasing demand for massive data, resulting in the risk of data processing speed and system crashes.
By dividing massive data into small pieces, using compression algorithms and hash sharding technology for distributed storage, and establishing an index structure to optimize the storage and processing flow of data.
It significantly improves storage space utilization and data processing efficiency, reduces the risk of single point failure, ensures data integrity and security, and reduces hardware and operational costs.
Smart Images

Figure CN119025037A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of data processing, and in particular to a method and system for processing mass data of multiple nodes. Background Art
[0002] In a cloud computing environment, multiple nodes refer to multiple server nodes deployed in a network, which can be physical servers or virtual machines, working together to provide services. Each node has the ability to process and store data, and multiple nodes are connected to each other through the network to form a distributed computing environment.
[0003] Each node is constantly generating, processing and storing data, which may come from a variety of sources such as user requests, system logs, sensor readings, etc. In a distributed computing environment, in order to ensure data consistency and availability, data is usually stored and backed up redundantly, which also increases the total amount of data.
[0004] Traditional technologies mainly use centralized storage and computing to process massive amounts of data, that is, all data is stored on one or more large central servers, and all computing tasks are completed on these servers. However, as the amount of data continues to increase, the storage capacity and performance of a single server are difficult to meet demand. When the amount of data reaches a certain scale, the processing power and storage space of the server will become a bottleneck, resulting in a decrease in data processing speed and even system crashes.
[0005] To this end, the present invention proposes a method and system for processing massive data at multiple nodes. Summary of the invention
[0006] In view of this, the embodiment of the present invention hopes to provide a method and system for processing massive data for multiple nodes to solve or alleviate the technical problems existing in the prior art, that is, how to further optimize the method for processing massive data on the basis of centralized or distributed storage hardware, so as to make fuller use of storage space, and at least provide a beneficial option for this; the technical solution of the present invention is implemented as follows:
[0007] First, a method for processing massive data on multiple nodes:
[0008] 1. Overview:
[0009] The present invention aims to solve the above technical problems. The present invention divides the massive data into multiple smaller data blocks, and then stores them in a distributed manner on multiple servers. The massive data is first compressed using a compression algorithm to reduce the bandwidth requirements for transmitting data, thereby improving the storage and reading efficiency of the data, and then the compressed data is imported into a pre-established index structure to reduce the storage space occupied to a certain extent.
[0010] (II) Technical solution:
[0011] 2.1 Step S1: Data Preparation:
[0012] Collect all the massive data that need to be processed and remove duplicate data to form a data set D' and then unify the format.
[0013] 2.2.1 Step S100, removing duplicate data:
[0014] First sort the data, then traverse the sorted data list and remove adjacent duplicate data. That is:
[0015] 1) Sort the data set;
[0016] 2) Traverse the sorted data set and compare adjacent data items;
[0017] 3) If adjacent data items are identical, remove the duplicate data;
[0018] For a given data set D = {d 1 ,d 2 ,...,d n}, sort D to get D', and then for i = 1 to n-1, perform the following operations:
[0019] [If D′[i]=D′[i+1], remove D′[i+1] from D′];
[0020] 2.2.2 Step S101, data format conversion:
[0021] Convert the dataset D' into JSON format. JSON (JavaScript Object Notation) is a lightweight data exchange format that is easy to read and write, and easy to machine parse and generate:
[0022] 1) Define the JSON format structure as {"id":"123","value":"example"}.
[0023] 2) Traverse the processed data set and convert each data according to the defined JSON structure;
[0024] 3) Save the converted JSON data.
[0025] 2.2 Step S2, data sharding:
[0026] Determine the data shard size based on the total amount of data set D', the storage capacity and processing performance of the server; use hash sharding to split the data into multiple data blocks d that are smaller than the data set D';
[0027] Execute the verification mechanism to verify the correctness of the sharding results, ensuring that each data block d contains correct data and there is no data loss or duplication.
[0028] 2.2.1 Step S200, determine the data shard size:
[0029] The shard size is calculated using the weighted average method based on the total amount of data set D' and the number of available servers. The weighted average method takes into account the performance differences of different servers and allocates a larger amount of data to servers with higher performance. The shard size S is set so that the amount of data in each shard is close to but does not exceed the optimal processing capacity of the server.
[0030] Given a dataset D', the number of servers M, the optimal amount of data to be processed by each server B i (i=1,2,...,M), then the shard size S determined by the weighted average method is:
[0031]
[0032] 2.2.2 Step S201, perform hash sharding:
[0033] A hash function h(x) is used to apply a hash function to each data item in the data set D' to obtain a hash value; where x is the keyword or identifier of the data item.
[0034] The data item is assigned to a specific shard based on the hash value and the preset number of shards N. This assignment is achieved using modular arithmetic, that is, for a data item (x∈D′), the shard index Index(x) to which it is assigned is:
[0035] Index(x) = [h(x) mod N];
[0036] Then similar hash values will be assigned to the same shard, and different hash values will be assigned to different shards, thus achieving uniform distribution of data.
[0037] 2.2.3 Step S202, verify the fragmentation result:
[0038] S2020, count each shard to ensure that the number of data items in the shard is consistent with expectations:
[0039] 1) Initialize a counter array count[], whose length is the number of slices, and each element is initialized to 0;
[0040] 2) Traverse the data items in each shard, and increase the counter of the corresponding shard by 1 each time a data item is encountered;
[0041] 3) After the traversal is complete, check the counter values of each shard to ensure that they match the expected shard size or number of data items.
[0042] S2021, verify that there is no data loss by comparing the original dataset D' and the sharded dataset:
[0043] Method 1 (double pointer): If the data set is ordered, the double pointer method can be used. One pointer traverses the original data set D', and the other pointer traverses the merged result of all shards, and compares whether the data items pointed to by the two pointers are consistent;
[0044] Method 2 (hash comparison): If the dataset is unordered, you can build hash tables for the original dataset D' and the dataset after the shards are merged. Then compare the two hash tables to ensure that they contain the same keys (data items);
[0045] S2022, check whether the data items in each shard are unique and there are no duplicate data items:
[0046] 1) For each shard, initialize an empty hash table uniqueItems;
[0047] 2) Traverse each data item in the shard, and for each data item:
[0048] 2.1) If the data item already exists in the uniqueItems hash table, it means there is a duplicate, and the duplicate item is recorded and processed;
[0049] 2.2) If the data item does not exist in uniqueItems, it is added to the hash table;
[0050] 3) After the traversal is complete, the items in the hash table should be all unique data items in the shard.
[0051] S2023, using a hash table or other data structure to assist the verification process and improve verification efficiency;
[0052] In S2021, a hash table can be constructed to store all data items in the original dataset D'. Then, when traversing the dataset after the shards are merged, each data item can be quickly checked to see if it exists in the hash table of the original dataset;
[0053] Likewise, in step S2022, the hash table is used to quickly detect duplicate data items.
[0054] 2.3 Step S3, data compression:
[0055] Each data block d is compressed to form a compressed data block d', so as to reduce the storage space occupied by the data and the transmission bandwidth requirement.
[0056] 2.3.1 Step S300, data segmentation:
[0057] Dividing the input data block d into multiple small data segments is conducive to parallel processing and local optimization.
[0058] 2.3.2 Step S301, search for repeated strings:
[0059] For each data segment, find and identify repeated string patterns. These repeated patterns are redundant parts of the data and can be compressed to save space.
[0060] 2.3.3 Step S302, encoding:
[0061] By encoding repetitive patterns, the encoded data will take up less storage space.
[0062] 2.3.4 Step S303, output compressed data:
[0063] Finally, the encoded data is output as a compressed data block d'.
[0064] 2.4 Step S4, distributed storage:
[0065] The distribution planning mechanism is executed to determine which servers the compressed data blocks d' will be stored on based on the sharding results and the storage capacity of the servers, and the compressed data blocks are transferred to the corresponding servers for storage.
[0066] Implement integrity verification and processing mechanisms to ensure that data is successfully stored and not corrupted.
[0067] 2.4.1 Step S400, executing the distribution planning mechanism:
[0068] The execution distribution planning mechanism is a load balancing strategy that ensures that data is evenly distributed across servers to avoid overloading some servers. Based on the sharding results and the load balancing strategy, it is determined which servers each compressed data block d' will be stored on.
[0069] Suppose there are M servers, each with a storage capacity of S i (i=1,2,...,M), the compressed data block size is D j (j=1,2,...,M), then the goal of data allocation is to maximize storage efficiency while maintaining load balance, that is:
[0070]
[0071] Among them, x ijis a binary variable indicating whether data block j is stored on server i. This is an optimization problem that can be solved by different algorithms (such as greedy algorithm, linear programming, etc.).
[0072] 2.4.2 Step S401, integrity verification and processing mechanism:
[0073] S4010, generating verification information: in the data compression stage, generating a hash value or checksum for each compressed data block d' as a fingerprint of the data;
[0074] S4011, store verification information: store the verification information together with the data block, or store it in a separate secure location;
[0075] S4012, Verification process: After the data is stored on the server, the server can recalculate the hash value or checksum of the data and compare it with the original verification information:
[0076] Let the hash value of the original data be H original , the hash value calculated from the data stored on the server is H stored , then the verification condition can be expressed as:
[0077]
[0078] If the equality is established, the data is complete and the process goes to S5; otherwise, the process goes to S4013.
[0079] S4013, Handling Corrupted Data:
[0080] The retransmission mechanism is triggered to request the data block to be resent. If the verification fails after multiple retransmissions, the server will mark the data block as corrupted and trigger an alarm or notify the administrator.
[0081] 2.5 Step S5, perform indexing:
[0082] A hash table is used to generate a corresponding index item for each data block d' and store it in the index structure.
[0083] S500, execute another hash function h(d'): map the data block d' to a unique hash value. It can be MD5, SHA-1, etc.;
[0084] S501, index item generation: for each data block d', use the hash function h(d') to calculate its hash value, and use the hash value as the keyword of the index item.
[0085] S502, generating an index structure: generating an index structure from a hash table or a key-value pair storage structure;
[0086] Hash collisions are inevitable. When two different data blocks are mapped to the same hash value, a conflict resolution strategy needs to be adopted, such as the chain address method, the open address method, etc.
[0087] S503, forming a storage index item: using the hash value of each data block d' as a key and the location information of the data block (such as the address and port of the storage server, the specific location of the data block on the server, etc.) as a value, and storing them in the index structure:
[0088] For each data block (d i ′∈D′), calculate its hash value h(d i '), and (h(d i '), location(d i ')) is stored in the index structure I, where location (d i ') represents data block d i 'Storage location information.
[0089] In this way, in the subsequent data retrieval process, the location of the required data block can be quickly located by querying the index structure I.
[0090] (III) Mechanism for solving technical problems:
[0091] The principle of this solution is to achieve efficient data processing and storage through key steps such as data preparation, data sharding, data compression, distributed storage and execution indexing. Each step is optimized for specific problems in massive data processing, thereby improving the utilization of storage space.
[0092] 3.1 The core idea of the solution:
[0093] (1) Data preparation: By removing duplicate data and unifying the data format, the uniqueness and consistency of stored data are ensured, avoiding the waste of storage space.
[0094] (2) Data Sharding: Sharding is performed based on the total amount of data sets and server capabilities to ensure balanced distribution and efficient processing of data. Hash sharding technology ensures the uniformity of data distribution, while the verification mechanism ensures the integrity and accuracy of the data.
[0095] (3) Data compression: Advanced compression algorithms are used to compress data, significantly reducing the storage space occupied by the data while maintaining the integrity and accessibility of the data.
[0096] (4) Distributed storage: The load balancing strategy evenly distributes data across servers, improving storage efficiency and data access speed. The integrity verification mechanism ensures data integrity and security.
[0097] (5) Execution indexing: By establishing an efficient index structure, fast data retrieval and location are achieved, while optimizing the use of storage space.
[0098] 3.2 Core Mechanism for Solving Technical Problems
[0099] (1) Data deduplication and format conversion mechanism: In the data preparation stage, duplicate data is removed through sorting and comparison, reducing unnecessary storage overhead. The data format is unified into JSON to facilitate subsequent processing and analysis.
[0100] (2) Data sharding and verification mechanism: Hash sharding is used to evenly divide data into small blocks for parallel processing and storage. The verification mechanism ensures the correctness of the sharding results through statistics, comparison, and inspection.
[0101] (3) Data compression mechanism: Using efficient compression algorithms to compress data reduces storage space usage. At the same time, steps such as data segmentation, repeated pattern recognition, and encoding during the compression process are all aimed at improving compression efficiency and storage space utilization.
[0102] (4) Distributed storage and integrity verification mechanism: Data storage planning is carried out according to server storage capacity and load balancing strategy to ensure efficient storage and access of data. The integrity verification mechanism ensures the integrity and security of data by generating verification information, storing and comparing hash values or checksums.
[0103] (5) Index mechanism: By establishing index structures such as hash tables, fast data retrieval and location are achieved. At the same time, optimizing the index structure also reduces the additional storage space occupied.
[0104] Second, a massive data processing system for multiple nodes:
[0105] like Figure 2 As shown, the system includes a processor and a memory connected to the processor, wherein program instructions are stored in the memory, and when the program instructions are executed by the processor, the processor executes the mass data processing method as described above.
[0106] Compared with the prior art, the present invention has the following beneficial effects:
[0107] 1. Storage space optimization: The present invention significantly reduces the storage space occupied by data through data compression technology. Data sharding technology allows data to be stored more compactly on multiple nodes, avoiding excessive storage pressure on a single node and further improving the utilization of storage space.
[0108] 2. Improved processing efficiency: The data sharding of the present invention enables data to be processed in parallel, greatly improving the speed and efficiency of data processing. By establishing an efficient index structure through the present invention, the data retrieval speed is significantly improved, thereby speeding up the overall process of data processing.
[0109] 3. Data security and integrity: The distributed storage and data sharding technology of the present invention increases data security, because data is stored in multiple nodes, reducing the risk of data loss due to single point failure. The integrity verification mechanism ensures the integrity of the data through hash values or checksums, preventing data from being tampered with or damaged during transmission or storage.
[0110] 4. Scalability and flexibility: The present invention has good scalability. When the amount of data increases, more nodes can be easily added to expand the storage capacity and processing power. The data sharding and distributed storage strategies enable the system to flexibly respond to different data processing requirements, whether adding, deleting or modifying data, it can be done efficiently.
[0111] 5. Cost reduction: By optimizing storage space and improving processing efficiency, the solution of the present invention reduces hardware costs and maintenance costs. Since data can be processed in parallel, processing time is reduced, thereby reducing operating costs. BRIEF DESCRIPTION OF THE DRAWINGS
[0112] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0113] Figure 1 It is a schematic diagram of the method flow of the present invention;
[0114] Figure 2 It is a schematic diagram of the system composition of the present invention;
[0115] Figure 3 is a schematic diagram of an implementation of step S202 of the present invention;
[0116] Figure 4 Schematic diagram of an implementation of step S4 of the present invention;
[0117] Figure 5 It is a schematic diagram of an implementation of step S5 of the present invention. DETAILED DESCRIPTION
[0118] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the accompanying drawings. In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below;
[0119] It should be noted that the various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same or similar parts between the various embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description.
[0120] Example: See Figure 1 This embodiment discloses a method for processing massive data for multiple nodes, which includes the following steps S1 to S5.
[0121] In this embodiment, regarding step S1: data preparation: collect all the massive data that need to be processed and remove duplicate data to form a data set D' and then unify the format.
[0122] Specifically, step S100, removing duplicate data: first sort the data, then traverse the sorted data list, and remove adjacent duplicate data. That is:
[0123] 1) Sort the data set; the sorting algorithm can be selected according to the characteristics and scale of the data, such as quick sort, merge sort, etc.
[0124] 2) After sorting is completed, adjacent duplicate data items can be easily identified by traversing the sorted data set D'.
[0125] 3) During the traversal process, if it is detected that adjacent data items are the same (ie, D'[i] = D'[i+1]), the duplicate data item (D'[i+1]) is removed from the data set. This process continues until the entire data set is traversed to ensure that all duplicate items are removed.
[0126] For a given data set D = {d 1 ,d 2 ,...,d n}, sort D to get D', and then for i = 1 to n-1, perform the following operations:
[0127] [If D′[i]=D′[i+1], remove D′[i+1] from D′];
[0128] Specifically, step S101, data format conversion: convert the data set D' into JSON format. JSON (JavaScript Object Notation) is a lightweight data exchange format that is easy to read and write, and easy to machine parse and generate:
[0129] 1) Define the JSON format structure as {"id":"123","value":"example"}.
[0130] 2) Traverse the processed data set and convert each data according to the defined JSON structure;
[0131] 3) Save the converted JSON data.
[0132] In this embodiment, regarding step S2, data sharding: determining the data shard size according to the total amount of the data set D', the storage capacity and processing performance of the server; using hash sharding to divide the data into multiple data blocks d that are smaller than the data set D';
[0133] Execute the verification mechanism to verify the correctness of the sharding results, ensuring that each data block d contains correct data and there is no data loss or duplication.
[0134] Specifically, step S200 determines the data shard size: according to the total amount of data set D' and the number of available servers, the shard size is calculated using the weighted average method. The weighted average method takes into account the performance differences of different servers and allocates a larger amount of data to servers with higher performance. The shard size S is set so that the amount of data in each shard is close to but does not exceed the optimal processing capacity of the server.
[0135] Given a dataset D', the number of servers M, the optimal amount of data to be processed by each server B i (i=1,2,...,M), then the shard size S determined by the weighted average method is:
[0136]
[0137] This size is designed to ensure that each server gets the amount of data that matches its processing capacity.
[0138] Specifically, step S201, performs hash sharding: uses a hash function h(x) to apply a hash function to each data item in the data set D' to obtain a hash value; wherein x is a keyword or identifier of the data item.
[0139] The data item is assigned to a specific shard based on the hash value and the preset number of shards N. This assignment is achieved using modular arithmetic, that is, for a data item (x∈D′), the shard index Index(x) to which it is assigned is:
[0140] Index(x) = [h(x) mod N];
[0141] This distribution method ensures that similar hash values are assigned to the same shard, while different hash values are assigned to different shards, thus achieving uniform distribution of data.
[0142] For details, please refer to Figure 3 , step S202, verify the fragmentation result:
[0143] Step S202: Verify the sharding result: This step is intended to ensure the correctness and integrity of the sharding process, and the sharding result is verified through multiple sub-steps.
[0144] S2020: Shard data statistics: Initialize a counter array to record the number of data items in each shard.
[0145] Iterate over each shard and update the counter, and finally check whether the counter value matches the expected value to ensure that the shard size meets the expected value.
[0146] S2021: Data loss verification: If the dataset is in order, use the double pointer method to compare the original dataset and the dataset after merging the shards. If the dataset is out of order, build a hash table to compare the two datasets to ensure that there is no data loss.
[0147] S2022: Data duplication check: For each shard, a hash table is used to record the data items that have appeared to detect and process duplicate data items. Ensure that the data items in each shard are unique.
[0148] S2023: Auxiliary Verification Structures: Use data structures such as hash tables to speed up the verification process, especially when checking for data loss and data duplication.
[0149] For example, in step S2021, a hash table may be constructed to store all data items in the original data set D'. Then, when traversing the data set after the shards are merged, each data item may be quickly checked to see if it exists in the hash table of the original data set; similarly, in step S2022, the hash table is used to quickly detect duplicate data items.
[0150] It is understandable that by splitting a data set into multiple small pieces, the processing power of multiple servers can be fully utilized to achieve parallel processing, thereby improving overall processing efficiency. Hash sharding uses hash functions and modulo operations to shard data, ensuring uniform distribution of data and avoiding data skew problems. The sharding verification mechanism ensures the correctness and integrity of the sharding results through a multi-step verification process, preventing problems such as data loss or duplication. These verification steps use data structures such as hash tables to improve verification efficiency.
[0151] In this embodiment, regarding step S3, data compression: compressing each data block d and forming a compressed data block d', so as to reduce the storage space occupied by the data and the transmission bandwidth requirement. The method flow includes:
[0152] (1) Step S300: Data segmentation: First, the input data block d is segmented into multiple small data segments. This approach has several advantages: it supports parallel processing because different data segments can be compressed on different processing units at the same time; in addition, it also helps to achieve local optimization because some compression algorithms may be more effective when processing smaller data segments.
[0153] (2) Step S301: Find repeated character strings: Next, for each data segment, the algorithm will find and identify repeated character string patterns. These repeated patterns are a manifestation of data redundancy. By compressing these patterns, storage space can be significantly saved.
[0154] (3) Step S302: Encoding: Once the repeated patterns are identified, the next step is to encode these patterns. The purpose of encoding is to represent the original data with a shorter bit sequence, thereby reducing storage requirements.
[0155] For example, common repeating patterns are assigned shorter codes, while uncommon or unique data is assigned longer codes. In this way, the compression algorithm is able to use storage space more efficiently.
[0156] (4) Step S303: Output compressed data: Finally, the encoded data is output as a compressed data block d'. This compressed data block now occupies much less storage space than the original data block d, thereby saving storage space and reducing the bandwidth required for transmission. The compressed data block d' can now be stored or transmitted, and then decompressed to restore the original data when needed.
[0157] Understandably, the main benefit of data compression is the reduction of storage space and transmission bandwidth requirements. This is achieved by identifying and eliminating redundant information in the data. Compression algorithms are able to find and compress repeated patterns so that fewer bits are used to represent the same information. Not only does this save storage space, it also reduces the bandwidth required to transmit data across the network, thereby reducing costs and increasing efficiency. In addition, by segmenting the data, the compression process can be more easily parallelized, which speeds up the process. Local optimization is also possible, as the compression algorithm can be optimized for smaller data segments, further improving compression efficiency.
[0158] In this example, see Figure 4 ,Regarding step S4, distributed storage: Execute the distribution planning mechanism, determine which servers the compressed data block d' will be stored on according to the sharding results and the storage capacity of the server, and transfer the compressed data block to the corresponding server for storage. At the same time, perform integrity verification and processing mechanisms to ensure that the data is successfully stored and not damaged.
[0159] Specifically, step S400, executing the distribution planning mechanism: executing the distribution planning mechanism is a load balancing strategy, the purpose of which is to ensure that data is evenly distributed among the servers, thereby avoiding the situation where some servers are overloaded while other servers are idle. Through reasonable data allocation, the storage efficiency and the overall performance of the system can be maximized. That is, ensure that data is evenly distributed among the servers, avoiding the overload of some servers. According to the sharding results and the load balancing strategy, determine which servers to store each compressed data block d':
[0160] Suppose there are M servers, each with a storage capacity of S i (i=1,2,...,M), the compressed data block size is D j (j=1,2,...,M), then the goal of data allocation is to maximize storage efficiency while maintaining load balance, that is:
[0161]
[0162] Among them, x ij is a binary variable indicating whether data block j is stored on server i. This is an optimization problem that can be solved by different algorithms (such as greedy algorithm, linear programming, etc.).
[0163] Specifically, step S401, integrity verification and processing mechanism:
[0164] S4010, generating verification information: in the data compression stage, generating a hash value or checksum for each compressed data block d' as a fingerprint of the data;
[0165] S4011, store verification information: store the verification information together with the data block, or store it in a separate secure location;
[0166] S4012, Verification process: After the data is stored on the server, the server can recalculate the hash value or checksum of the data and compare it with the original verification information:
[0167] Let the hash value of the original data be H original , the hash value calculated from the data stored on the server is H stored , then the verification condition can be expressed as:
[0168]
[0169] If the equality is established, the data is complete and the process goes to S5; otherwise, the process goes to S4013.
[0170] S4013, Handling Corrupted Data:
[0171] The retransmission mechanism is triggered to request the data block to be resent. If the verification fails after multiple retransmissions, the server will mark the data block as corrupted and trigger an alarm or notify the administrator.
[0172] In this example, see Figure 5 , Regarding step S5, perform indexing: use the hash table to generate a corresponding index item for each data block d' and store it in the index structure.
[0173] S500, execute another hash function h(d'): map the data block d' to a unique hash value. It can be MD5, SHA-1, etc.;
[0174] S501, index item generation: for each data block d', use the hash function h(d') to calculate its hash value, which will be used as the keyword of the index item to uniquely identify the data block;
[0175] S502, generating an index structure: generating an index structure from a hash table or a key-value pair storage structure;
[0176] It is understandable that hash collisions are inevitable. When two different data blocks are mapped to the same hash value, conventional conflict resolution strategies need to be adopted, such as chain address method, open address method, etc.
[0177] S503, forming a storage index item: using the hash value of each data block d' as a key and the location information of the data block (such as the address and port of the storage server, the specific location of the data block on the server, etc.) as a value, and storing them in the index structure:
[0178] For each data block (d i ′∈D′), calculate its hash value h(d i '), and (h(d i '), location(d i ')) is stored in the index structure I, where location (d i ') represents data block d i 'Storage location information.
[0179] In this way, in the subsequent data retrieval process, the location of the required data block can be quickly located by querying the index structure I.
[0180] It is understandable that by establishing an index, we can quickly locate the storage location of the data block directly through the hash value of the data block without traversing the entire data set. This greatly reduces the retrieval time and improves the efficiency of data processing.
[0181] Although the index structure itself takes up a certain amount of storage space, this space overhead is relatively small compared to the entire data set. Moreover, the index can more effectively manage and utilize storage space, avoiding data redundancy and repeated storage. The index structure also makes data addition, deletion, modification and query operations more convenient and efficient. When a data block needs to be updated or deleted, we only need to perform the corresponding operation in the index structure without making large-scale modifications to the entire data set.
[0182] All the above embodiments only express the implementation methods of the relevant practical applications of the present invention, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be based on the attached claims.
[0183] For those skilled in the art, it can be further appreciated that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.
[0184] At the same time, those skilled in the art can understand that all or part of the processes in all the above-mentioned embodiments can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media provided in this application and used in the embodiments may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (SSRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).
Claims
1. A method for processing massive data on multiple nodes, characterized in that: The steps include: S1, data preparation: collect all the massive data that need to be processed and remove duplicate data to form a unified format after forming a data set D'; S2, data sharding: determine the data shard size based on the total amount of data set D', the storage capacity and processing performance of the server; use hash sharding to divide the data into multiple data blocks d that are smaller than the data set D'; Execute the verification mechanism to verify the correctness of the sharding results; S3, data compression: compress each data block d and form a compressed data block d' to reduce the storage space occupied by the data and the transmission bandwidth requirement; S4, distributed storage: execute the distribution planning mechanism to determine which servers to store the compressed data block d' on based on the sharding results and the storage capacity of the server; Implement integrity verification and processing mechanisms to ensure that data is successfully stored and not corrupted; S5, perform indexing: use the hash table to generate a corresponding index item for each data block d' and store it in the index structure.
2. The method for processing massive data according to claim 1, characterized in that: In S1, the removal and unification methods are: S100, removing duplicate data: first sort the data, then traverse the sorted data list to remove adjacent duplicate data; S101, data format conversion: convert the data set D' into JSON format.
3. The method for processing massive data according to claim 1, characterized in that: In S2, the determination method is: according to the total amount of data set D' and the number of available servers, the shard size is calculated using a weighted average method, and the shard size S is set so that the amount of data in each shard is close to but does not exceed the optimal processing capacity of the server: Where M is the number of servers, B i is the optimal amount of data to be processed by each server.
4. The method for processing massive data according to claim 1, characterized in that: In S2, the hash sharding method is to use a hash function h(x) to apply a hash function to each data item in the data set D' to obtain a hash value; wherein x is a keyword or identifier of the data item, and the data item is allocated to a specific shard according to the hash value and the preset number of shards N: Index(x)=[h(x)mod N].
5. The method for processing massive data according to claim 4, characterized in that: In S2, the verification mechanism is: S2020, count each shard to ensure that the number of data items in the shard is consistent with expectations; S2021, verify that there is no data loss by comparing the original dataset D' and the sharded dataset; S2022, check whether the data items in each shard are unique and there are no duplicate data items; S2023, use hash tables or other data structures to improve verification efficiency.
6. The method for processing massive data according to claim 1, characterized in that: In S3, the compression method is: Divide the input data block d into multiple small data segments; For each data segment, find and identify repeated string patterns; Encode repetitive patterns; The encoded data is output as a compressed data block d'.
7. The method for processing massive data according to claim 1, characterized in that: In S4, the distribution planning mechanism determines which servers to store each compressed data block d' on according to the sharding result and the load balancing strategy: For M servers, the storage capacity of each server is S i , the compressed data block size is D j , then the goal of data allocation is to maximize storage efficiency while maintaining load balance: Among them, x ij is a binary variable indicating whether data block j is stored on server i.
8. The method for processing massive data according to claim 1, characterized in that: In S4, the integrity verification and processing mechanism is: S4010, generating verification information: generating a hash value or a checksum for each compressed data block d'; S4011, storing verification information: storing the verification information together with the data block; S4012, Verification process: After the data is stored on the server, the server recalculates the hash value or checksum of the data and compares it with the original verification information: The hash value of the original data is H original , the hash value calculated from the data stored on the server is H stored : If the equality sign holds, the data is complete and the process goes to S5; Otherwise, execute S4013; S4013, Handling Corrupted Data: Trigger the retransmission mechanism and request to resend the data block.
9. The method for processing massive data according to claim 1, characterized in that: In S5, the generating method is: S500, executing a hash function h(d') to map the data block d' to a unique hash value; S501, for each data block d', use the hash function h(d') to calculate its hash value, and use the hash value as the keyword of the index item; S502, generating an index structure: generating an index structure from a hash table or a key-value pair storage structure; S503, forming a storage index item: using the hash value of each data block d' as a key and the location information of the data block as a value, and storing them in the index structure.
10. A mass data processing system for multiple nodes, characterized in that: The system includes a processor and a memory connected to the processor, wherein program instructions are stored in the memory, and when the program instructions are executed by the processor, the processor executes the mass data processing method according to any one of claims 1 to 9.
Citation Information
Cited By
Data perspective table data expansion method and device
CN119202066A
