A hierarchical hash-based data sharding extraction method and system

Abstract

The application relates to the technical field of big data, in particular to a data sharding extraction method and system based on hierarchical hashing. The method can analyze the characteristics of a sharding object, i.e. a data key value of a string type, in at least two databases participating in distributed storage of big data, generate a disturbance factor vector for hybrid bypass and hierarchical mapping, and based on the elements in the vector, perform hybrid disturbance and hierarchical mapping on the hash result of the data key value to obtain a sharding result, thereby realizing unified sharding logic between different databases, realizing portability of the sharding result, forming a unified sharding strategy under a distributed storage architecture, realizing overall planning of data in each database, improving big data processing efficiency, improving data randomness, making sharding more balanced, reducing data skew, and reducing the occurrence of situations such as excessive pressure on part of the sharding nodes, idle resources of part of the nodes and the like affecting the overall performance.

Description

Technical Field

[0001] This application relates to the field of big data technology, specifically to a data fragmentation extraction method and system based on hierarchical hashing. Background Technology

[0002] In big data technology scenarios, ETL (Extract-Transform-Load) data sharding is a core step in improving data processing efficiency. Its goal is to achieve parallel extraction and distributed collaborative processing by evenly dividing massive amounts of data into several fragments.

[0003] Existing ETL systems have well-established solutions for data sharding and extraction, particularly for numeric fields. These solutions typically utilize modulo and absolute value (abs) operations to achieve even data sharding, effectively ensuring efficient extraction of numeric data. However, with the diversification of business operations, string fields (such as user IDs, order numbers, device numbers, and asset identifiers) are appearing more frequently in data tables, becoming a core target for data sharding and extraction. Existing sharding solutions for such fields have significant limitations:

[0004] First, existing solutions mostly rely on built-in hash functions in the database (such as CRC32, ORA_HASH, HASHBUCKET, etc.) to implement sharding mapping for string fields. However, different databases have inconsistent underlying implementation logic for these hash functions, resulting in differences in the hash results calculated for the same string field in different databases, thus causing unstable distribution of sharding results. This problem directly leads to poor portability of the sharding mechanism in ETL systems, making it impossible to implement a unified sharding strategy in a distributed storage architecture with multiple databases working together.

[0005] Secondly, in big data scenarios, massive amounts of data are distributed across multiple heterogeneous databases. The differences in the built-in hash functions of each database make it impossible to maintain cross-database consistency in sharding results. Therefore, a separate sharding extraction process must be designed and executed for each database. This distributed sharding model not only increases system configuration and maintenance costs but also limits data extraction efficiency to the worst-performing database. It makes it impossible to optimize global extraction performance through comprehensive planning, thus failing to meet the needs of efficient large-scale data processing.

[0006] Third, a single hash function can easily map string fields with similar characteristics (such as "APPLE" and "APPLEE") to the same shard, causing subsequent data modification, deletion, and differentiation operations to interfere with each other, reducing the accuracy and efficiency of data processing.

[0007] Fourth, existing sharding schemes do not fully consider the ease of data reading and load balancing requirements. Sharding results are prone to data skew, with some shard nodes experiencing excessive storage pressure while others have idle resources, further restricting the overall performance of data reading and subsequent processing.

[0008] In summary, existing ETL systems' sharding extraction schemes for string fields suffer from technical problems such as poor cross-database consistency, insufficient portability, ineffective load balancing, and interference from similar field processing. These issues make it difficult to meet the high-performance, high-consistency, and flexible scalability requirements of big data distributed storage scenarios. There is an urgent need for a unified, efficient, and scalable string field sharding hashing solution. Summary of the Invention

[0009] In view of this, this application discloses a data sharding extraction method based on hierarchical hashing, applied to ETL tasks. The method includes: obtaining data key values ​​corresponding to the data to be extracted; the data key values ​​are string fields; the data to be extracted comes from at least two databases; generating a perturbation factor vector based on the character frequency distribution, character entropy value, and length entropy value of the string field; the perturbation factor vector includes at least one perturbation factor for performing mixed perturbation processing on the hash result of the data key values ​​and at least one weight factor used in hierarchical mapping of the hash result; hashing the data key values ​​to obtain a first hash result; using the perturbation factor, performing mixed perturbation processing on the first hash result to obtain a second hash result; using the weight factor, performing hierarchical mapping on the second hash result to obtain sharding results allocated to the data to be extracted; and extracting data from the at least two databases based on the sharding results.

[0010] In some embodiments, generating a perturbation factor vector based on the character frequency distribution, character entropy value, and length entropy value of the string field includes: counting the occurrence frequency of each character in the string field; generating the character entropy value based on the occurrence frequency; generating the length entropy value based on the character length of the string field; and generating a perturbation factor vector containing a preset number of factors based on the occurrence frequency, the character entropy value, and the length entropy value. Wherein, if the occurrence frequency, the number of character entropy values, and the length entropy value are greater than the preset number, the largest value is selected as the factor; if the occurrence frequency, the number of character entropy values, and the length entropy value are less than the preset number, the occurrence frequency, the character entropy value, and the length entropy value are used as factors, followed by a default value.

[0011] In some embodiments, the hybrid perturbation is a three-layer perturbation, including an XOR perturbation, an amplification perturbation, and a cyclic perturbation.

[0012] In some embodiments, the step of using the perturbation factor to perform mixed perturbation processing on the first hash result to obtain a second hash result includes: obtaining a first perturbation factor in the perturbation factor vector; performing an XOR operation on the first hash result and the first perturbation factor to generate a first intermediate result; obtaining a second perturbation factor in the perturbation factor vector; using the product of the first intermediate result and the second perturbation factor as a second intermediate result; obtaining a third perturbation factor in the perturbation factor vector to generate an offset; performing a cyclic offset on the first hash result according to the offset to obtain a third intermediate result; and taking the modulo of the sum of the second intermediate result and the third intermediate result to obtain the second hash result.

[0013] In some embodiments, the step of using the weight factor to perform hierarchical mapping on the second hash result to obtain the sharding result allocated to the data to be extracted includes: obtaining a preset number of shards; and when the number of shards is less than a preset value, taking the second hash result modulo the number of shards to obtain the shard number corresponding to the data to be extracted.

[0014] In some embodiments, the step of using the weight factor to perform hierarchical mapping on the second hash result to obtain the sharding result allocated to the data to be extracted includes: when the number of shards is greater than a preset value, dividing the number of shards into a first number of groups, wherein each group contains a second number of shards; the product of the first number and the second number is the number of shards; obtaining a first weight factor in the perturbation factor vector; taking the modulo of the first number using the product of the second hash result and the first weight factor to generate a group number; obtaining a second weight factor in the perturbation factor vector; summing the product of the group number and the second weight factor with the second hash result to generate a summation result, and then taking the modulo of the second number using the summation result to generate a shard number within the group; and summing the product of the group number and the second number with the shard number within the group to generate the shard number allocated to the data to be extracted.

[0015] In some embodiments, before generating a group number by taking the modulo of the first quantity using the product of the second hash result and the first weight factor, a sensitivity adjustment factor is used to perform sensitivity condition processing on the first weight factor to obtain an adjusted first weight factor; before generating a summation result by summing the product of the group number and the second weight factor with the second hash result, the sensitivity adjustment factor is used to perform sensitivity condition processing on the second weight factor to obtain an adjusted second weight factor.

[0016] In some embodiments, the method for obtaining the sensitivity adjustment factor includes: receiving a configuration request for the sensitivity adjustment factor; and configuring the sensitivity adjustment factor in response to the configuration request; or, obtaining the sensitivity adjustment factor in the perturbation factor vector.

[0017] In some embodiments, the data extraction from the at least two databases based on the sharding result includes: allocating a number of threads corresponding to the number of shards indicated by the sharding result, and extracting data from the data shards corresponding to the at least two databases; wherein, the method for determining the data extraction time period and extraction rate of each thread includes: for each thread, obtaining the historical data extraction situation corresponding to the thread; determining the daily data extraction volume and data writing time period of the thread from the historical extraction situation; taking the time period other than the data writing time period of the day as the data extraction time period of the thread, and taking the quotient of the daily data collection volume and the data extraction time period as the extraction rate of the thread.

[0018] This application also proposes a data sharding extraction system based on hierarchical hashing, applied to ETL tasks. The system includes: an acquisition module for acquiring data key values ​​corresponding to the data to be extracted; the data key values ​​are string fields; the data to be extracted comes from at least two databases; a generation module for generating a perturbation factor vector based on the character frequency distribution, character entropy value, and length entropy value of the string field; the perturbation factor vector includes at least one perturbation factor for performing mixed perturbation processing on the hash result of the data key values ​​and at least one weight factor used in hierarchical mapping of the hash result; a hashing module for hashing the data key values ​​to obtain a first hash result; a perturbation module for performing mixed perturbation processing on the first hash result using the perturbation factors to obtain a second hash result; a hierarchical mapping module for performing hierarchical mapping on the second hash result using the weight factors to obtain sharding results allocated to the data to be extracted; and an extraction module for extracting data from the at least two databases based on the sharding results.

[0019] In the aforementioned embodiment, the solution involves analyzing the characteristics of the sharded objects (i.e., string-type data keys) in at least two databases participating in the distributed big data storage, generating a perturbation factor vector for hybrid routing and hierarchical mapping, and then performing hybrid perturbation and hierarchical mapping on the hash results of the data keys based on the elements in this vector to obtain the sharding result. This achieves several advantages: First, regardless of the database, a unified hashing process and sharding logic are used, ensuring unified sharding logic across different databases and making the sharding result portable, forming a unified sharding strategy under the distributed storage architecture. Second, the unified sharding logic across databases allows for comprehensive planning of data across databases, not limited to the worst-performing database, thus improving big data processing efficiency. Third, the sharding result fully considers the characteristics of the data keys, and combines these characteristics to perform sufficient hybrid perturbation and hierarchical mapping on the initial hash result, increasing data randomness and preventing similar fields from being placed in the same shard, making subsequent processing simpler. Fourth, through hybrid perturbation and hierarchical mapping, the sharding becomes more balanced, reducing data skew, excessive pressure on some shard nodes, and idle resources on some nodes, which negatively impact overall performance. Attached Figure Description

[0020] The accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below.

[0021] Figure 1 This is a flowchart illustrating a data fragmentation extraction method based on hierarchical hashing, as shown in an embodiment of this application.

[0022] Figure 2 This is a schematic diagram illustrating the method for generating the perturbation factor vector in this application.

[0023] Figure 3 This is a schematic diagram illustrating the method flow of hybrid perturbation as described in this application.

[0024] Figure 4 This is a schematic diagram of the layered mapping method shown in this application.

[0025] Figure 5 This is a schematic diagram illustrating a method for confirming the data extraction time period and extraction rate, as shown in this application.

[0026] Figure 6 This is a schematic diagram of the structure of a data fragmentation extraction system based on hierarchical hashing, as shown in this application. Detailed Implementation

[0027] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of devices and methods consistent with some aspects of this application as detailed in the appended claims.

[0028] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items. It should also be understood that the word “if” as used herein, depending on the context, can be interpreted as “when,” “in response to a determination,” or “when…”.

[0029] In view of the problems existing in the prior art, this application discloses a data fragmentation extraction method based on hierarchical hashing. This method analyzes the characteristics of sharded objects (string-type data keys) in at least two databases participating in big data distributed storage, generates a perturbation factor vector for hybrid perturbation and hierarchical mapping, and then performs hybrid perturbation and hierarchical mapping on the hash results of the data keys based on the elements in this vector to obtain the sharding result. This results in four main ways: First, regardless of the database, a unified hashing process and sharding logic are used, achieving unified sharding logic across different databases and making the sharding result portable, forming a unified sharding strategy under the distributed storage architecture. Second, the unified sharding logic across databases allows for comprehensive planning of data across databases, not limited to the worst-performing database, improving big data processing efficiency. Third, the sharding result fully considers the characteristics of the data keys, and combines these characteristics to perform sufficient hybrid perturbation and hierarchical mapping on the initial hash result, increasing data randomness and preventing similar fields from being placed in the same shard, making subsequent processing simpler. Fourth, through hybrid perturbation and hierarchical mapping, the sharding is more balanced, reducing data skew, excessive pressure on some shard nodes, and idle resources on some nodes, which affect overall performance.

[0030] The following description, in conjunction with the accompanying drawings, illustrates the embodiments. Please refer to the attached figures. Figure 1 , Figure 1 This is a flowchart illustrating a data fragmentation extraction method based on hierarchical hashing, as shown in an embodiment of this application.

[0031] Figure 1The illustrated hierarchical hash-based data fragmentation extraction method can be applied to electronic devices. These electronic devices can execute the method by incorporating software logic corresponding to the hierarchical hash-based data fragmentation extraction method. The type of electronic device can be a laptop, computer, server, mobile phone, PDA, etc. This application does not specifically limit the type of electronic device. The electronic device can also be a client device, a server device, or a combination of both.

[0032] like Figure 1 As shown, the method may include steps S102-S112. Unless otherwise specified, this application does not specifically limit the order in which these steps are performed.

[0033] S102, obtain the data key value corresponding to the data to be extracted.

[0034] The data keys are string fields. The data to be extracted comes from at least two databases.

[0035] With the development of the internet, the volume of data is increasing dramatically. To ensure data access performance, distributed architectures are generally used to store data. Due to the wide variety of data types, distributed storage typically utilizes multiple database types. For example, structured data is stored in distributed relational databases (such as TiDB and OceanBase), adapting to complex relational queries (such as statistical correlations between asset information and vulnerability data). Semi-structured data is stored in document-oriented databases (such as MongoDB and Elasticsearch), adapting to flexible field queries and full-text retrieval (such as vulnerability descriptions and threat intelligence reports). Time-series data is stored in distributed time-series databases (such as InfluxDB and ClickHouse), adapting to fast writing and time-range queries of high-frequency real-time data (such as vulnerability scanning task status and system monitoring metrics). Massive amounts of unstructured data are stored in distributed file systems (such as HDFS and MinIO), adapting to low-cost storage and batch reading of large files (such as POC scripts, patch files, and log backups). High-concurrency cached data is stored in distributed cache databases (such as Redis Cluster), adapting to rapid response to frequently accessed data (such as hot vulnerability information and asset whitelists).

[0036] It's even possible to use multiple databases to store data for the same field. For example, a provincial-level database might use OceanBase, while a city-level database might use MySQL, requiring data extraction from different databases. In other words, data is stored in at least two databases. In the field of data sharding, key-value pairs are often used for sharding. These key-value pairs can be numeric or string-based. As data volume increases, more and more scenarios will use strings as key-value pairs. This application discusses the data sharding logic when strings are used as key-value pairs.

[0037] Data key values ​​refer to string fields in the data to be processed that are used to uniquely identify a record, such as user ID (“user_00123”), order number (“ORD_20251026_8765”), device serial number (“SN_PC_7890”), etc.

[0038] The data to be extracted refers to data that exists in at least two databases and needs to be extracted via ETL.

[0039] S104, generate a perturbation factor vector based on the character frequency distribution, character entropy value and length entropy value of the string field.

[0040] The character frequency distribution refers to the proportion of times each character appears in a string relative to its total length. For example, in "APPLE", "P" appears twice, and the total length is 5, so the frequency of "P" is 2 / 5 = 0.4.

[0041] The character entropy value measures the randomness of the character distribution in a string, and is calculated using the following formula: ;in, (where is the frequency of the i-th character). The higher the entropy value, the more dispersed the character distribution. For example, the entropy value of "ABCDE" is higher than that of "AAAAA".

[0042] The length entropy value is related to the character length; the longer the character, the greater the entropy value. The calculation formula is as follows: Where l is the character length and a is a preset constant. For example, 1, 2, or 3, etc.

[0043] The perturbation factor vector includes at least one perturbation factor for mixing and perturbing the hash result of the data key value and at least one weighting factor used in the hierarchical mapping of the hash result. In other words, the perturbation factor vector contains the factors needed for subsequent perturbation and hierarchical mapping of the hash result.

[0044] This section emphasizes the difference from related technologies. Related technologies typically apply only one or two perturbations to the hash result, and the perturbation factors are generally pre-set and unrelated to the data key (string field). This application applies multiple mixed perturbations to the hash result, resulting in better perturbation effects. Furthermore, the perturbation factors are generated after analyzing the characteristics of the data key (string field), and are closely related to these characteristics, increasing hash randomness and causing similar characters to be distributed across different shards.

[0045] In addition, in related technologies, when using hash results for shard mapping (modulo), it is generally only a single-level modulo operation. When there are many shards, data skew is likely to occur, resulting in some shards having large amounts of data and others having small amounts of data. However, this application uses layered modulo operation, which makes it easier to balance the shards.

[0046] The length of the perturbation factor vector is generally preset based on experience, for example, it can be 5, 7, or 9. The following example uses 5. An example format of the perturbation factor vector is as follows: ,in, This refers to the frequency of high-frequency characters, where Z is the character entropy value and M is the length entropy value. It should be noted that if the perturbation factor vector has a relatively long preset length, after using the character frequency distribution, character entropy value, and length entropy value as factors, the remaining factors can be filled with the default value d. If the perturbation factor vector has a relatively short length, then the frequencies of high-frequency characters can be selected from the character frequency distribution to fill in the factors.

[0047] Some factors in the perturbation factor vector can be used directly, or after pre-processing, as factors in the mixed perturbation in S108, and are referred to as perturbation factors. Some factors in the perturbation factor vector can be used directly, or after pre-processing, as factors used in the hierarchical mapping in S110, and are referred to as weight factors. The positions of the perturbation factors and weight factors in the perturbation factor vector are pre-set. For example, when the factor vector length is 5, the first three are perturbation factors and the last two are weight factors, or the first two are weight factors and the last three are perturbation factors. It should be noted that the pre-processing may include normalization, standardization, etc., and can be selected according to the actual situation.

[0048] This step can be based on the interpretation of character frequency distribution, character entropy value, and length entropy value to obtain corresponding values, which are then filled into the perturbation factor vector to form a vector. One relevant step illustrated in this application will be described in subsequent embodiments.

[0049] S106, perform hash processing on the data key value to obtain the first hash result.

[0050] This step can use conventional hash algorithms, such as SHA-256 or MurmurHash, to calculate the data key-value pairs, converting the string into a fixed-length numerical value (hash value), called the first hash result. This step ensures that different strings generate unique initial hash values, providing a basis for subsequent perturbations. In this step, the string can first be represented using character code points, such as ASCII or Unicode, and then the hash operation can be performed.

[0051] S108, using the perturbation factor, perform a mixed perturbation process on the first hash result to obtain a second hash result.

[0052] The hybrid perturbation is a three-layer perturbation, including an XOR perturbation, an amplification perturbation, and a cyclic perturbation. In this step, an operator step containing three layers of perturbation can be pre-designed, where the factors used in each layer of perturbation are derived from the perturbation factor vector generated in S104. One operator step illustrated in this application will be described in subsequent embodiments.

[0053] S110, using the weight factor, perform hierarchical mapping on the second hash result to obtain the sharding result allocated to the data to be extracted.

[0054] The hierarchical mapping refers to splitting the total number of fragments into multiple layers of product (e.g., total fragments 12 = 3 × 4, i.e., two layers, the first layer includes 3 groups, and each group in the second layer includes 4 fragments), and each layer is assigned a modulo ratio according to the weight factor (e.g., weight factors 0.6 and 0.4 correspond to modulo 3 for the first layer and modulo 4 for the second layer).

[0055] The final output of the hierarchical mapping of data key-value pairs (string fields) (such as "3" and "6") indicates the target shard number to which the data should be assigned, thus obtaining the sharding result.

[0056] S112, Data is extracted from the at least two databases based on the sharding results.

[0057] Based on the sharding results obtained in step 5, data corresponding to each shard is extracted from each database to achieve unified sharding and parallel processing of cross-database data.

[0058] The scheme described in S102-S112 allows for the generation of perturbation factor vectors for hybrid bypassing and hierarchical mapping by analyzing the characteristics of sharded objects (i.e., string-type data keys) in at least two databases participating in big data distributed storage. Based on the elements in this vector, the hash results of the data keys are subjected to hybrid perturbation and hierarchical mapping to obtain the sharding result. This achieves two goals: First, regardless of the database, a unified hashing process and sharding logic are used, unifying the sharding logic across different databases, making the sharding result portable, and forming a unified sharding strategy under the distributed storage architecture. Second... Each database adopts a unified sharding logic, which enables comprehensive planning of data across databases, not limited to the worst-performing database, thus improving the efficiency of big data processing. Third, the sharding results fully consider the characteristics of data key values, and combine these characteristics to perform sufficient mixing perturbation and hierarchical mapping on the initial hash results, improving data randomness and preventing similar fields from being assigned to the same shard, making subsequent processing simpler. Fourth, through mixing perturbation and hierarchical mapping, the sharding is more balanced, reducing data skew, excessive pressure on some shard nodes, and idle resources on some nodes, which affect the overall performance.

[0059] For example, a company needs to extract data from MySQL (which stores user order data) and MongoDB (which stores device log data), and shard it using string data key-value pairs. The target total number of shards is 6 (with a 2×3 hierarchical mapping). The perturbation factor vector has a preset length of 5 (the first 3 are perturbation factors, and the last 2 are weight factors).

[0060] S102: Retrieve data key-value pairs.

[0061] MySQL: Order numbers “ORD_001” (K1) and “ORD_002” (K2);

[0062] MongoDB: Device IDs “DEV_APPLE” (K3) and “DEV_APPLEE” (K4, similar string).

[0063] S104: Generate the perturbation factor vector.

[0064] Analyze character frequency, calculate character entropy (Z) and length entropy (M), and generate a vector:

[0065] K1: High-frequency character frequency [0.33, 0.2, 0.17], character entropy value Z=1.8, length entropy value M=6 / 2=3, resulting in vector [0.33,0.2, 0.17, 1.8, 3];

[0066] K2: High-frequency character frequency [0.33, 0.2, 0.17], character entropy value Z=1.78, length entropy value M=6 / 2=3, resulting in vector [0.33,0.2, 0.17, 1.78, 3];

[0067] K3: High-frequency character frequency [0.25, 0.25, 0.125], character entropy value Z=2.1, length entropy value M=8 / 2=4, resulting in vector [0.25, 0.25, 0.125, 2.1, 4];

[0068] K4: High-frequency character frequency [0.22, 0.22, 0.11], character entropy value Z=2.2, length entropy value M=9 / 2=4.5, resulting in vector [0.22, 0.22, 0.11, 2.2, 4.5].

[0069] S106: Hash processing.

[0070] Using the MurmurHash algorithm, the first hash result (simplified value) is obtained:

[0071] K1: 15689; K2: 18923; K3: 21456; K4: 23781.

[0072] S108: Mixed disturbance.

[0073] The second hash result (simplified value) is obtained through a three-layer perturbation process: XOR + multiply-accumulate + loop.

[0074] K1: 3215; K2: 3876; K3: 4521; K4: 5132.

[0075] S110: Layered mapping.

[0076] Total number of slices 6 = 2 × 3, weighting factors are directly taken from the last two digits of the vector (character entropy value and length entropy value normalized and adapted to the layer ratio):

[0077] K1: First layer 3215mod2=1, second layer 3215mod3=2, resulting in 1×3+2=5 fragments;

[0078] K2: First layer 3876 mod 2 = 0, second layer 3876 mod 3 = 0, resulting in fragment 0×3+0=0;

[0079] K3: First layer 4521mod2=1, second layer 4521mod3=0, resulting in fragment 1×3+0=3;

[0080] K4: First layer 5132mod2=0, second layer 5132mod3=2, resulting in fragment 0×3+2=2.

[0081] S112: Cross-database extraction.

[0082] Extract data from shard 5 (K1) and shard 0 (K2) from MySQL, and extract data from shard 3 (K3) and shard 2 (K4) from MongoDB.

[0083] As can be seen, firstly, MySQL and MongoDB use the same logic, avoiding the sharding inconsistencies caused by the built-in hash functions of traditional databases, and thus enabling the portability of sharding results.

[0084] Second, MySQL and MongoDB use the same logic to achieve cross-database coordinated extraction.

[0085] Third, K3 ("DEV_APPLE") and K4 ("DEV_APPLEE") are assigned to fragments 3 and 2 respectively, so that they do not interfere with each other when modified or deleted, thus improving processing efficiency.

[0086] Fourth, the proportion of data in each shard is relatively balanced, with no data skew, which improves resource utilization compared to traditional single-layer modulo.

[0087] In some embodiments, a method for generating a perturbation factor vector is illustrated. See also Figure 2 , Figure 2 This is a schematic diagram illustrating the method for generating the perturbation factor vector in this application. Figure 2 This is an explanation of S104. For example... Figure 2 As shown, the method may include S202-S208.

[0088] S202, Calculate the frequency of occurrence of each character in the string field.

[0089] This step iterates through each character in the string field, records the frequency of each character, and then calculates the ratio of that frequency to the total string length to obtain the frequency of each character. The aim is to extract the character composition features of the string, providing foundational data for subsequent character entropy calculations and perturbation factor selection.

[0090] S204, Generate the character entropy value based on the frequency of occurrence.

[0091] Based on the frequency of each character obtained from S202, substitute it into the entropy formula. This generates character entropy values. The aim is to quantify the randomness of character distribution in a string; a higher entropy value indicates a more dispersed character distribution and more unique characteristics.

[0092] S206, Generate the length entropy value based on the character length of the string field.

[0093] In this step, the actual character length of the string can be used as a basis, and the preset formula can be substituted into it. This generates a length entropy value. The aim is to transform the string length feature into a quantization parameter that can be used for perturbation, reflecting the differences between strings of different lengths.

[0094] S208, based on the occurrence frequency, the character entropy value, and the length entropy value, generate a perturbation factor vector containing a preset quantity factor.

[0095] Wherein, if the occurrence frequency, the number of character entropy values ​​and the number of length entropy values ​​are greater than the preset number, then the largest value is selected as the factor; if the occurrence frequency, the number of character entropy values ​​and the number of length entropy values ​​are less than the preset number, then the occurrence frequency, the character entropy value and the length entropy value are used as factors and then filled with default values ​​as factors.

[0096] In this step, the preset length of the perturbation factor vector can be determined first (e.g., 5, 7, etc., set according to business needs); then, based on the obtained occurrence frequency, character entropy value, and length entropy value, a set of feature parameters can be formed.

[0097] If the total number of feature parameters is greater than the preset length: select the highest-valued ones from the frequency of occurrence, and combine them with the character entropy value and the length entropy value to make up a vector of the preset length.

[0098] If the total number of feature parameters is less than the preset length: fill in all feature parameters first, and fill the remaining positions with default values ​​(such as 0.5, 1.0, etc.).

[0099] Steps S202-S208 can generate a fixed-length vector containing the core features of the string, providing parameter support for subsequent hybrid perturbation and hierarchical mapping.

[0100] The following explanation uses the generation of perturbation factor vectors using "APPLE" and "APPLEE" as an example. The default length of the perturbation factor vector is 5.

[0101] S202: Count the frequency of occurrence of each character in a string field.

[0102] In the string "APPLE" (length 5), the character A appears once, P appears twice, L appears once, and E appears once. The frequencies are: A: 0.2, P: 0.4, L: 0.2, E: 0.2, resulting in the frequency set [0.4, 0.2, 0.2, 0.2].

[0103] In the string "APPLEE" (length 6), the characters A appear once, P appear twice, L appear once, and E appear twice, with frequencies of A: 0.17, P: 0.33, L: 0.17, E: 0.33, resulting in the frequency set [0.33, 0.33, 0.17, 0.17].

[0104] S204: Generate character entropy values ​​based on frequency of occurrence.

[0105] The following simplified steps for S204, as described above, yield a character entropy value H of 1.92 for "APPLE" and 1.98 for "APPLEE".

[0106] S206: Generate a length entropy value based on the character length.

[0107] The following simplified steps for S204 can be referenced from the previous description. Assuming the constant is a=2, the length entropy value M of "APPLE" is 2.5 and the length entropy value M of "APPLEE" is 3.

[0108] S208: Generate a perturbation factor vector containing a preset number of factors.

[0109] The preset vector length is 5. The vector is obtained by selecting the maximum 3 frequencies, character entropy value, and length entropy value.

[0110] For the word "APPLE", the three largest frequencies are selected as 0.4 (P), 0.2 (A), and 0.2 (L). Combined with a character entropy value of 1.92 and a length entropy value of 2.5, the perturbation factor vector [0.4, 0.2, 0.2, 1.92, 2.5] can be obtained.

[0111] For “APPLEE”, the three largest frequencies are selected as 0.33 (P), 0.33 (E), and 0.17 (A). Combined with the character entropy value of 1.98 and the length entropy value of 3, the perturbation factor vector [0.33, 0.33, 0.17, 1.98, 3] can be obtained.

[0112] As can be seen from steps S202-S208, although the two strings are similar, their character features are still quite different, and the perturbation vector factors obtained are also quite different. Subsequently, by combining hybrid perturbation and hierarchical mapping, similar strings can be divided into different segments.

[0113] In some embodiments, a method for entering a mixed disturbance is illustrated. See also Figure 3 , Figure 3 This is a schematic diagram illustrating the method flow of hybrid perturbation as described in this application. Figure 3 This is an explanation of S108. For example... Figure 3As shown, the method may include S302-S308. The hybrid perturbation is a three-layer perturbation, including XOR perturbation, amplification perturbation, and cyclic perturbation.

[0114] S302, obtain the first perturbation factor in the perturbation factor vector, and perform XOR operation on the first hash result and the first perturbation factor to generate a first intermediate result.

[0115] In this step, a first perturbation factor (such as the first element in the vector) at a preset position can be extracted from the perturbation factor vector. To perform perturbation calculations, the first element can be multiplied by a preset factor, such as 1000 or 10000. An XOR operation (a bitwise operation where identical bits are 0 and different bits are 1) is performed on the first hash result (numerical type) obtained in step S106 to obtain the first intermediate result. This can be expressed as a formula. ,in, This is the first intermediate result. This is the first hash result, where R is the first perturbation factor. This is an XOR operation.

[0116] This step aims to break the original bit distribution pattern of the first hash result through XOR operation, thereby initially enhancing the randomness of the hash value.

[0117] S304, obtain the second perturbation factor in the perturbation factor vector, and take the product of the first intermediate result and the second perturbation factor as the second intermediate result.

[0118] In this step, the second perturbation factor (such as the second element) is extracted from the perturbation vector and multiplied by the first intermediate result obtained in S302 to obtain the second intermediate result. In this step, the second element is generally a probability less than 1. To amplify the difference, the second element is magnified by a preset factor (e.g., 10 or 20 times) as the second perturbation factor. This can be expressed as a formula. ,in, This is the second intermediate result. This is the second disturbance factor.

[0119] This step amplifies the numerical differences in the first intermediate results through multiplication, thereby enhancing the distinguishability of different string hash values.

[0120] S306, obtain the third perturbation factor in the perturbation factor vector to generate an offset, and perform a cyclic offset on the first hash result according to the offset to obtain a third intermediate result.

[0121] In this step, the third perturbation factor (e.g., the third element) is extracted from the vector and converted into an integer offset. If the third element is greater than 1, only the integer part is taken; if the third element is less than 1, it can be magnified tenfold before taking the integer part. Then, the first hash result can be cyclically shifted (shifting the binary bits of the value to the left or right by the offset, and padding any overflow bits to the other end) to obtain the third intermediate result. This can be expressed as: ,in, This is the third intermediate result. As the third disturbance factor, This is an expression for loop operations.

[0122] This step aims to change the binary structure of the hash value through cyclic offsets, thereby further enhancing randomness and preventing similar strings from having the same hash value.

[0123] S308, take the modulo of the sum of the second intermediate result and the third intermediate result to obtain the second hash result.

[0124] In this step, the second intermediate result of S304 is added to the third intermediate result of S306, and then modulo the result by a preset modulus (e.g., 100000) to obtain the final second hash result. This can be expressed as a formula. ,in, This is the second hash result, i.e., the hash result after mixing perturbations, where b is the preset modulus. The formula can be transformed as follows: .

[0125] This step aims to limit the result to a fixed range through modulo operations, while incorporating the effects of the first two perturbation layers to ensure a uniform distribution of hash values.

[0126] Mixed perturbations can be achieved through S302-S308, increasing the randomness and uniformity of hash distribution.

[0127] Using "APPLE" and "APPLEE" as data keys, a mixed perturbation is performed based on the previously generated perturbation factor vector.

[0128] The first hash result of “APPLE” is 21456, and the perturbation factor vector is [0.4, 0.2, 0.2, 1.92, 2.5] (the first 3 are perturbation factors); the first hash result of “APPLEE” is 23781, and the perturbation factor vector is [0.33, 0.33, 0.17, 1.98, 3] (the first 3 are perturbation factors).

[0129] S302: XOR processing. For "APPLE", a first perturbation factor of 0.4 can be used (amplified by 10000 to get 4000), resulting in 17616 (first intermediate result).

[0130] For “APPLEE”, a first perturbation factor of 0.33 can be used (amplified by 10000 to get 3300), resulting in 20541 (first intermediate result).

[0131] S304: Product operation. For "APPLE", a second perturbation factor of 0.2 can be used (amplified by 10 to get 2), 17616×2 = 35232 (second intermediate result).

[0132] For “APPLEE”, a second perturbation factor of 0.33 can be used (magnified by 10 to get 3), 20541×3 = 61623 (second intermediate result).

[0133] S306: Cyclic offset. For "APPLE", a third perturbation factor of 0.2 can be used (converted to an offset of 2). 21456 is cyclically shifted right by 2 bits to obtain 5364 (the third intermediate result).

[0134] For “APPLEE”, the third perturbation factor is 0.17 (converted to offset 1), and 23781 is cyclically shifted right by 1 bit to obtain 11890 (the third intermediate result).

[0135] S308: Summation and Modulo. For "APPLE", 35232 + 5364 = 40596, modulo 10000 gives 596 (second hash result). For "APPLEE", 61623 + 11890 = 73513, modulo 10000 gives 3513 (second hash result).

[0136] As can be seen, similar strings can be broken down into hash results that are relatively far apart using S302-S308.

[0137] In some embodiments, a method for hierarchical mapping is illustrated. A preset number of shards is obtained. If the number of shards is less than a preset value, the second hash result is modulo the number of shards to obtain the shard number corresponding to the data to be extracted.

[0138] The preset number of shards is a value preset based on experience, such as 6 or 8. Extensive experiments have shown that the uniformity of hashing can be guaranteed even with this number of shards. To improve computational efficiency, a single-layer mapping can be used.

[0139] Please see Figure 4 , Figure 4 This is a schematic diagram of the layered mapping method shown in this application. Figure 4 This illustration shows the steps for dividing the mapping into two layers when the number of fragments exceeds a preset value (e.g., 8). Figure 4As shown, the method may include S402-S410.

[0140] S402, if the number of fragments is greater than a preset value, the fragments of the specified number are divided into a first number of groups, wherein each group contains a second number of fragments. The product of the first number and the second number is the number of fragments.

[0141] In this step, when the total number of fragments exceeds a preset threshold (e.g., 8), the total number of fragments is decomposed into the product of two integers. For example, if the total number of fragments is 12, we can get 12 = 3 × 4, where the first number = 3 and the second number = 4, forming a two-layer structure of "grouping - fragmentation within a group".

[0142] The purpose of this step is to reduce the pressure of single-layer mapping by layering, avoid data skew caused by single modulo, and improve the balance of sharding.

[0143] S404, obtain the first weight factor in the perturbation factor vector.

[0144] In this step, the first weight factor (such as the fourth element in the vector) at a preset position is extracted from the perturbation factor vector and used for the proportional allocation of the first layer grouping mapping.

[0145] S406, using the product of the second hash result and the first weight factor, modulo the first quantity to generate a group number.

[0146] In this step, the second hash result obtained in S108 can be multiplied by the first weight factor, and then modulo the first quantity (total number of groups) to obtain the group number to which the current data belongs (starting from 0).

[0147] S408, obtain the second weighting factor in the disturbance factor vector.

[0148] In this step, a second weight factor (such as the fifth element in the vector) at a preset position is extracted from the perturbation factor vector and used to adjust the segmentation mapping within the second layer group.

[0149] S410, the product of the group number and the second weight factor is summed with the second hash result to generate a summation result, and then the summation result is modulo the second quantity to generate the fragment number within the group.

[0150] In this step, the product of the group number and the second weight factor is first calculated, then summed with the second hash result, and finally modulo the second quantity (the number of shards per group) to obtain the shard number within the group (starting from 0).

[0151] S412, using the product of the group number and the second quantity, and summing it with the shard number within the group, the shard number to be assigned to the data to be extracted is generated.

[0152] In this step, the group number can be multiplied by the second quantity (the number of segments per group), and then the segment number within the group can be added to obtain a unique total segment number.

[0153] The steps illustrated in S402-S412 can achieve two-layer mapping, increasing the uniformity of the slices.

[0154] Taking "APPLE" and "APPLEE" as examples, the total number of fragments is 12 (split into 3×4), the preset threshold is 8, and the perturbation factor vectors are as follows: "APPLE": [0.4, 0.2, 0.2, 1.92, 2.5] (first weight factor = 1.92, second weight factor = 2.5), second hash result = 596; "APPLEE": [0.33, 0.33, 0.17, 1.98, 3] (first weight factor = 1.98, second weight factor = 3), second hash result = 3513.

[0155] S402: Total number of segments 12 = 3 × 4 (first quantity = 3, second quantity = 4).

[0156] For S404-S406, the group number for "APPLE" is 1; for "APPLEE", the group number is 0.

[0157] S408-S410, for "APPLE", the fragment number within the group is 2; for "APPLEE", the fragment number within the group is 1.

[0158] S412, "APPLE" is 6, "APPLEE" is 1.

[0159] By following the steps above, similar characters can be assigned to different shards. Similar characters can be assigned to shards under different groups, which can increase hash randomness and shard uniformity.

[0160] In some embodiments, a sensitivity factor can be added to facilitate control during hierarchical mapping. The sensitivity factor can be represented by γ. When γ > 1 (high sensitivity), the weights are amplified, and the "amplification effect" during mapping is stronger. Similarly... Values ​​multiplied together will have greater differences, making them more likely to be mapped to different shards. This is suitable for fields that require strong dispersion (such as user IDs) to avoid clustering of similar IDs.

[0161] When γ≈1 (medium sensitivity): maintain the default perturbation effect.

[0162] When γ < 1 (low sensitivity): weights are reduced, and the mapping process is "smoother." Even The values ​​may differ significantly, but the final results may be quite similar.

[0163] Specifically, before generating the group number by taking the modulo of the first quantity using the product of the second hash result and the first weight factor, the first weight factor can be subjected to sensitivity condition processing using a sensitivity adjustment factor to obtain the adjusted first weight factor.

[0164] Before summing the product of the group number and the second weight factor with the second hash result to generate the summation result, the second weight factor is subjected to sensitivity condition processing using the sensitivity adjustment factor to obtain the adjusted second weight factor.

[0165] Therefore, the sensitivity factor can be used to flexibly adjust the hierarchical mapping, increasing the degree of controllability.

[0166] In some embodiments, the method for obtaining the sensitivity adjustment factor includes:

[0167] Receive a configuration request for the sensitivity adjustment factor; and in response to the configuration request, configure the sensitivity adjustment factor; or, obtain the sensitivity adjustment factor from the perturbation factor vector.

[0168] In this embodiment, two methods can be provided to obtain the sensitivity adjustment factor. One method is through configuration, which allows for manual configuration via an interactive approach. The other method, in order to increase the randomness of similar character segmentation, obtains elements from the perturbation factor vector and processes them (e.g., standardization, expansion, etc.) to obtain the sensitivity adjustment factor.

[0169] In some embodiments, after obtaining the sharding results, data ETL extraction can be performed using sharding threads. A number of threads corresponding to the number of shards indicated by the sharding results can be allocated to extract data from the data shards corresponding to the at least two databases. This allows for efficient data extraction through separate threads for each shard.

[0170] In some embodiments, in order to extract data more reasonably, ensuring data writing without causing resource consumption, the collection period and extraction rate can be determined based on historical data.

[0171] Please see Figure 5 , Figure 5 This is a schematic flowchart illustrating a method for confirming the data extraction time period and extraction rate, as described in this application. Figure 5 As shown, the method may include S502-S506.

[0172] S502, for each thread, obtain the historical data extraction status corresponding to that thread.

[0173] In this step, we can collect historical execution records for each thread responsible for extracting specific data shards, including extraction time, data volume extracted, and corresponding database write operation time over a period of time (e.g., 30 days).

[0174] S504, determine the daily data extraction volume and data writing time period of the thread from the historical extraction data.

[0175] In this step, statistical analysis can be performed on the historical data acquired by S502. The daily data extraction volume is obtained, i.e., the average daily data volume of the corresponding shard for this thread is calculated (e.g., the average daily data volume over the past 30 days). The data write time period is obtained, i.e., the high-frequency periods for the database to write data to this shard are identified (e.g., determining from historical write logs that 9:00-11:00 is the daily write peak).

[0176] S506, the time period other than the data writing time period of each day is used as the data extraction time period of the thread, and the ratio of the daily data collection amount and the data extraction time period is used as the extraction rate of the thread.

[0177] In this step, the extraction time period can be obtained. Excluding peak database write times, the remaining time is the effective extraction window for this thread (e.g., 0:00-9:00 or 11:00-24:00 daily). The extraction rate is obtained by dividing the average daily extraction volume by the total duration of the extraction time period to get the suggested extraction rate per unit time. For example, 10GB ÷ 22 hours equals 0.45GB / hour.

[0178] The schemes described in S502-S506 can avoid read / write conflicts, ensure data integrity, optimize resource utilization, reduce system load, achieve controllable extraction efficiency, and ensure data timeliness and load balance.

[0179] This application also proposes a data fragmentation and extraction system based on hierarchical hashing. This system is applied to ETL tasks. See also... Figure 6 , Figure 6 This is a schematic diagram illustrating the structure of a data fragmentation and extraction system based on hierarchical hashing, as shown in this application. Figure 6 As shown, a data sharding extraction system 600 based on hierarchical hashing includes:

[0180] The acquisition module 610 acquires the data key value corresponding to the data to be extracted; the data key value is a string field; the data to be extracted comes from at least two databases;

[0181] The generation module 620 generates a perturbation factor vector based on the character frequency distribution, character entropy value, and length entropy value of the string field; the perturbation factor vector includes at least one perturbation factor for performing mixed perturbation processing on the hash result of the data key value and at least one weight factor used in the hierarchical mapping of the hash result;

[0182] Hash module 630 performs hash processing on the data key value to obtain a first hash result;

[0183] The perturbation module 640 uses the perturbation factor to perform a mixed perturbation process on the first hash result to obtain a second hash result;

[0184] The hierarchical mapping module 650 uses the weight factor to perform hierarchical mapping on the second hash result to obtain the sharding result allocated to the data to be extracted;

[0185] The extraction module 660 extracts data from the at least two databases based on the sharding results.

[0186] In some embodiments, the generation module 620 further includes:

[0187] Calculate the frequency of occurrence of each character in the string field;

[0188] The character entropy value is generated based on the frequency of occurrence.

[0189] The length entropy value is generated based on the character length of the string field;

[0190] Based on the occurrence frequency, the character entropy value, and the length entropy value, a perturbation factor vector containing a preset number of factors is generated; wherein, if the occurrence frequency, the number of character entropy values, and the length entropy value are greater than the preset number, the largest value is selected as the factor; if the occurrence frequency, the number of character entropy values, and the length entropy value are less than the preset number, the occurrence frequency, the character entropy value, and the length entropy value are used as factors, and then a default value is filled in as the factor.

[0191] In some embodiments, the hybrid perturbation is a three-layer perturbation, including an XOR perturbation, an amplification perturbation, and a cyclic perturbation.

[0192] In some embodiments, the disturbance module 640 further includes:

[0193] Obtain the first perturbation factor in the perturbation factor vector, and perform an XOR operation between the first hash result and the first perturbation factor to generate a first intermediate result;

[0194] Obtain the second perturbation factor from the perturbation factor vector, and use the product of the first intermediate result and the second perturbation factor as the second intermediate result;

[0195] Obtain the third perturbation factor from the perturbation factor vector to generate an offset, and perform a cyclic offset on the first hash result according to the offset to obtain a third intermediate result;

[0196] The second hash result is obtained by taking the modulo of the sum of the second intermediate result and the third intermediate result.

[0197] In some embodiments, the hierarchical mapping module 650 further includes:

[0198] Get the preset number of shards;

[0199] If the number of shards is less than a preset value, the second hash result is modulo the number of shards to obtain the shard number corresponding to the data to be extracted.

[0200] In some embodiments, the hierarchical mapping module 650 further includes:

[0201] If the number of fragments is greater than a preset value, the fragments of the specified number of fragments are divided into a first number of groups, wherein each group contains a second number of fragments; the product of the first number and the second number is the number of fragments.

[0202] Obtain the first weight factor in the disturbance factor vector;

[0203] The group number is generated by taking the modulo of the first quantity using the product of the second hash result and the first weight factor.

[0204] Obtain the second weighting factor from the disturbance factor vector;

[0205] The product of the group number and the second weight factor is summed with the second hash result to generate a summation result. Then, the summation result is modulo the second quantity to generate the fragment number within the group.

[0206] The product of the group number and the second quantity is summed with the shard number within the group to generate the shard number assigned to the data to be extracted.

[0207] In some embodiments, before taking the modulo of the first quantity using the product of the second hash result and the first weight factor to generate the group number, the first weight factor is subjected to sensitivity condition processing using a sensitivity adjustment factor to obtain the adjusted first weight factor.

[0208] Before summing the product of the group number and the second weight factor with the second hash result to generate the summation result, the second weight factor is subjected to sensitivity condition processing using the sensitivity adjustment factor to obtain the adjusted second weight factor.

[0209] In some embodiments, the system 600 further includes a sensitivity factor acquisition module, wherein the sensitivity factor acquisition module:

[0210] Receive a configuration request for the sensitivity adjustment factor; and in response to the configuration request, configure the sensitivity adjustment factor;

[0211] or,

[0212] Obtain the sensitivity adjustment factor in the perturbation factor vector.

[0213] In some embodiments, the extraction module 660:

[0214] Allocate a number of threads corresponding to the number of shards indicated by the sharding result to extract data from the data shards corresponding to the at least two databases;

[0215] The system 600 further includes a determining module, the determining module:

[0216] For each thread, obtain the historical data extraction information corresponding to that thread;

[0217] From the historical extraction data, determine the daily data extraction volume and data writing time period of the thread;

[0218] The time period of each day excluding the data writing time period is used as the data extraction time period of the thread, and the ratio of the daily data collection amount to the data extraction time period is used as the extraction rate of the thread.

[0219] The above-described scheme analyzes the characteristics of the sharded objects (i.e., string-type data keys) in at least two databases participating in big data distributed storage, generating a perturbation factor vector for hybrid perturbation and hierarchical mapping. Based on the elements in this vector, the hash results of the data keys are subjected to hybrid perturbation and hierarchical mapping to obtain the sharding results. This results in the following ways: First, regardless of the database, a unified hashing process and sharding logic are used, achieving unified sharding logic across different databases and making the sharding results portable, forming a unified sharding strategy under the distributed storage architecture. Second, the use of unified sharding logic in each database allows for comprehensive planning of data across databases, not limited to the worst-performing database, thus improving big data processing efficiency. Third, the sharding results fully consider the characteristics of the data keys, and these characteristics are combined to fully perturb and hierarchically map the initial hash results, increasing data randomness and preventing similar fields from being placed in the same shard, making subsequent processing simpler. Fourth, through hybrid perturbation and hierarchical mapping, the sharding becomes more balanced, reducing data skew, excessive pressure on some shard nodes, and idle resources on some nodes, which affect overall performance.

[0220] Those skilled in the art will understand that one or more embodiments of this application can be provided as a method, system, or computer program product. Therefore, one or more embodiments of this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, one or more embodiments of this application can take the form of a computer program product implemented on one or more computer-usable storage media (which may include, but are not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0221] In this application, "and / or" indicates that at least one of the two is present. The various embodiments in this application are described in a progressive manner, and similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the data processing device embodiments are basically similar to the method embodiments, so the description is relatively simple, and relevant parts can be referred to the description of the method embodiments.

[0222] While this application contains numerous specific implementation details, these should not be construed as limiting the scope of any disclosure or the scope of the claims, but rather are primarily used to describe the features of specific embodiments of a particular disclosure. Certain features described in the multiple embodiments of this application may also be implemented in combination in a single embodiment. Conversely, various features described in a single embodiment may also be implemented separately in multiple embodiments or in any suitable sub-combination. Furthermore, while features may function in certain combinations as described and even initially claimed in this way, one or more features from a claimed combination may be removed from that combination in some cases, and a claimed combination may refer to a sub-combination or a variation of a sub-combination.

[0223] Similarly, although operations are depicted in a specific order in the accompanying drawings, this should not be construed as requiring these operations to be performed in the specific order shown or sequentially, or requiring all illustrated operations to be performed to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the described embodiments should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0224] The above are merely preferred embodiments of one or more embodiments of this application and are not intended to limit the scope of one or more embodiments of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of one or more embodiments of this application should be included within the scope of protection of one or more embodiments of this application.

Claims

1. A data fragmentation and extraction method based on hierarchical hashing, applied to ETL tasks, characterized in that, The method includes: Obtain the data key value corresponding to the data to be extracted; the data key value is a string field; the data to be extracted comes from at least two databases; A perturbation factor vector is generated based on the character frequency distribution, character entropy value, and length entropy value of the string field; the perturbation factor vector includes at least one perturbation factor for performing mixed perturbation processing on the hash result of the data key value and at least one weight factor used in the hierarchical mapping of the hash result; The data key value is hashed to obtain a first hash result; Using the perturbation factor, the first hash result is subjected to a mixed perturbation process to obtain the second hash result; Using the weighting factor, the second hash result is mapped hierarchically to obtain the sharding result allocated to the data to be extracted; Data is extracted from the at least two databases based on the sharding results.

2. The data fragmentation extraction method based on hierarchical hashing according to claim 1, characterized in that, The generation of the perturbation factor vector based on the character frequency distribution, character entropy value, and length entropy value of the string field includes: Calculate the frequency of occurrence of each character in the string field; The character entropy value is generated based on the frequency of occurrence. Generate the length entropy value based on the character length of the string field; Based on the occurrence frequency, the character entropy value, and the length entropy value, a perturbation factor vector containing a preset number of factors is generated; wherein, if the occurrence frequency, the number of character entropy values, and the length entropy value are greater than the preset number, the largest value is selected as the factor; if the occurrence frequency, the number of character entropy values, and the length entropy value are less than the preset number, the occurrence frequency, the character entropy value, and the length entropy value are used as factors, and then a default value is filled in as the factor.

3. The data fragmentation extraction method based on hierarchical hashing according to claim 1, characterized in that, The hybrid perturbation is a three-layer perturbation, including XOR perturbation, amplification perturbation, and cyclic perturbation.

4. The data fragmentation extraction method based on hierarchical hashing according to claim 3, characterized in that, The step of using the perturbation factor to perform a mixed perturbation process on the first hash result to obtain the second hash result includes: Obtain the first perturbation factor in the perturbation factor vector, and perform an XOR operation between the first hash result and the first perturbation factor to generate a first intermediate result; Obtain the second perturbation factor from the perturbation factor vector, and use the product of the first intermediate result and the second perturbation factor as the second intermediate result; Obtain the third perturbation factor from the perturbation factor vector to generate an offset, and perform a cyclic offset on the first hash result according to the offset to obtain a third intermediate result; The second hash result is obtained by taking the modulo of the sum of the second intermediate result and the third intermediate result.

5. The data fragmentation extraction method based on hierarchical hashing according to claim 1, characterized in that, The step of using the weight factor to perform hierarchical mapping on the second hash result to obtain the sharding result allocated to the data to be extracted includes: Get the preset number of shards; If the number of shards is less than a preset value, the second hash result is modulo the number of shards to obtain the shard number corresponding to the data to be extracted.

6. The data fragmentation extraction method based on hierarchical hashing according to claim 5, characterized in that, The step of using the weight factor to perform hierarchical mapping on the second hash result to obtain the sharding result allocated to the data to be extracted includes: If the number of fragments is greater than a preset value, the fragments of the specified number of fragments are divided into a first number of groups, wherein each group contains a second number of fragments; the product of the first number and the second number is the number of fragments. Obtain the first weight factor in the disturbance factor vector; The group number is generated by taking the modulo of the first quantity using the product of the second hash result and the first weight factor. Obtain the second weighting factor from the disturbance factor vector; The product of the group number and the second weight factor is summed with the second hash result to generate a summation result. Then, the summation result is modulo the second quantity to generate the fragment number within the group. The product of the group number and the second quantity is summed with the shard number within the group to generate the shard number assigned to the data to be extracted.

7. The data fragmentation extraction method based on hierarchical hashing according to claim 6, characterized in that, Before generating a group number by taking the modulo of the first quantity using the product of the second hash result and the first weight factor, the first weight factor is subjected to sensitivity condition processing using a sensitivity adjustment factor to obtain the adjusted first weight factor. Before summing the product of the group number and the second weight factor with the second hash result to generate the summation result, the second weight factor is subjected to sensitivity condition processing using the sensitivity adjustment factor to obtain the adjusted second weight factor.

8. The data fragmentation extraction method based on hierarchical hashing according to claim 7, characterized in that, The method for obtaining the sensitivity adjustment factor includes: Receive a configuration request for the sensitivity adjustment factor; and in response to the configuration request, configure the sensitivity adjustment factor; or, Obtain the sensitivity adjustment factor in the perturbation factor vector.

9. The data fragmentation extraction method based on hierarchical hashing according to claim 1, characterized in that, The data extraction from the at least two databases based on the sharding results includes: Allocate a number of threads corresponding to the number of shards indicated by the sharding result to extract data from the data shards corresponding to the at least two databases; The methods for determining the data extraction time period and extraction rate for each thread include: For each thread, obtain the historical data extraction information corresponding to that thread; From the historical extraction data, determine the daily data extraction volume and data writing time period of the thread; The time period of each day other than the data writing time period is used as the data extraction time period of the thread, and the ratio of the daily data collection amount to the data extraction time period is used as the extraction rate of the thread.

10. A data fragmentation and extraction system based on hierarchical hashing, applied to ETL tasks, characterized in that, The system includes: The acquisition module acquires the data key value corresponding to the data to be extracted; the data key value is a string field; the data to be extracted comes from at least two databases; The generation module generates a perturbation factor vector based on the character frequency distribution, character entropy value, and length entropy value of the string field; the perturbation factor vector includes at least one perturbation factor for performing mixed perturbation processing on the hash result of the data key value and at least one weight factor used in the hierarchical mapping of the hash result; The hash module performs hash processing on the data key-value pairs to obtain a first hash result; The perturbation module uses the perturbation factor to perform a mixed perturbation process on the first hash result to obtain a second hash result; The hierarchical mapping module uses the weight factor to perform hierarchical mapping on the second hash result to obtain the sharding result allocated to the data to be extracted; The extraction module extracts data from the at least two databases based on the sharding results.

Images