A full volume update query slicing system and method for high concurrency environments

Abstract

The application relates to the technical field of data updating, and discloses a full-amount updating query slicing system and method for a high-concurrency environment, which comprises a pre-slicing unit, a Worker node allocation unit, a Worker node monitoring unit and a dynamic slicing adjustment unit.The pre-slicing unit is used for dividing data into a plurality of groups of logical fragments according to a preset slicing strategy.The Worker node allocation unit is used for performing matching analysis on the performance parameters of Worker nodes and the sizes of the logical fragments, and obtaining matching analysis results of the logical fragments and the Worker nodes.The Worker node monitoring unit is used for monitoring the processing speed of each Worker node.The dynamic slicing adjustment unit is used for dynamically adjusting the logical fragments according to the processing speed of each Worker node and the number of Worker node concurrency controls.The application can dynamically adjust the logical fragments through the processing speed of Worker nodes and the number of Worker node concurrency controls on the basis that the logical fragments meet data transmission, ensures the processing rate of Worker nodes, and improves the overall efficiency of a data full-amount updating query process.

Description

Technical Field

[0001] The present application relates to the technical field of data updating, and in particular to a full-update query slicing system and method for a high-concurrency environment. Background Art

[0002] In databases or distributed systems, massive data updates need to be processed. Full update queries refer to updating all relevant data at once during data processing or database operations, rather than incremental updates. This consumes a large amount of resources and affects the efficiency of data updates. Therefore, existing technologies divide data into multiple fragments (slices) through slicing to improve processing efficiency and performance.

[0003] Existing pre-sharding methods include range sharding and hash sharding. Range sharding mainly divides continuous intervals according to primary keys or timestamps (such as ID 1-1000 for shard 1, 1001-2000 for shard 2). Its advantage is high range query efficiency, but if new data is concentrated in the last shard, it is easy to generate hot spots, affecting the update efficiency. Hash sharding mainly distributes data evenly by calculating the hash value of the shard key (such as user ID) and then taking the modulus. Its advantage is good load balancing, but cross-shard scanning is required for range queries.

[0004] The existing slicing method mainly considers the data division process. However, in the actual operation process, it will be limited by the processing status of the Worker node and the number of concurrent Worker nodes. Therefore, how to dynamically slicing data while ensuring the processing rate of the Worker node is the fundamental problem to be solved by the present invention. Summary of the Invention

[0005] In order to dynamically shard data while ensuring the processing rate of Worker nodes, this application provides a full update query slicing system for high concurrency environments.

[0006] In the first aspect, the present application provides a full update query slicing system for a high-concurrency environment, which adopts the following technical solutions:

[0007] A full update query slicing system for high-concurrency environments, including:

[0008] A pre-slicing unit is used to divide the data into several groups of logical slicing according to a preset slicing strategy;

[0009] The Worker node allocation unit is used to perform matching analysis based on the performance parameters of the Worker node and the size of the logical shard, and obtain the matching analysis results between the logical shard and the Worker node;

[0010] Worker node monitoring unit, used to monitor the processing speed of each Worker node;

[0011] The dynamic slicing adjustment unit is used to dynamically adjust the logical slicing according to the processing speed of each Worker node and the number of concurrent Worker nodes.

[0012] By adopting the above technical solution, the performance parameters of the Worker node and the size of the logical shard are matched and analyzed, so that the logical shard is matched with the Worker node. Therefore, the logical segment with larger data volume can be processed preferentially by the Worker node with better processing capability, thereby improving the overall data full update query rate; in addition, the Worker node monitoring unit is used to monitor the processing speed of each Worker node, and then the logical shard is dynamically adjusted through the dynamic slice adjustment unit according to the processing speed of each Worker node and the number of concurrent control of the Worker node. On the basis of the logical shard meeting the data transmission, the logical shard can be dynamically adjusted according to the processing speed of the Worker node and the number of concurrent control of the Worker node, thereby ensuring the processing rate of the Worker node and improving the overall efficiency of the data full update query process.

[0013] Optionally, the preset slicing strategy includes:

[0014] Separate the data into hot and cold parts according to the data content, and obtain new data shards and old data shards;

[0015] Divide the new data shards in an orderly manner according to the first continuous interval of the timestamp to obtain the first shard group;

[0016] Divide the old data shards in an orderly manner according to the second continuous interval of the timestamp to obtain the second shard group;

[0017] The first continuous interval of timestamps is smaller than the second continuous interval of timestamps;

[0018] The shards in the first shard group and the second shard group are divided into several groups of logical shards.

[0019] By adopting the above technical solution, the new data shards are divided in order according to the first continuous interval of the timestamp to obtain the first shard group; the old data shards are divided in order according to the second continuous interval of the timestamp to obtain the second shard group; since the lengths of the first continuous interval of the timestamp and the second continuous interval of the timestamp are different, and the first continuous interval of the timestamp is smaller than the second continuous interval of the timestamp, for the newer data, it uses a finer granularity for sharding, thereby reducing the problem of hot spots generated during the full data update query process, and the shards in the first shard group and the second shard group are several groups of logical shards after division, and the several groups of logical shards obtained can meet the needs of the full data update query.

[0020] Optionally, the matching analysis process includes:

[0021] Sort several groups of logical shards in descending order of data volume to obtain a logical shard sequence;

[0022] Perform performance analysis on each Worker node based on performance parameters, and sort the nodes from strong to weak based on the performance analysis results to obtain a Worker node performance sequence.

[0023] Align the logical shard sequence with the Worker node performance sequence according to the serial number, and match the logical shards and Worker nodes in the same sequence.

[0024] By adopting the above technical solution, several groups of logical shards can be sorted in order from large to small according to the data volume, and can be allocated according to the sorting. Then, the performance analysis of each Worker node can be performed according to the performance parameters, and the processing capacity of each Worker node can be judged. They are sorted in order from strong to weak. Then, the logical shard sequence and the Worker node performance sequence are aligned according to the serial number, and the logical shards and Worker nodes in the same sequence are matched, so that logical shards with larger data volumes are given priority by Worker nodes with stronger processing capabilities. The logical shards can be allocated more reasonably, thereby improving the overall efficiency of the full update query process.

[0025] Optionally, the performance parameters include memory and network throughput of the Worker node;

[0026] The process of performance analysis for each Worker node includes:

[0027] The performance coefficient x of each Worker node is calculated using formulas (1)-(2);

[0028] (1)

[0029] (2)

[0030] Among them, p is the processing capacity value of the Worker node, is the benchmark value corresponding to the processing capacity value, s is the volatility coefficient of the Worker node, 、 is the adjustment coefficient, Rm is the memory value of the Worker node, and Rs is the preset resident memory value of the Worker node. is the efficiency coefficient, and satisfies <1, Nt is the throughput of the Worker node, Indicates taking and the minimum value in Nt.

[0031] By adopting the above technical solution, the performance coefficient x can be obtained, and then the performance capability of each Worker node can be ranked according to the size of the performance coefficient, so that logical shards with larger data volumes are preferentially processed by Worker nodes with stronger processing capabilities. Logical shards can be allocated more reasonably, thereby improving the overall efficiency of the full update query process.

[0032] Optionally, the process of obtaining the volatility coefficient s includes:

[0033] Perform continuous ping tests and network speed tests on the Worker nodes to obtain the ping test curve c(t) and network speed change curve h(t);

[0034] Select time periods for c(t) and h(t), and ensure that the selected time periods are of the same length. Collect N time points at fixed time intervals in the selected time periods.

[0035] By formula Calculate and obtain the volatility coefficient s;

[0036] Where i is a positive integer and i∈[1, N], is the value of c(t) corresponding to the i-th time point, For all The mean of is the value of h(t) corresponding to the i-th time point, For all The mean of Preset fixed comparison value for ping stability, Preset fixed comparison value for network speed stability.

[0037] By adopting the above technical solution, it is possible to comprehensively consider the factors of network delay and network speed stability and make a comprehensive judgment on the volatility of each Worker node. When the volatility of one or more of the obtained delay or network speed is large, the obtained volatility coefficient s value will also be relatively large. Therefore, when calculating the performance coefficient x for each Worker node, the calculation of the volatility coefficient s in this embodiment can provide a more accurate judgment basis.

[0038] Optionally, the dynamic adjustment includes shard merging and shard splitting.

[0039] By adopting the above technical solution, the processing process of each logical shard can be dynamically adjusted. While ensuring that the Worker node meets the needs, the overall efficiency of the full data update query process can be improved by sharding the logical shards.

[0040] Optionally, the process of dynamically adjusting the logical shards includes:

[0041] S1. Compare the processing speed of each Worker node monitored by the Worker node monitoring unit with a preset gradient threshold set {A1, A2, ..., Am}, where A1>A2>...>Am;

[0042] S2: When the processing speed of the Worker node is greater than A1, the logical shards to be subsequently processed by the corresponding Worker node are merged;

[0043] S3. When the processing speed ∈ [A1, A2], the logical sharding of the subsequent processing of the corresponding Worker node remains unchanged;

[0044] S4: When the processing speed of the Worker node is less than A2, the logical shards to be subsequently processed by the Worker node are pre-sharded.

[0045] S5. Compare the concurrency of the Worker nodes after the shard splitting pre-operation with the control number of the Worker node concurrency:

[0046] If the concurrent number of Worker nodes is less than or equal to the concurrent number control number of Worker nodes, the shard splitting pre-operation process is executed;

[0047] Otherwise, the Worker node processing speed is compared with A3, and the logical shards subsequently processed by the Worker node corresponding to the Worker node processing speed less than A3 are pre-sharded, and step S5 is repeated while successively lowering the preset gradient threshold. When the Worker node processing speed is less than Am and the condition of the Worker node concurrency number ≤ the Worker node concurrency control number is still not satisfied, the logical shards subsequently processed by the corresponding Worker node remain unchanged.

[0048] By adopting the above technical solution, by continuously lowering the speed standard, while ensuring that the number of Worker nodes is less than the control number of Worker node concurrency, logical nodes with slower processing speed are logically sharded as much as possible, thereby improving the overall efficiency of the full update query process as much as possible. When the Worker node processing speed is less than Am and the condition of the Worker node concurrency ≤ the control number of Worker node concurrency is still not met, the logical sharding of the subsequent processing of the corresponding Worker node remains unchanged, giving priority to ensuring the stability of the system. Through the above dynamic adjustment process, the control of the Worker node concurrency and the efficiency of the full update query process can be effectively balanced.

[0049] Optionally, the preset slicing strategy further includes:

[0050] First, shard by geographical location, and then divide by time range.

[0051] Optionally, the preset slicing strategy further includes:

[0052] First calculate the hash value of the shard key and distribute it modulo, then divide it by time range.

[0053] In summary, this application includes at least one of the following beneficial technical effects:

[0054] 1. The present invention matches the logical shards with the Worker nodes by matching and analyzing the performance parameters of the Worker nodes with the size of the logical shards. Therefore, the logical segments with larger data volumes can be processed preferentially by the Worker nodes with better processing capabilities, thereby improving the overall data full update query rate. In addition, the Worker node monitoring unit is used to monitor the processing speed of each Worker node, and then the dynamic slicing adjustment unit is used to dynamically adjust the logical shards according to the processing speed of each Worker node and the number of concurrent control of the Worker nodes. On the basis of the logical shards meeting the data transmission requirements, the logical shards can be dynamically adjusted according to the processing speed of the Worker nodes and the number of concurrent control of the Worker nodes, thereby ensuring the processing rate of the Worker nodes and improving the overall efficiency of the data full update query process.

[0055] 2. The present invention can divide the new data shards in an orderly manner according to the first continuous interval of the timestamp to obtain the first shard group through the preset slicing strategy set; divide the old data shards in an orderly manner according to the second continuous interval of the timestamp to obtain the second shard group; since the lengths of the first continuous interval of the timestamp and the second continuous interval of the timestamp are different, and the first continuous interval of the timestamp is smaller than the second continuous interval of the timestamp, for the newer data, it uses a finer granularity for sharding, thereby reducing the problem of hot spots generated during the full data update query process, and then the shards in the first shard group and the second shard group are divided into several groups of logical shards, and the obtained several groups of logical shards can meet the needs of the full data update query.

[0056] 3. The present invention can comprehensively judge the volatility of each Worker node by integrating the factors of network delay and network speed stability through the calculation process of the volatility coefficient. When the volatility of one or more of the obtained delay or network speed is large, the obtained volatility coefficient s value will also be relatively large. Therefore, when calculating the performance coefficient x of each Worker node, the calculation of the volatility coefficient s in this embodiment can provide a more accurate judgment basis.

[0057] 4. In the process of dynamically adjusting the logical sharding, the present invention ensures that the number of Worker nodes is less than the control number of Worker node concurrency by continuously lowering the speed standard, while performing logical sharding processing on the logical nodes with slower processing speed as much as possible, thereby improving the overall efficiency of the full update query process as much as possible. When the Worker node processing speed is less than Am and the condition of the Worker node concurrency ≤ the control number of Worker node concurrency is still not met, the logical sharding of the subsequent processing of the corresponding Worker node remains unchanged, giving priority to ensuring the stability of the system. Through the above-mentioned dynamic adjustment process, the control of the Worker node concurrency and the efficiency of the full update query process can be effectively balanced. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 This is a logical block diagram of a full update query pre-slicing system for a high-concurrency environment.

[0059] Figure 2 This is a flowchart of the steps of the full update query pre-slicing method for high-concurrency environments. DETAILED DESCRIPTION

[0060] Embodiments of the present application are described in detail below, examples of which are illustrated in the accompanying drawings.

[0061] Throughout this specification, reference to the terms "certain embodiments," "one embodiment," "some embodiments," "illustrative embodiments," "examples," "specific examples," or "some examples" means that a specific feature, structure, material, or characteristic described in conjunction with the embodiment or example is included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

[0062] The present application embodiment discloses a full update query slicing system for a high concurrency environment, referring to Figure 1 , including a pre-slicing unit, a Worker node allocation unit, a Worker node monitoring unit and a dynamic slicing adjustment unit, wherein the pre-slicing unit is used to divide the data into several groups of logical slicings according to a preset slicing strategy, wherein the preset slicing strategy adopts a composite slicing method according to the needs of data transmission, that is, a strategy of combining splitting methods in multiple ways, so that the obtained several groups of logical slicings can meet the needs of different data types. In addition, it should be noted that several groups of logical slicings will be dynamically adjusted later; in addition, the Worker node allocation unit is used to perform matching analysis based on the performance parameters of the Worker node and the size of the logical slicing, and obtain the matching analysis results of the logical slicing and the Worker node. Since the data processing capabilities of each Worker node will be different, and the preset slicing strategy cannot guarantee that the data size of each logical slicing is consistent, this embodiment uses the Worker node The performance parameters are matched and analyzed with the size of the logical shards, so that the logical shards are matched with the Worker nodes. Therefore, the logical segments with larger data volumes can be processed preferentially by the Worker nodes with better processing capabilities, thereby improving the overall data full update query rate. In addition, this embodiment also provides a Worker node monitoring unit, which monitors the processing speed of each Worker node through the Worker node monitoring unit, and then dynamically adjusts the logical shards through the dynamic slicing adjustment unit according to the processing speed of each Worker node and the number of concurrent control of the Worker nodes. Therefore, through the above process, the logical shards can be dynamically adjusted based on the processing speed of the Worker nodes and the number of concurrent control of the Worker nodes on the basis of the logical shards meeting the data transmission requirements, thereby ensuring the processing rate of the Worker nodes and improving the overall efficiency of the data full update query process.

[0063] In one embodiment, a preset slicing strategy is provided, which includes: first, performing cold and hot separation according to the content of the data to obtain new data shards and old data shards. It should be noted that the cold and hot separation indicates the newness of the data, and its specific separation process belongs to the existing common technology and will not be further described in detail; then, the new data shards are orderly divided according to the first continuous interval of the timestamp to obtain the first shard group; the old data shards are orderly divided according to the second continuous interval of the timestamp to obtain the second shard group; since the lengths of the first continuous interval of the timestamp and the second continuous interval of the timestamp are different, and the first continuous interval of the timestamp is smaller than the second continuous interval of the timestamp, for the newer data, it uses a finer granularity for sharding, thereby reducing the problem of hot spots generated during the full data update query process, and the shards in the first shard group and the second shard group are the groups of logical shards after division, so the obtained groups of logical shards can meet the needs of the full data update query.

[0064] In one embodiment, a matching analysis process is provided, including: sorting several groups of logical shards in order of data volume from large to small to obtain a logical shard sequence; performing performance analysis on each Worker node according to performance parameters, and sorting the shards in order from strong to weak according to the performance analysis results to obtain a Worker node performance sequence; aligning the logical shard sequence with the Worker node performance sequence according to sequence numbers, and matching logical shards and Worker nodes in the same sequence. Through the above technical solution, several groups of logical shards are first sorted in order of data volume from large to small, and can be allocated according to the sorting. Then, performance analysis is performed on each Worker node according to performance parameters to determine the processing capability of each Worker node, and sort them in order from strong to weak. Then, the logical shard sequence is aligned with the Worker node performance sequence according to sequence numbers, and logical shards and Worker nodes in the same sequence are matched, so that logical shards with larger data volumes are preferentially processed by Worker nodes with stronger processing capabilities. Logical shards can be allocated more reasonably, thereby improving the overall efficiency of the full update query process.

[0065] In one embodiment, a process for performing performance analysis on each Worker node is provided, wherein the performance parameters selected are the memory and network throughput of the Worker node for analysis, and the performance coefficient x of each Worker node is calculated by formulas (1)-(2);

[0066] (1)

[0067] (2)

[0068] Among them, Rm is the memory value of the Worker node, and Rs is the preset resident memory value of the Worker node. This parameter is set according to the configuration status of the Worker node. is the efficiency coefficient, and satisfies <1, efficiency coefficient According to the historical data and test data fitting settings of different Worker nodes, Nt is the throughput of the Worker node, Indicates taking The minimum value in Nt, so through the calculation process of formula (2), the processing capacity value p of the Worker node can be obtained. Through the processing capacity value p of the Worker node, the processing capacity of each Worker node can be judged, and the processing capacity value corresponding to the benchmark value can be set. , and then judge its relative capabilities. It should be noted that the processing capability value corresponds to the benchmark value It is set based on empirical data. In addition, s is the volatility coefficient of the Worker node. It can be calculated according to the common method of calculating node volatility in the existing technology, or the volatility evaluation strategy can be set according to the needs. There is no restriction here. 、 To adjust the coefficient, it is obtained by fitting the test data according to different evaluation methods of volatility and the influence weight of node volatility and processing capacity. The performance coefficient x is obtained through the calculation process of the above formulas (1)-(2), and then the performance capacity of each Worker node is ranked according to the size of the performance coefficient, so that the logical shards with larger data volume are preferentially allocated by the Worker nodes with stronger processing capacity. The logical shards can be allocated more reasonably, thereby improving the overall efficiency of the full update query process.

[0069] In one embodiment, a process for obtaining a volatility coefficient s is provided, including: performing a continuous ping test and a network speed test on a Worker node, respectively, to obtain a ping test curve c(t) and a network speed change curve h(t); selecting a time period for c(t) and h(t), ensuring that the selected time periods are of the same length, and collecting N time points at fixed time intervals in the selected time period; and calculating the value of the time period using the formula Calculate the volatility coefficient s; where i is a positive integer and i∈[1, N], is the value of c(t) corresponding to the i-th time point, For all The mean of is the value of h(t) corresponding to the i-th time point, For all The mean of Preset fixed comparison value for ping stability, Set fixed comparison values ​​for network speed stability, including ping stability. and network speed stability preset fixed comparison value They are all obtained by fitting empirical data. Therefore, through the calculation process of the above-mentioned volatility coefficient s, the factors of network delay and network speed stability can be comprehensively considered to comprehensively judge the volatility of each Worker node. When the volatility of one or more of the obtained delay or network speed is large, the obtained volatility coefficient s value will also be relatively large. Therefore, when calculating the performance coefficient x for each Worker node, the calculation of the volatility coefficient s in this embodiment can provide a more accurate judgment basis.

[0070] In one embodiment, dynamic adjustment includes shard merging and shard splitting. Through the shard merging and shard splitting process of logical shards, the processing process of each logical shard can be dynamically adjusted. While ensuring that the Worker node meets the demand, the overall efficiency of the full data update query process can be improved through the sharding of logical shards.

[0071] In one embodiment, the process of dynamically adjusting the logical shards includes:

[0072] S1. Compare the processing speed of each Worker node monitored by the Worker node monitoring unit with the preset gradient threshold set {A1, A2, ..., Am}, wherein the preset gradient threshold set {A1, A2, ..., Am} is set according to empirical data and satisfies A1>A2>...>Am. Therefore, in step S2, when the processing speed of the Worker node is greater than A1, the logical shards to be subsequently processed by the corresponding Worker node are merged, thereby ensuring the processing efficiency while reducing the use of Worker nodes. On the one hand, the vacant Worker nodes can be provided to the logical shards with slower processing speed, and on the other hand, On the other hand, retaining idle Worker nodes can avoid exhaustion of database connections and ensure stable operation of the system; S3, when the processing speed ∈ [A1, A2], it means that the processing of logical allocation is within a reasonable range, so the logical shards of subsequent processing of the corresponding Worker node remain unchanged; S4, when the processing speed of the Worker node is less than A2, it is necessary to split the node to ensure the overall efficiency of the data full update query process, and at the same time, in order to avoid exhaustion of database connections due to excessive occupation of Worker nodes, this embodiment performs a shard splitting pre-operation on the logical shards of subsequent processing of the corresponding Worker node; S5, split the shards Compare the concurrency of the Worker nodes after the pre-sharding operation with the control number of the concurrency of the Worker nodes: If the concurrency of the Worker nodes is ≤ the control number of the concurrency of the Worker nodes, it means that the concurrency of the Worker nodes after the shard splitting pre-operation is within the controllable range, so the shard splitting pre-operation process is performed to improve the overall efficiency of the full update query process; otherwise, compare the processing speed of the Worker nodes with A3, and perform shard splitting pre-operation on the logical shards subsequently processed by the Worker nodes corresponding to the Worker node processing speed < A3, repeat step S5 and reduce the preset gradient threshold in turn, so by continuously reducing The speed standard approach ensures that the number of Worker nodes is less than the Worker node concurrency control number, while performing logical sharding processing on logical nodes with slower processing speeds as much as possible, thereby improving the overall efficiency of the full update query process as much as possible. When the Worker node processing speed is less than Am and the condition of Worker node concurrency ≤ Worker node concurrency control number is still not met, the logical sharding of the subsequent processing of the corresponding Worker node remains unchanged, giving priority to ensuring system stability. Through the above dynamic adjustment process, the control of Worker node concurrency and the efficiency of the full update query process can be effectively balanced.

[0073] In addition, the preset slicing strategy in the above scheme also includes: first slicing according to geographical location, and then dividing by time range; and first calculating the hash value of the slicing key and distributing it by modulus, and then dividing by time range. The logical slicing obtained by the above preset slicing strategy can meet different needs. At the same time, due to the differences in the size of logical slicing, it is also applicable to the slicing method in the present invention.

[0074] In one embodiment, see the attached Figure 2 A full-update query slicing method for a high-concurrency environment is provided. The method adopts a full-update query slicing system for a high-concurrency environment, including: step 1, dividing the data into several groups of logical slicings according to a preset slicing strategy through a pre-slicing unit; step 2, performing a matching analysis based on the performance parameters of the Worker node and the size of the logical slicing through a Worker node allocation unit to obtain a matching analysis result between the logical slicing and the Worker node; step 3, monitoring the processing speed of each Worker node through a Worker node monitoring unit; step 4, dynamically adjusting the logical slicing according to the processing speed of each Worker node and the number of concurrent control of the Worker node through a dynamic slicing adjustment unit. By matching and analyzing the performance parameters of the Worker node with the size of the logical shard, the logical shard is matched with the Worker node. Therefore, the logical segments with larger data volumes can be processed preferentially by the Worker node with better processing capabilities, thereby improving the overall data full update query rate. In addition, the Worker node monitoring unit is used to monitor the processing speed of each Worker node, and then the dynamic shard adjustment unit is used to dynamically adjust the logical shard based on the processing speed of each Worker node and the number of concurrent control of the Worker node. On the basis of the logical shard meeting the data transmission requirements, the logical shard can be dynamically adjusted based on the processing speed of the Worker node and the number of concurrent control of the Worker node, thereby ensuring the processing rate of the Worker node and improving the overall efficiency of the data full update query process.

[0075] Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limitations on the present application. Ordinary technicians in this field can change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A full update query slicing system for high concurrency environments, characterized by: include: A pre-slicing unit is used to divide the data into several groups of logical slicing according to a preset slicing strategy; The Worker node allocation unit is used to perform matching analysis based on the performance parameters of the Worker node and the size of the logical shard, and obtain the matching analysis results between the logical shard and the Worker node; Worker node monitoring unit, used to monitor the processing speed of each Worker node; Dynamic slicing adjustment unit, used to dynamically adjust logical slicing based on the processing speed of each Worker node and the number of concurrent Worker nodes; The preset slicing strategy includes: Separate the data into hot and cold parts according to the data content, and obtain new data shards and old data shards; Divide the new data shards in an orderly manner according to the first continuous interval of the timestamp to obtain the first shard group; Divide the old data shards in an orderly manner according to the second continuous interval of the timestamp to obtain the second shard group; The first continuous interval of timestamps is smaller than the second continuous interval of timestamps; The shards in the first shard group and the second shard group are divided into several groups of logical shards; The matching analysis process includes: Sort several groups of logical shards in descending order of data volume to obtain a logical shard sequence; Perform performance analysis on each Worker node based on performance parameters, and sort the nodes from strong to weak based on the performance analysis results to obtain a Worker node performance sequence. Align the logical shard sequence with the Worker node performance sequence based on the sequence number, and match the logical shards and Worker nodes in the same sequence; The performance parameters include the memory and network throughput of the Worker node; The process of performance analysis for each Worker node includes: The performance coefficient x of each Worker node is calculated using formulas (1)-(2); (1) (2) Among them, p is the processing capacity value of the Worker node, is the benchmark value corresponding to the processing capacity value, s is the volatility coefficient of the Worker node, 、 is the adjustment coefficient, Rm is the memory value of the Worker node, and Rs is the preset resident memory value of the Worker node. is the efficiency coefficient, and satisfies <1, Nt is the throughput of the Worker node, Indicates taking and the minimum value in Nt, is the allowed throughput corresponding to the unit memory value; The dynamic adjustment includes shard merging and shard splitting; The process of dynamically adjusting logical shards includes: S1. Compare the processing speed of each Worker node monitored by the Worker node monitoring unit with a preset gradient threshold set {A1, A2, ..., Am}, where A1>A2>...>Am; S2: When the processing speed of the Worker node is greater than A1, the logical shards to be subsequently processed by the corresponding Worker node are merged; S3. When the processing speed ∈ [A1, A2], the logical sharding of the subsequent processing of the corresponding Worker node remains unchanged; S4: When the processing speed of the Worker node is less than A2, the logical shards to be subsequently processed by the Worker node are pre-sharded. S5. Compare the concurrency of the Worker nodes after the shard splitting pre-operation with the control number of the Worker node concurrency: If the concurrent number of Worker nodes is less than or equal to the concurrent number control number of Worker nodes, the shard splitting pre-operation process is executed; Otherwise, the Worker node processing speed is compared with A3, and the logical shards subsequently processed by the Worker node corresponding to the Worker node processing speed less than A3 are pre-sharded, and step S5 is repeated while successively lowering the preset gradient threshold. When the Worker node processing speed is less than Am and the condition of the Worker node concurrency number ≤ the Worker node concurrency control number is still not satisfied, the logical shards subsequently processed by the corresponding Worker node remain unchanged.

2. A full update query slicing system for a high-concurrency environment according to claim 1, characterized in that: The process of obtaining the volatility coefficient s includes: Perform continuous ping tests and network speed tests on the Worker nodes to obtain the ping test curve c(t) and network speed change curve h(t); Select time periods for c(t) and h(t), and ensure that the selected time periods are of the same length. Collect N time points at fixed time intervals in the selected time periods. By formula Calculate and obtain the volatility coefficient s; Where i is a positive integer and i∈[1, N], is the value of c(t) corresponding to the i-th time point, For all The mean of is the value of h(t) corresponding to the i-th time point, For all The mean of Preset fixed comparison value for ping stability, Preset fixed comparison value for network speed stability.

3. The full update query slicing system for a high-concurrency environment according to claim 1 is characterized in that: The preset slicing strategy also includes: First, shard by geographical location, and then divide by time range.

4. The full update query slicing system for a high-concurrency environment according to claim 1 is characterized in that: The preset slicing strategy also includes: First calculate the hash value of the shard key and distribute it modulo, then divide it by time range.

5. A full update query slicing method for a high-concurrency environment, characterized in that: The method adopts a full update query slicing system for a high-concurrency environment according to any one of claims 1 to 4, comprising: Step 1: Divide the data into several logical shards according to the preset sharding strategy through the pre-slicing unit; The preset slicing strategy includes: Separate the data into hot and cold parts according to the data content, and obtain new data shards and old data shards; Divide the new data shards in an orderly manner according to the first continuous interval of the timestamp to obtain the first shard group; Divide the old data shards in an orderly manner according to the second continuous interval of the timestamp to obtain the second shard group; The first continuous interval of timestamps is smaller than the second continuous interval of timestamps; The shards in the first shard group and the second shard group are divided into several groups of logical shards; Step 2: The Worker node allocation unit performs a matching analysis based on the performance parameters of the Worker node and the size of the logical shard, and obtains the matching analysis results between the logical shard and the Worker node; The matching analysis process includes: Sort several groups of logical shards in descending order of data volume to obtain a logical shard sequence; Perform performance analysis on each Worker node based on performance parameters, and sort the nodes from strong to weak based on the performance analysis results to obtain a Worker node performance sequence. Align the logical shard sequence with the Worker node performance sequence based on the sequence number, and match the logical shards and Worker nodes in the same sequence; The performance parameters include the memory and network throughput of the Worker node; The process of performance analysis for each Worker node includes: The performance coefficient x of each Worker node is calculated using formulas (1)-(2); (1) (2) Among them, p is the processing capacity value of the Worker node, is the benchmark value corresponding to the processing capacity value, s is the volatility coefficient of the Worker node, 、 is the adjustment coefficient, Rm is the memory value of the Worker node, and Rs is the preset resident memory value of the Worker node. is the efficiency coefficient, and satisfies <1, Nt is the throughput of the Worker node, Indicates taking and the minimum value in Nt, is the allowed throughput corresponding to the unit memory value; Step 3: Monitor the processing speed of each Worker node through the Worker node monitoring unit; Step 4: Dynamically adjust the logical shards using the dynamic sharding adjustment unit based on the processing speed of each Worker node and the number of concurrent Worker nodes. The dynamic adjustment includes shard merging and shard splitting; The process of dynamically adjusting logical shards includes: S1. Compare the processing speed of each Worker node monitored by the Worker node monitoring unit with a preset gradient threshold set {A1, A2, ..., Am}, where A1>A2>...>Am; S2: When the processing speed of the Worker node is greater than A1, the logical shards to be subsequently processed by the corresponding Worker node are merged; S3. When the processing speed ∈ [A1, A2], the logical sharding of the subsequent processing of the corresponding Worker node remains unchanged; S4: When the processing speed of the Worker node is less than A2, the logical shards to be subsequently processed by the Worker node are pre-sharded. S5. Compare the concurrency of the Worker nodes after the shard splitting pre-operation with the control number of the Worker node concurrency: If the concurrent number of Worker nodes is less than or equal to the concurrent number control number of Worker nodes, the shard splitting pre-operation process is executed; Otherwise, the Worker node processing speed is compared with A3, and the logical shards subsequently processed by the Worker node corresponding to the Worker node processing speed less than A3 are pre-sharded, and step S5 is repeated while successively lowering the preset gradient threshold. When the Worker node processing speed is less than Am and the condition of the Worker node concurrency number ≤ the Worker node concurrency control number is still not satisfied, the logical shards subsequently processed by the corresponding Worker node remain unchanged.

Images