Full-quantity update query slicing system and method for high-concurrency environment

Abstract

The invention relates to the technical field of data updating, and discloses a full-amount updating query slicing system and method for a high-concurrency environment, and the system comprises a pre-slicing unit which is used for dividing data into a plurality of groups of logic fragments according to a preset slicing strategy; the Worker node distribution unit is used for carrying out matching analysis according to the performance parameters of the Worker nodes and the sizes of the logic fragments to obtain matching analysis results of the logic fragments and the Worker nodes; the Worker node monitoring unit is used for monitoring the processing speed of each Worker node; and the dynamic slice adjusting unit is used for dynamically adjusting the logic slices according to the processing speed of each Worker node and the concurrent number control number of the Worker nodes. According to the method, the logic fragments can be dynamically adjusted through the processing speed of the Worker nodes and the concurrent number control number of the Worker nodes on the basis that the logic fragments meet data transmission, the processing speed of the Worker nodes is guaranteed, and the overall efficiency of the data full-amount updating query process is improved.

Description

Technical Field

[0001] The present application relates to the technical field of data updating, and in particular to a full-update query slicing system and method for a high-concurrency environment. Background Art

[0002] In a database or distributed system, massive data updates need to be processed. A full update query refers to updating all relevant data at once during data processing or database operations, rather than incremental updates. This will take up a lot of resources and affect the efficiency of data updates. Therefore, existing technologies divide data into multiple fragments (slices) through slicing to improve processing efficiency and performance.

[0003] Existing pre-slicing methods include range sharding and hash sharding. Range sharding mainly divides continuous intervals according to primary keys or timestamps (such as ID 1-1000 is shard 1, 1001-2000 is shard 2). Its advantage is high range query efficiency, but if new data is concentrated in the last shard, hot spots are easily generated, affecting the update efficiency; hash sharding mainly calculates the hash value of the shard key (such as user ID) and then takes the modulus to evenly distribute data. Its advantage is good load balancing, but cross-shard scanning is required for range queries.

[0004] The existing slicing method mainly considers the data division process. However, in the actual operation process, it will be limited due to the processing status of the Worker node and the concurrent number of the Worker node. Therefore, how to dynamically slice the data while ensuring the processing rate of the Worker node is the fundamental problem to be solved by the present invention. Summary of the invention

[0005] In order to dynamically shard data while ensuring the processing rate of Worker nodes, the present application provides a full update query slicing system for a high concurrency environment.

[0006] In a first aspect, the present application provides a full update query slice system for a high-concurrency environment, which adopts the following technical solution:

[0007] A full update query slicing system for a high-concurrency environment, comprising:

[0008] A pre-slicing unit is used to divide the data into several groups of logical slicing according to a preset slicing strategy;

[0009] The Worker node allocation unit is used to perform matching analysis based on the performance parameters of the Worker node and the size of the logical shard, and obtain the matching analysis results between the logical shard and the Worker node;

[0010] Worker node monitoring unit, used to monitor the processing speed of each Worker node;

[0011] The dynamic slice adjustment unit is used to dynamically adjust the logical slices according to the processing speed of each Worker node and the number of concurrent control of Worker nodes.

[0012] By adopting the above technical solution, the performance parameters of the Worker node and the size of the logical shard are matched and analyzed, so that the logical shard is matched with the Worker node. Therefore, the logical segment with a larger data volume can be processed preferentially by the Worker node with better processing capability, thereby improving the overall data full update query rate; in addition, the processing speed of each Worker node is monitored by the Worker node monitoring unit, and then the logical shard is dynamically adjusted by the dynamic slice adjustment unit according to the processing speed of each Worker node and the number of concurrent control of the Worker node. On the basis of the logical shard meeting the data transmission, the logical shard can be dynamically adjusted by the processing speed of the Worker node and the number of concurrent control of the Worker node, thereby ensuring the processing rate of the Worker node and improving the overall efficiency of the data full update query process.

[0013] Optionally, the preset slicing strategy includes:

[0014] Separate the data into hot and cold parts according to the data content, and obtain new data shards and old data shards;

[0015] The new data shards are divided in order according to the first continuous interval of the timestamp to obtain the first shard group;

[0016] The old data shards are divided in order according to the second continuous interval of the timestamp to obtain the second shard group;

[0017] The first continuous interval of timestamps is smaller than the second continuous interval of timestamps;

[0018] The shards in the first shard group and the second shard group are divided into several groups of logical shards.

[0019] By adopting the above technical solution, the new data shards are divided in order according to the first continuous interval of the timestamp to obtain the first shard group; the old data shards are divided in order according to the second continuous interval of the timestamp to obtain the second shard group; since the lengths of the first continuous interval of the timestamp and the second continuous interval of the timestamp are different, and the first continuous interval of the timestamp is smaller than the second continuous interval of the timestamp, for the newer data, it uses a finer granularity for sharding, thereby reducing the problem of hot spots generated during the full data update query process, and the shards in the first shard group and the second shard group are several groups of logical shards after division, and the several groups of logical shards obtained can meet the needs of the full data update query.

[0020] Optionally, the matching analysis process includes:

[0021] Sort several groups of logical shards in descending order of data volume to obtain a logical shard sequence;

[0022] Perform performance analysis on each Worker node according to the performance parameters, sort the nodes from strong to weak according to the performance analysis results, and obtain the Worker node performance sequence;

[0023] Align the logical shard sequence with the Worker node performance sequence according to the sequence number, and match the logical shards and Worker nodes in the same sequence.

[0024] By adopting the above technical solution, several groups of logical shards can be sorted in order from large to small according to the data volume, and can be allocated according to the sorting. Then, the performance analysis of each Worker node can be performed according to the performance parameters, and the processing capacity of each Worker node can be judged. They are sorted in order from strong to weak, and then the logical shard sequence and the Worker node performance sequence are aligned according to the serial number, and the logical shards and Worker nodes of the same sequence are matched, so that the logical shards with larger data volume are preferentially handled by the Worker nodes with stronger processing capabilities. The logical shards can be allocated more reasonably, thereby improving the overall efficiency of the full update query process.

[0025] Optionally, the performance parameters include memory and network throughput of the Worker node;

[0026] The process of performance analysis for each Worker node includes:

[0027] The performance coefficient x of each Worker node is calculated using formulas (1)-(2);

[0028] (1)

[0029] (2)

[0030] Where p is the processing capacity value of the Worker node. is the benchmark value corresponding to the processing capacity value, s is the volatility coefficient of the Worker node, , is the adjustment coefficient, Rm is the memory value of the Worker node, and Rs is the preset resident memory value of the Worker node. is the efficiency coefficient and satisfies <1, Nt is the throughput of the Worker node, Indicates taking with the minimum value among Nt.

[0031] By adopting the above technical solution, the performance coefficient x can be obtained, and then the performance capability of each Worker node can be ranked according to the size of the performance coefficient, so that logical shards with larger data volumes are preferentially processed by Worker nodes with stronger processing capabilities. The logical shards can be allocated more reasonably, thereby improving the overall efficiency of the full update query process.

[0032] Optionally, the process of obtaining the volatility coefficient s includes:

[0033] Perform continuous ping tests and network speed tests on the Worker nodes respectively to obtain the ping test curve c(t) and network speed change curve h(t);

[0034] Select time periods for c(t) and h(t), and ensure that the lengths of the selected time periods are the same. Collect N time points at fixed time intervals in the selected time periods.

[0035] By formula Calculate and obtain the volatility coefficient s;

[0036] Where i is a positive integer and i∈[1,N], is the value of c(t) corresponding to the i-th time point, For all The mean of is the value of h(t) corresponding to the i-th time point, For all The mean of Preset fixed comparison values ​​for ping stability. Preset fixed comparison value for network speed stability.

[0037] By adopting the above technical solution, it is possible to comprehensively consider the factors of network delay and network speed stability, and comprehensively judge the volatility of each Worker node. When the volatility of one or more of the obtained delay or network speed is large, the obtained volatility coefficient s value will also be larger. Therefore, when calculating the performance coefficient x of each Worker node, the calculation of the volatility coefficient s in this embodiment can provide a more accurate judgment basis.

[0038] Optionally, the dynamic adjustment includes shard merging and shard splitting.

[0039] By adopting the above technical solution, the processing process of each logical shard can be dynamically adjusted. While ensuring that the Worker node meets the needs, the overall efficiency of the full data update query process can be improved through the sharding of logical shards.

[0040] Optionally, the process of dynamically adjusting the logical shards includes:

[0041] S1, comparing the processing speed of each Worker node monitored by the Worker node monitoring unit with a preset gradient threshold set {A1, A2, ..., Am}, where A1>A2>...>Am;

[0042] S2: When the processing speed of the Worker node is greater than A1, the logical shards to be subsequently processed by the corresponding Worker node are merged;

[0043] S3. When the processing speed ∈ [A1, A2], the logical sharding of the subsequent processing of the corresponding Worker node remains unchanged;

[0044] S4: When the processing speed of the Worker node is less than A2, the logical shards to be subsequently processed by the corresponding Worker node will be pre-sharded;

[0045] S5. Compare the concurrent number of Worker nodes after the shard splitting pre-operation with the concurrent number control number of Worker nodes:

[0046] If the concurrent number of Worker nodes is ≤ the concurrent number of Worker nodes, the shard splitting pre-operation process is executed;

[0047] Otherwise, the processing speed of the Worker node is compared with A3, and the logical shards subsequently processed by the Worker node corresponding to the Worker node processing speed < A3 are pre-sharded, and step S5 is repeated while the preset gradient threshold is lowered in sequence; when the Worker node processing speed < Am still does not meet the condition that the number of concurrent Worker nodes ≤ the number of concurrent Worker nodes controlled, the logical shards subsequently processed by the corresponding Worker node remain unchanged.

[0048] By adopting the above technical solution, by continuously lowering the speed standard, while ensuring that the number of Worker nodes is less than the control number of Worker node concurrency, logical nodes with slower processing speed are logically sharded as much as possible, the overall efficiency of the full update query process is improved as much as possible. When the Worker node processing speed is less than Am and the condition of the Worker node concurrency ≤ the control number of Worker node concurrency is still not met, the logical sharding of the subsequent processing of the corresponding Worker node is kept unchanged, giving priority to ensuring the stability of the system. Through the above dynamic adjustment process, the control of the Worker node concurrency and the efficiency of the full update query process can be effectively balanced.

[0049] Optionally, the preset slicing strategy also includes:

[0050] First, shard by geographical location, and then divide by time range.

[0051] Optionally, the preset slicing strategy also includes:

[0052] First, the hash value of the shard key is calculated and distributed modulo, and then divided by time range.

[0053] In summary, the present application includes at least one of the following beneficial technical effects:

[0054] 1. The present invention matches the logical shards with the Worker nodes by matching and analyzing the performance parameters of the Worker nodes with the size of the logical shards. Therefore, the logical segments with larger data volumes can be processed preferentially by the Worker nodes with better processing capabilities, thereby improving the overall data full update query rate. In addition, the processing speed of each Worker node is monitored by the Worker node monitoring unit, and then the logical shards are dynamically adjusted by the dynamic slicing adjustment unit according to the processing speed of each Worker node and the concurrent number control quantity of the Worker nodes. On the basis that the logical shards meet the data transmission requirements, the logical shards can be dynamically adjusted according to the processing speed of the Worker nodes and the concurrent number control quantity of the Worker nodes, thereby ensuring the processing rate of the Worker nodes and improving the overall efficiency of the data full update query process.

[0055] 2. The present invention can divide the new data shards in an orderly manner according to the first continuous interval of the timestamp to obtain the first shard group through the preset slicing strategy set; divide the old data shards in an orderly manner according to the second continuous interval of the timestamp to obtain the second shard group; since the lengths of the first continuous interval of the timestamp and the second continuous interval of the timestamp are different, and the first continuous interval of the timestamp is smaller than the second continuous interval of the timestamp, for the newer data, it uses a finer granularity for sharding, thereby reducing the problem of hot spots generated during the full data update query process, and then the shards in the first shard group and the second shard group are divided into several groups of logical shards, and the several groups of logical shards obtained can meet the needs of the full data update query.

[0056] 3. The present invention can comprehensively judge the volatility of each Worker node by taking into account the factors of network delay and network speed stability through the calculation process of the volatility coefficient. When the volatility of one or more of the delay or network speed is large, the obtained volatility coefficient s value will also be large. Therefore, when calculating the performance coefficient x of each Worker node, the calculation of the volatility coefficient s in this embodiment can provide a more accurate basis for judgment.

[0057] 4. In the process of dynamically adjusting the logical sharding, the present invention ensures that the number of Worker nodes is less than the control number of concurrent Worker nodes by continuously lowering the speed standard, and at the same time, performs logical sharding processing on the logical nodes with slower processing speed as much as possible, so as to improve the overall efficiency of the full update query process as much as possible. When the Worker node processing speed is less than Am and still does not meet the condition that the concurrent number of Worker nodes is ≤ the control number of concurrent Worker nodes, the logical sharding of the subsequent processing of the corresponding Worker node is kept unchanged, and the stability of the system is guaranteed first. Through the above-mentioned dynamic adjustment process, the control of the concurrent number of Worker nodes and the efficiency of the full update query process can be effectively balanced. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 This is a logical block diagram of the full update query pre-slicing system for high-concurrency environments;

[0059] Figure 2 This is a flowchart of the steps of the full update query pre-slicing method for a high-concurrency environment. DETAILED DESCRIPTION

[0060] Embodiments of the present application are described in detail below, examples of which are shown in the accompanying drawings.

[0061] In the description of this specification, the description with reference to the terms "certain embodiments", "one embodiment", "some embodiments", "illustrative embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials, or characteristics described in conjunction with the embodiments or examples are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0062] The present application embodiment discloses a full update query slice system for a high concurrency environment, referring to Figure 1 , including a pre-slicing unit, a Worker node allocation unit, a Worker node monitoring unit and a dynamic slicing adjustment unit, wherein the pre-slicing unit is used to divide the data into several groups of logical slicings according to a preset slicing strategy, wherein the preset slicing strategy adopts a composite slicing method according to the demand for data transmission, that is, a strategy of combining splitting methods in multiple ways, so that the obtained several groups of logical slicings can meet the needs of different data types. In addition, it should be noted that several groups of logical slicings will be dynamically adjusted later; in addition, the Worker node allocation unit is used to perform matching analysis according to the performance parameters of the Worker node and the size of the logical slicing, and obtain the matching analysis results of the logical slicing and the Worker node. Since the data processing capability of each Worker node will be different, and the data size of each logical slicing cannot be guaranteed to be consistent according to the preset slicing strategy, this embodiment uses the Worker node The performance parameters are matched and analyzed with the size of the logical shards so that the logical shards are matched with the Worker nodes. Therefore, the logical segments with larger data volumes can be processed preferentially by the Worker nodes with better processing capabilities, thereby improving the overall data full update query rate. In addition, this embodiment further provides a Worker node monitoring unit, which monitors the processing speed of each Worker node through the Worker node monitoring unit, and then dynamically adjusts the logical shards through the dynamic slice adjustment unit according to the processing speed of each Worker node and the concurrent number control quantity of the Worker nodes. Therefore, through the above process, the logical shards can be dynamically adjusted through the processing speed of the Worker nodes and the concurrent number control quantity of the Worker nodes on the basis of satisfying the data transmission of the logical shards, thereby ensuring the processing rate of the Worker nodes and improving the overall efficiency of the data full update query process.

[0063] In one embodiment, a preset slicing strategy is provided, which includes: firstly, performing cold and hot separation according to the content of the data to obtain new data shards and old data shards. It should be noted that the cold and hot separation indicates the newness or oldness of the data, and its specific separation process belongs to the existing common technology, which will not be further described in detail; then, the new data shards are orderly divided according to the first continuous interval of the timestamp to obtain the first shard group; the old data shards are orderly divided according to the second continuous interval of the timestamp to obtain the second shard group; since the lengths of the first continuous interval of the timestamp and the second continuous interval of the timestamp are different, and the first continuous interval of the timestamp is smaller than the second continuous interval of the timestamp, for the newer data, it uses a finer granularity for sharding, thereby reducing the problem of hot spots generated during the full data update query process, and the shards in the first shard group and the second shard group are the several groups of logical shards after the division, so the several groups of logical shards obtained can meet the needs of the full data update query.

[0064] In one embodiment, a matching analysis process is provided, including: sorting a plurality of groups of logical shards in order of data volume from large to small to obtain a logical shard sequence; performing performance analysis on each Worker node according to performance parameters, and sorting them in order from strong to weak according to the performance analysis results to obtain a Worker node performance sequence; aligning the logical shard sequence with the Worker node performance sequence according to the sequence number, and matching the logical shards and Worker nodes of the same sequence. Through the above technical solution, firstly, sorting a plurality of groups of logical shards in order of data volume from large to small, and being able to allocate them according to the sorting, and then performing performance analysis on each Worker node according to the performance parameters, and being able to judge the processing capacity of each Worker node, and sorting them in order from strong to weak, and then aligning the logical shard sequence with the Worker node performance sequence according to the sequence number, and matching the logical shards and Worker nodes of the same sequence, so that the logical shards with larger data volume are preferentially handled by the Worker nodes with stronger processing capacity, and the logical shards can be allocated more reasonably, thereby improving the overall efficiency of the full update query process.

[0065] In one embodiment, a process of performing performance analysis on each Worker node is provided, wherein the performance parameters selected are the memory and network throughput of the Worker node for analysis, and the performance coefficient x of each Worker node is calculated by formulas (1)-(2);

[0066] (1)

[0067] (2)

[0068] Among them, Rm is the memory value of the Worker node, and Rs is the preset resident memory value of the Worker node. This parameter is set according to the configuration status of the Worker node. is the efficiency coefficient and satisfies <1, efficiency coefficient According to the historical data and test data of different Worker nodes, Nt is the throughput of the Worker node. Indicates taking and the minimum value in Nt. Therefore, through the calculation process of formula (2), the processing capacity value p of the Worker node can be obtained. Through the processing capacity value p of the Worker node, the processing capacity of each Worker node can be judged. The processing capacity value corresponding to the benchmark value is set , and then judge its relative ability. It should be noted that the processing ability value corresponds to the benchmark value. It is set according to empirical data. In addition, s is the volatility coefficient of the Worker node. It can be calculated according to the common calculation method of node volatility in the prior art, or the volatility evaluation strategy can be set according to the needs. There is no restriction here. , The adjustment coefficient is obtained by fitting the test data according to different volatility evaluation methods and the influence weights of node volatility and processing capacity. The performance coefficient x is obtained through the calculation process of the above formulas (1)-(2). Then, the performance capacity of each Worker node is ranked according to the size of the performance coefficient, so that logical shards with larger data volumes are preferentially allocated by Worker nodes with stronger processing capabilities. The logical shards can be allocated more reasonably, thereby improving the overall efficiency of the full update query process.

[0069] In one embodiment, a process for obtaining a volatility coefficient s is provided, including: performing a continuous ping test and a network speed test on a Worker node respectively, obtaining a ping test curve c(t) and a network speed change curve h(t); selecting a time period for c(t) and h(t), and ensuring that the selected time period has the same length, and collecting N time points at a fixed time interval for the selected time period; and obtaining the value of the ping test curve c(t) and the network speed change curve h(t) by the formula Calculate the volatility coefficient s; where i is a positive integer and i∈[1,N], is the value of c(t) corresponding to the i-th time point, For all The mean of is the value of h(t) corresponding to the i-th time point, For all The mean of Preset fixed comparison values ​​for ping stability. Preset fixed comparison values ​​for network speed stability, including preset fixed comparison values ​​for ping stability and network speed stability preset fixed comparison value They are all obtained by fitting empirical data. Therefore, through the calculation process of the above volatility coefficient s, the factors of network delay and network speed stability can be comprehensively considered to comprehensively judge the volatility of each Worker node. When one or more of the obtained delay or network speed has a large volatility, the obtained volatility coefficient s value will also be larger. Therefore, when calculating the performance coefficient x for each Worker node, the calculation of the volatility coefficient s in this embodiment can provide a more accurate judgment basis.

[0070] In one embodiment, dynamic adjustment includes shard merging and shard splitting. Through the shard merging and shard splitting process of logical shards, the processing process of each logical shard can be dynamically adjusted. While ensuring that the Worker node meets the demand, the overall efficiency of the full data update query process can be improved through the sharding of logical shards.

[0071] In one embodiment, the process of dynamically adjusting the logical shards includes:

[0072] S1. Compare the processing speed of each Worker node monitored by the Worker node monitoring unit with the preset gradient threshold set {A1, A2, ..., Am}, wherein the preset gradient threshold set {A1, A2, ..., Am} is set according to empirical data and satisfies A1>A2> ...>Am. Therefore, in step S2, when the processing speed of the Worker node is greater than A1, the logical shards to be subsequently processed by the corresponding Worker node are merged, thereby ensuring the processing efficiency while reducing the use of Worker nodes. On the one hand, the vacant Worker nodes can be provided to the logical shards with slower processing speed, and on the other hand, On the other hand, keeping idle Worker nodes can avoid exhaustion of database connections and ensure stable operation of the system; S3, when the processing speed ∈ [A1, A2], it means that the processing of the logical allocation is within a reasonable range, so the logical shards of the subsequent processing of the corresponding Worker node remain unchanged; S4, when the processing speed of the Worker node is less than A2, the node needs to be split to ensure the overall efficiency of the data full update query process, and at the same time, in order to avoid exhaustion of database connections due to excessive occupation of Worker nodes, this embodiment performs a shard split pre-operation on the logical shards of the subsequent processing of the corresponding Worker node; S5, split the shards The concurrency of the Worker nodes after the pre-sharding operation is compared with the control number of the concurrency of the Worker nodes: if the concurrency of the Worker nodes is ≤ the control number of the concurrency of the Worker nodes, it means that the concurrency of the Worker nodes after the sharding pre-operation is within the controllable range, so the sharding pre-operation process is performed to improve the overall efficiency of the full update query process; otherwise, the processing speed of the Worker nodes is compared with A3, and the logical shards subsequently processed by the Worker nodes corresponding to the processing speed of the Worker nodes < A3 are subjected to the sharding pre-operation, and step S5 is repeated and the preset gradient threshold is lowered in turn, so by continuously lowering The speed standard method ensures that the number of Worker nodes is less than the control number of Worker node concurrency, and at the same time, logical nodes with slower processing speed are logically sharded as much as possible, so as to improve the overall efficiency of the full update query process as much as possible. When the Worker node processing speed is less than Am and the condition of the Worker node concurrency ≤ the control number of Worker node concurrency is still not met, the logical sharding of the subsequent processing of the corresponding Worker node is kept unchanged, giving priority to ensuring the stability of the system. Through the above dynamic adjustment process, the control of the Worker node concurrency and the efficiency of the full update query process can be effectively balanced.

[0073] In addition, the preset slicing strategy in the above scheme also includes: first slicing by geographical location, and then dividing by time range; and first calculating the hash value of the slicing key and distributing it by modulus, and then dividing by time range. The logical slicing obtained by the above preset slicing strategy can meet different needs. At the same time, due to the differences in the size of logical slicing, it is also applicable to the slicing method in the present invention.

[0074] In one embodiment, see the attached Figure 2 A full update query slicing method for a high concurrency environment is provided. The method adopts a full update query slicing system for a high concurrency environment, including: step one, dividing the data into a plurality of groups of logical slicings according to a preset slicing strategy by a pre-slicing unit; step two, performing a matching analysis based on the performance parameters of the Worker node and the size of the logical slicing by a Worker node allocation unit, and obtaining a matching analysis result between the logical slicing and the Worker node; step three, monitoring the processing speed of each Worker node by a Worker node monitoring unit; step four, dynamically adjusting the logical slicing by a dynamic slicing adjustment unit according to the processing speed of each Worker node and the number of concurrent control of the Worker node. By matching and analyzing the performance parameters of the Worker nodes with the size of the logical shards, the logical shards are matched with the Worker nodes. Therefore, the logical segments with larger data volumes can be processed preferentially by the Worker nodes with better processing capabilities, thereby improving the overall data full update query rate. In addition, the Worker node monitoring unit is used to monitor the processing speed of each Worker node, and then the logical shards are dynamically adjusted through the dynamic slice adjustment unit according to the processing speed of each Worker node and the number of concurrent control of the Worker nodes. On the basis of the logical shards meeting the data transmission requirements, the logical shards can be dynamically adjusted through the processing speed of the Worker nodes and the number of concurrent control of the Worker nodes, thereby ensuring the processing rate of the Worker nodes and improving the overall efficiency of the data full update query process.

[0075] Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limitations on the present application. Ordinary technicians in this field can change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A full update query slicing system for a high-concurrency environment, characterized in that: include: A pre-slicing unit is used to divide the data into several groups of logical slicing according to a preset slicing strategy; The Worker node allocation unit is used to perform matching analysis based on the performance parameters of the Worker node and the size of the logical shard, and obtain the matching analysis results between the logical shard and the Worker node; Worker node monitoring unit, used to monitor the processing speed of each Worker node; The dynamic slice adjustment unit is used to dynamically adjust the logical slices according to the processing speed of each Worker node and the number of concurrent control of Worker nodes.

2. According to claim 1, a full update query slice system for a high-concurrency environment is characterized in that: The preset slicing strategy includes: Separate the data into hot and cold parts according to the data content, and obtain new data shards and old data shards; The new data shards are divided in order according to the first continuous interval of the timestamp to obtain the first shard group; The old data shards are divided in order according to the second continuous interval of the timestamp to obtain the second shard group; The first continuous interval of timestamps is smaller than the second continuous interval of timestamps; The shards in the first shard group and the second shard group are divided into several groups of logical shards.

3. According to claim 2, a full update query slice system for a high-concurrency environment is characterized in that: The matching analysis process includes: Sort several groups of logical shards in descending order of data volume to obtain a logical shard sequence; Perform performance analysis on each Worker node according to the performance parameters, sort the nodes from strong to weak according to the performance analysis results, and obtain the Worker node performance sequence; Align the logical shard sequence with the Worker node performance sequence according to the sequence number, and match the logical shards and Worker nodes in the same sequence.

4. A full update query slice system for a high concurrency environment according to claim 3, characterized in that: The performance parameters include the memory and network throughput of the Worker node; The process of performance analysis for each Worker node includes: The performance coefficient x of each Worker node is calculated using formulas (1)-(2); (1) (2) Where p is the processing capacity value of the Worker node. is the benchmark value corresponding to the processing capacity value, s is the volatility coefficient of the Worker node, , is the adjustment coefficient, Rm is the memory value of the Worker node, and Rs is the preset resident memory value of the Worker node. is the efficiency coefficient and satisfies <1, Nt is the throughput of the Worker node, Indicates taking with the minimum value among Nt.

5. According to claim 4, a full update query slice system for a high-concurrency environment is characterized in that: The process of obtaining the volatility coefficient s includes: Perform continuous ping tests and network speed tests on the Worker nodes respectively to obtain the ping test curve c(t) and network speed change curve h(t); Select time periods for c(t) and h(t), and ensure that the lengths of the selected time periods are the same. Collect N time points at fixed time intervals in the selected time periods. By formula Calculate and obtain the volatility coefficient s; Where i is a positive integer and i∈[1,N], is the value of c(t) corresponding to the i-th time point, For all The mean of is the value of h(t) corresponding to the i-th time point, For all The mean of Preset fixed comparison values ​​for ping stability. Preset fixed comparison value for network speed stability.

6. A full update query slice system for a high concurrency environment according to claim 4, characterized in that: The dynamic adjustment includes shard merging and shard splitting.

7. A full update query slice system for a high-concurrency environment according to claim 6, characterized in that: The process of dynamically adjusting logical shards includes: S1, comparing the processing speed of each Worker node monitored by the Worker node monitoring unit with a preset gradient threshold set {A1, A2, ..., Am}, where A1>A2>...>Am; S2: When the processing speed of the Worker node is greater than A1, the logical shards to be subsequently processed by the corresponding Worker node are merged; S3. When the processing speed ∈ [A1, A2], the logical sharding of the subsequent processing of the corresponding Worker node remains unchanged; S4: When the processing speed of the Worker node is less than A2, the logical shards to be subsequently processed by the corresponding Worker node will be pre-sharded; S5. Compare the concurrent number of Worker nodes after the shard splitting pre-operation with the concurrent number control number of Worker nodes: If the concurrent number of Worker nodes is ≤ the concurrent number of Worker nodes, the shard splitting pre-operation process is executed; Otherwise, the processing speed of the Worker node is compared with A3, and the logical shards subsequently processed by the Worker node corresponding to the Worker node processing speed < A3 are pre-sharded, and step S5 is repeated while the preset gradient threshold is lowered in sequence; when the Worker node processing speed < Am still does not meet the condition that the number of concurrent Worker nodes ≤ the number of concurrent Worker nodes controlled, the logical shards subsequently processed by the corresponding Worker node remain unchanged.

8. The full update query slicing system for a high-concurrency environment according to claim 1, characterized in that: The preset slicing strategy also includes: First, shard by geographical location, and then divide by time range.

9. A full update query slice system for a high-concurrency environment according to claim 1, characterized in that: The preset slicing strategy also includes: First, the hash value of the shard key is calculated and distributed modulo, and then divided by time range.

10. A full update query slicing method for a high concurrency environment, characterized in that: The method adopts a full update query slice system for a high-concurrency environment as described in any one of claims 1 to 9, including: Step 1: Divide the data into several groups of logical shards according to a preset slicing strategy through a pre-slicing unit; Step 2: The Worker node allocation unit performs matching analysis based on the performance parameters of the Worker node and the size of the logical shard, and obtains the matching analysis result between the logical shard and the Worker node; Step 3: Monitor the processing speed of each Worker node through the Worker node monitoring unit; Step 4: Dynamically adjust the logical shards according to the processing speed of each Worker node and the number of concurrent worker nodes through the dynamic slicing adjustment unit.

Images