Traffic partitioning method and device, equipment and storage medium

Abstract

The invention provides a traffic partitioning method and device, equipment and a storage medium, and the method comprises the steps: determining a traffic distribution feature corresponding to an original data stream based on the original data stream obtained in real time; wherein the flow distribution characteristics are used for representing flow types contained in the original data flow and the proportion of each flow type; determining a partition label mapping table according to the flow distribution characteristics; wherein the partition label mapping table comprises a mapping relation between each flow type and a partition range; routing the original data flow to a corresponding partition according to the partition label mapping table and the flow type corresponding to the original data flow; intelligent flow distribution and dynamic load balancing can be realized, single-point overheating is avoided, the system stability and the resource utilization rate are improved, and the operation and maintenance cost is reduced.

Description

Technical Field

[0001] This application relates to the field of big data technology, and in particular to a traffic partitioning method, apparatus, device and storage medium. Background Technology

[0002] In the field of big data processing, traditional data distribution mechanisms suffer from severe performance bottlenecks. The KeyBy operator uses a fixed hash allocation strategy, always routing data with the same key to the same processing node. When a "hot key" appears, massive amounts of data are concentrated on a single node, causing CPU overload on that node while other nodes are idle, leading to significant data skew and uneven load distribution. At the same time, existing static partitioning strategies lack dynamic adaptability. While the rebalance strategy can distribute data evenly, it disrupts key-based state calculations and cannot suppress hotspots when the data source itself is skewed, instead spreading the pressure to the entire cluster and affecting overall throughput. The broadcast strategy, in low-traffic scenarios, performs a full copy of each piece of data, resulting in huge communication overhead and serious waste of network and computing resources. In summary, fixed hash allocation leads to hotspot concentration, while static partitioning strategies perform poorly in dealing with different traffic patterns: high communication overhead at low traffic and ineffective suppression of skew at high traffic. Summary of the Invention

[0003] This application provides a traffic partitioning method, apparatus, device, and storage medium for realizing intelligent traffic splitting and dynamic load balancing, avoiding single-point overheating, improving system stability and resource utilization, and reducing operation and maintenance costs.

[0004] Firstly, this application provides a traffic partitioning method, including:

[0005] Based on the real-time acquired raw data stream, the traffic distribution characteristics corresponding to the raw data stream are determined; wherein, the traffic distribution characteristics are used to characterize the traffic types contained in the raw data stream and the proportion of each traffic type;

[0006] Based on the traffic distribution characteristics, a partition label mapping table is determined; wherein, the partition label mapping table contains the mapping relationship between each traffic type and the partition range;

[0007] Based on the partition label mapping table and the traffic type corresponding to the original data stream, the original data stream is routed to the corresponding partition.

[0008] In one or more possible embodiments, determining the partition label mapping table based on the traffic distribution characteristics includes:

[0009] Based on the proportion of each traffic type, determine the number of partitions allocated to each traffic type;

[0010] Based on the number of partitions allocated to each traffic type, a continuous partition is allocated to any one of the traffic types.

[0011] Based on each traffic type and the allocation of consecutive partitions for each traffic type, the mapping relationship between each traffic type and the partition range is determined, and a partition label mapping table is generated.

[0012] In one or more possible embodiments, it also includes:

[0013] Get the load corresponding to each of the multiple partitions contained in any physical node;

[0014] When the difference between the loads of any two partitions exceeds a second preset threshold, the partition label mapping table is updated according to the same-node partition migration strategy.

[0015] In one or more possible embodiments, it also includes:

[0016] Obtain the load standard deviation for multiple physical nodes;

[0017] When the load standard deviation is determined to exceed a third preset threshold, the partition label mapping table is updated according to the cross-node partition migration strategy.

[0018] In one or more possible embodiments, the same-node partition migration strategy and the cross-node partition migration strategy are based on a multi-factor weighted evaluation model to determine the target partition of the original partition to be migrated; wherein, the multi-factor weighted evaluation model includes at least one of the following evaluation factors: migration cost factors related to the amount of data to be migrated, load factors related to the busyness of the target node, and spatial isolation factors related to the relative position of the target partition and the original partition.

[0019] Updating the partition label mapping table includes:

[0020] Modify the label mapping relationship of the original partition in the partition label mapping table to point to the label mapping relationship of the target partition.

[0021] In one or more possible embodiments, determining the traffic distribution characteristics corresponding to the raw data stream based on the real-time acquired raw data stream includes:

[0022] The original data stream is continuously sampled using a preset sampling window;

[0023] When the amount of sampled data reaches a fourth preset threshold, the traffic distribution characteristics corresponding to the amount of sampled data are determined, and the traffic distribution characteristics corresponding to the amount of sampled data are used as the traffic distribution characteristics corresponding to the raw data stream currently acquired in real time.

[0024] In one or more possible embodiments, it also includes:

[0025] Continuously obtain the utilization rate of each partition;

[0026] When it is determined that the utilization rate of any partition is lower than the sixth preset threshold and the duration exceeds the seventh preset threshold, the partition is designated as a slot for the idle period.

[0027] The traffic contained in any of the partitions is allocated to any high-utilization partition.

[0028] Secondly, the application provides a traffic partitioning device, including:

[0029] The traffic distribution feature determination module is used to determine the traffic distribution features corresponding to the raw data stream based on the raw data stream acquired in real time; wherein, the traffic distribution features are used to characterize the traffic types contained in the raw data stream and the proportion of each traffic type;

[0030] The partition label mapping table determination module is used to determine the partition label mapping table based on the traffic distribution characteristics; wherein, the partition label mapping table contains the mapping relationship between each traffic type and the partition range;

[0031] The data routing module is used to route the original data stream to the corresponding partition based on the partition label mapping table and the traffic type corresponding to the original data stream.

[0032] Thirdly, this application provides an electronic device, comprising:

[0033] At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform any of the methods in the first aspect.

[0034] Fourthly, this application provides a computer storage medium storing a computer program for causing a computer to perform any of the methods described in the first aspect.

[0035] According to the traffic partitioning method, apparatus, device and storage medium provided in this application, intelligent traffic splitting and dynamic load balancing are realized, avoiding single-point overheating, improving system stability and resource utilization, and reducing operation and maintenance costs. Attached Figure Description

[0036] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application, and do not constitute an undue limitation of this application.

[0037] Figure 1 This is a schematic diagram of an application scenario provided according to an embodiment;

[0038] Figure 2 This is a flowchart illustrating a traffic partitioning method according to an embodiment.

[0039] Figure 3 This is a flowchart illustrating a sampler sampling method according to an embodiment.

[0040] Figure 4 This is a detailed flowchart of a custom partitioner provided according to an embodiment;

[0041] Figure 5 This is a schematic diagram of a traffic partitioning system according to an embodiment;

[0042] Figure 6 This is a detailed flowchart of a partition controller provided according to an embodiment;

[0043] Figure 7 This is a schematic diagram of a flow zoning device according to an embodiment;

[0044] Figure 8 This is a schematic diagram of an electronic device according to an embodiment;

[0045] Figure 9 This is a schematic diagram of a computer-readable storage medium provided according to an embodiment. Detailed Implementation

[0046] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this invention, and not all of them. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0047] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0048] Furthermore, in the description of the embodiments of this application, unless otherwise stated, " / " means "or". For example, A / B can mean A or B. The "and / or" in the text is merely a description of the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist simultaneously, and B exists alone. In addition, in the description of the embodiments of this application, "multiple" means two or more.

[0049] For ease of understanding, the terms used in the embodiments of this invention are explained below:

[0050] Kakfa Data Source continuously pumps raw data streams from Kafka data to subsequent processing steps;

[0051] Flink Custom Partitions transform traffic statistics results into a feasible partition allocation scheme;

[0052] The Flow Sampler samples raw data streams from a Kafka data source in real time.

[0053] Partition controllers ensure that data streams are properly allocated to physical partition slots through precise calculations and real-time adjustments.

[0054] In the field of big data processing, the parallel allocation strategy of real-time data streams is crucial to the performance and stability of the system. Currently, mainstream frameworks (such as Apache Flink) widely use the KeyBy operator to implement data grouping. Specifically, KeyBy logically groups the data stream according to the specified key, such as sending click records of the same user ID to the same processing node to achieve aggregation calculations such as per-user click statistics. Its default implementation is to apply a hash function to the key value and then take the modulo of the downstream parallelism to determine the target partition.

[0055] However, this fixed hashing mechanism also brings significant problems: the same key value is always mapped to the same task instance. When some keys become frequently occurring "hot keys", a massive amount of data will continuously flood into a single node, causing severe data skew. This makes it easy for the node's CPU load to reach saturation, while other nodes in the cluster are in a state of low utilization, resulting in uneven overall resource utilization, increased processing latency, and even job failure.

[0056] Besides KeyBy, existing static partitioning strategies such as Rebalance and Broadcast also have their drawbacks. Static partitioning means that the partitioning rules are fixed after the job starts and cannot be dynamically adjusted according to data characteristics. The Rebalance strategy uses a round-robin approach to distribute data evenly to all downstream tasks. Although it can achieve load balancing, it destroys the context consistency of downstream stateful computations. Specifically, data with the same logical key is scattered across different nodes, making effective aggregation impossible. More seriously, if the data source itself is skewed (such as a partition in Kafka with extremely high traffic), Rebalance will spread the hot pressure to the entire cluster, thus expanding the scope of impact and failing to achieve hotspot isolation. The Broadcast strategy replicates each piece of data and sends it to all downstream nodes. It is suitable for scenarios such as configuration distribution, but under low traffic conditions (such as only a few monitoring messages per second), it will generate huge network overhead, resulting in a waste of bandwidth and processing resources.

[0057] In summary, existing partitioning mechanisms lack flexibility when facing dynamically changing and unevenly distributed streaming data: they cannot identify and alleviate hotspots, struggle to balance balance and computational consistency, and suffer from efficiency deficiencies in different scenarios, such as low-volume or high-volume traffic. Therefore, there is an urgent need for an intelligent partitioning method that can dynamically sense data distribution and adjust routing strategies in real time to improve system throughput, resource utilization, and overall stability.

[0058] To address the aforementioned issues, this application provides a traffic partitioning method, apparatus, device, and storage medium. By monitoring traffic distribution in real time and dynamically adjusting partitioning strategies, multi-level label routing is used instead of static hashing. Migration is triggered based on intra-node deviation or inter-node standard deviation, and migration targets are comprehensively decided based on capacity, load, and spatial isolation. This effectively solves the problem of data skew, improves resource utilization and system stability, and reduces operation and maintenance costs.

[0059] Example 1

[0060] For ease of understanding, the traffic partitioning method provided in this application embodiment will be described in detail below with reference to the accompanying drawings:

[0061] like Figure 1 The diagram shown illustrates an application scenario of a traffic partitioning method provided in this embodiment. The diagram includes: network 10, server 20, and storage 30. Server 20 can be the execution node responsible for running core components such as the traffic sampler, partition controller, and custom partitioner in this embodiment.

[0062] The description in this application focuses on a single server only. However, those skilled in the art should understand that the illustrated network 10, server 20, and memory 30 are intended to illustrate the operation of the electronic devices, servers, and memory involved in the technical solutions of this application. The detailed description of a single server and memory is at least for ease of explanation and does not imply any limitation on the number, type, or location of servers. It should be noted that adding additional modules to or removing individual modules from the illustrated environment will not change the underlying concept of the exemplary embodiments of this application. Furthermore, although detailed descriptions are provided for ease of explanation... Figure 1 The diagram shows a bidirectional arrow from memory 30 to server 20, but those skilled in the art will understand that the sending and receiving of the aforementioned data also needs to be achieved through network 10.

[0063] It should be noted that the memory in the embodiments of this application can be, for example, a cache system, hard disk storage, memory storage, etc. Furthermore, the traffic partitioning method proposed in this application is not only applicable to… Figure 1 The application scenarios shown can also be used in other possible streaming data processing scenarios, and the embodiments of this application do not impose any limitations.

[0064] To address the aforementioned issues, this application provides a traffic partitioning method, specifically as follows: Figure 2 As shown, it includes:

[0065] Step 201: Based on the real-time acquired raw data stream, determine the traffic distribution characteristics corresponding to the raw data stream; wherein, the traffic distribution characteristics are used to characterize the traffic types contained in the raw data stream and the proportion of each traffic type.

[0066] In one or more possible embodiments, determining the traffic distribution characteristics corresponding to the raw data stream based on the real-time acquired raw data stream includes: continuously sampling the raw data stream with a preset sampling window; determining the traffic distribution characteristics corresponding to the sampled data volume when the sampled data volume reaches a fourth preset threshold, and using the traffic distribution characteristics corresponding to the sampled data volume as the traffic distribution characteristics corresponding to the currently acquired raw data stream; specifically, the sampler is configured with preset sampling window rules to sample the continuously flowing raw data stream in real time. The sampling window can be triggered based on a time interval, for example, collecting one data sample every specific time interval (e.g., 5 seconds); or triggered based on the data volume, for example, collecting one sample every specific number of raw data streams (e.g., 100); a sampling mode that is triggered by a mixture of time and data volume is also supported; the sampler continuously receives raw data streams from message middleware such as Kafka and performs the above sampling operation;

[0067] When the sampler accumulates a fourth preset threshold (e.g., 200 samples), the distribution characteristic calculation process is triggered. This threshold can be configured according to actual business needs, data characteristics, and processing capabilities. After reaching the fourth preset threshold, the sampler or its associated analysis module performs statistical analysis on the current batch of sample data to determine the traffic distribution characteristics. The aforementioned traffic distribution characteristics are a set of quantitative indicators, including the total number of identified traffic types, the frequency of each traffic type, and the proportion of each type in the total sample size. Subsequently, the calculated sample data distribution characteristics are used as an estimate of the overall traffic distribution characteristics of the current original data stream. This characteristic information is encapsulated into a structured distribution report. The report generation module outputs the results in a predefined format, such as a JSON document. Its content includes a global report identifier, the total number of traffic types, a flag indicating the existence of minor traffic types, and a detailed list describing the identifier, specific quantity, and proportion of each traffic type. This provides a quantitative basis for subsequent data partitioning, load balancing, or routing strategies. The specific format of the traffic distribution report obtained based on the above distribution characteristics is as follows:

[0068] {"Traffic Report":{"id":1001,"Number of Traffic Types":5,"Small Traffic Exists":true,"Traffic Distribution":[{"Type":"A","Quantity":50,"Percentage":0.25},{"Type":"B","Quantity":30,"Percentage":0.15},{"Type":"C","Quantity":60,"Percentage":0.3},{"Type":"D","Quantity":10,"Percentage":0.05},{"Type":"E","Quantity":50,"Percentage":0.25}]}}

[0069] In one or more possible embodiments, the sampler can also set a global upper limit for the total amount of samples (e.g., 10,000 data points). When the cumulative amount of sampled data reaches this upper limit, the sampler outputs a process end flag and automatically resets or clears the sampling counter to prepare for a new round of sampling and feature analysis. Furthermore, the sampling process can also be designed as a sliding window mode, that is, retaining some new samples and discarding old samples after calculation, thereby achieving dynamic real-time updates of distribution features. The above-mentioned sampler can efficiently and almost in real-time capture the distribution changes of large-scale data streams with controllable computational overhead. The specific process is as follows: Figure 3 As shown, it includes:

[0070] Step 301: Receive the raw data stream and sample the raw data stream.

[0071] Step 302: Determine if the number of samples has reached 200. If yes, proceed to step 303; otherwise, proceed to step 304.

[0072] Step 303: Determine if the number of samples has reached 10,000. If not, proceed to step 305; if yes, proceed to step 306.

[0073] Step 304: Continue sampling the original data stream;

[0074] Step 305: Output a traffic distribution report;

[0075] Step 306: Output the end flag and clear the count;

[0076] Data is continuously accumulated until 200 samples are reached, triggering type distribution analysis and sending the analysis report downstream. When the number of traffic items is between 200 and 10,000, new data is continuously received if the upper limit is not reached. When the upper limit of 10,000 data items is reached, an end flag is added.

[0077] Step 202: Based on the above traffic distribution characteristics, determine the partition label mapping table; wherein the partition label mapping table contains the mapping relationship between each traffic type and the partition range;

[0078] In one or more possible embodiments, determining the partition label mapping table based on the traffic distribution characteristics includes: determining the number of partitions allocated to each traffic type based on the proportion of each traffic type; allocating consecutive partitions to any one of the traffic types based on the number of partitions allocated to each traffic type; determining the mapping relationship between each traffic type and the partition range based on each traffic type and the consecutive partitions allocated to each traffic type, and generating a partition label mapping table; in this application, an example is given using 4 physical nodes, each physical node with 5 partitions, for a total of 20 partitions; the partition controller uniformly manages the resource pool (20 partitions) composed of 20 fixed partitions and, based on different traffic... The real-time proportion of traffic types is used to dynamically allocate specific partition numbers to each type proportionally; the partition controller continuously monitors the input proportion of each traffic type and calculates the number of physical partitions it should receive based on this; the allocation process strictly follows the "sequential allocation" principle, achieving fine-grained resource management by maintaining a global, structured partition mapping table; independent and continuous partition blocks are allocated to traffic of different types, which can avoid resource competition between partitions, and dedicated partitions are allocated to each type of traffic to achieve physical isolation between traffic; the aforementioned partition label mapping table achieves accurate mapping from traffic type to continuous partitions, ensuring continuity based on a fixed traffic type order, and eliminating gap fragmentation of traditional hash partitions.

[0079] In one or more possible embodiments, the aforementioned partition label mapping table is defined and stored in JSON format, clearly recording the unique identifier, theoretical ratio, number of allocated partitions, start and end indexes of each traffic type, and a label with clear semantics for each partition. For example, the naming convention for these labels is "start partition number_end partition number_traffic type," such as "0_8_TypeA," which indicates that traffic type A exclusively occupies 9 consecutive partitions from number 0 to number 8. These labels enable rapid identification, location, and retrieval of partition resources. To ensure friendliness and availability for low-volume traffic types (traffic types with a very small percentage), any traffic type whose theoretical percentage is lower than or equal to a preset fixed value is considered low-volume traffic. To prevent low-volume traffic from being completely ignored in resource contention due to its low percentage, a method for forced allocation of low-volume traffic is provided, which enforces a guarantee strategy of allocating at least one partition for low-volume traffic. In the mapping table, this behavior is marked by the `forced_allocation` field (set to true), clearly distinguishing that this is a mandatory guarantee measure for low-volume traffic in this application, rather than a result of proportional calculation. The partition label mapping table given in this application is shown below, in JSON format.

[0080] {"Partition_label_mapping_table":{"id":1001,"partition_total":20,"μ":0.5,"σ":0.1,"small_flowexist":true,"method":"sequential","gap_size":0,"overlap":false,"partit ion_mapping":

[0081] [{"type":"TypeA","ratio":0.25,"partition_count":5,"start_partition":0,"end_partition":4,"label":"0_4_TypeA","forced_allocation":false},

[0082] {"type":"TypeB","ratio":0.15,"partition_count":3,"start_partition":5,"end_partition":7,"label":"5_7_TypeA","forced_allocation":false},

[0083] {"type":"TypeC","ratio":0.3,"partition_count":6,"start_partition":8,"end_partiti on":13,"label":"8_13_TypeA","forced_allocation":false},

[0084] {"type":"TypeD","ratio":0.05,"partition_count":1,"start_partition":14,"end_partition":14,"label":"14_14_TypeD","forced_allocation":true},

[0085] {"type":"TypeE","ratio":0.25,"partition_count":5,"start_partition":15,"end_partition":19,"label":"15_19_TypeE","forced_allocation":false}]}}

[0086] For example, the cluster in this application has 4 physical nodes, each with 5 partitions, for a total of 20 partitions. Type C, accounting for 30%, will be allocated 6 partitions, Type A and Type E, accounting for 25%, will each receive 5 partitions, Type B will be allocated 3 partitions, and Type D will be forcibly allocated 1 partition to ensure its data processing capacity and prevent important but sparse data from being overwhelmed. Meanwhile, μ represents the average node load, referring to the average CPU utilization, processing load, or other resource utilization indicators of all nodes (or partitions); σ represents the standard deviation of the load between nodes, used to measure the dispersion of the load of each node from the average load, i.e., whether the load is balanced; the above type proportions are calculated using the following formula:

[0087]

[0088] As can be seen from the partition label mapping table above, Type D accounts for 5%, and the corresponding forced allocation flag is true, indicating that its actual proportion is less than the preset fixed value, but it still obtains the necessary partition resources through the fallback mechanism. The preset fixed value can be determined by the user according to the actual situation. For example, this application determines the preset fixed value to be 5%, and Type D accounts for 5%, so Type D is identified as low traffic and is forcibly allocated 1 partition (14). Type E accounts for 25%, and as can be seen from the partition label mapping table above, the forced_allocation flag is false, which means that Type E does not need to be partitioned through the fallback mechanism above. The partition label mapping table in this application, through this combination of dynamic and forced strategies, ensures both the resource requirements of high throughput traffic and the processing capacity of low traffic tasks.

[0089] Step 203: Based on the partition label mapping table and the traffic type corresponding to the original data stream, route the original data stream to the corresponding partition.

[0090] In one or more possible embodiments, the custom partitioner receives all raw data streams, determines the traffic type for each stream, and then randomly routes the traffic to a selected partition within the partition label mapping table based on the traffic type. The specific process of the custom partitioner is as follows: Figure 4 As shown, it includes:

[0091] Step 401: Receive at least one traffic and partition label mapping table from the raw data stream;

[0092] Step 402: Determine the traffic type;

[0093] Step 403: Determine the partition range corresponding to the traffic based on the traffic type and the partition label mapping table mentioned above;

[0094] Step 404: Randomly select a partition from the partition range corresponding to the above traffic as the selected partition;

[0095] Step 405: Route the traffic to the selected partition mentioned above.

[0096] This application provides a traffic partitioning system that includes a Kafka data source, a traffic sampler, a partition controller, and a custom partitioner, specifically as follows: Figure 5As shown, Kafka, as a distributed message queue, provides a high-throughput data source. The traffic collector receives Kafka data and performs preliminary feature analysis, outputting a traffic distribution report. The partition controller generates a dynamic allocation strategy based on the traffic distribution report and obtains a partition label mapping table. The custom partitioner, based on the partition mapping relationship in the partition label mapping table, constructs a complete resource pool from partitions 0-19. Finally, all partition data is aggregated to the downstream operator instance.

[0097] In one or more possible embodiments, the method further includes: obtaining the load corresponding to each of the multiple partitions contained in any physical node; when the difference between the loads corresponding to any two partitions exceeds a second preset threshold, updating the partition label mapping table according to the same-node partition migration strategy; obtaining the load standard deviation corresponding to multiple physical nodes; when the load standard deviation exceeds a third preset threshold, updating the partition label mapping table according to the cross-node partition migration strategy; the same-node partition migration strategy and the cross-node partition migration strategy are determined by a multi-factor weighted evaluation model to determine the target partition of the original partition to be migrated; wherein the multi-factor weighted evaluation model includes at least one of the following evaluation factors: migration cost factors related to the amount of migrated data, load factors related to the busyness of the target node, and spatial isolation factors related to the relative position of the target partition and the original partition; updating the partition label mapping table includes: modifying the label mapping relationship of the original partition in the partition label mapping table to a label mapping relationship pointing to the target partition.

[0098] In one or more possible embodiments, this application provides a two-level imbalance classification model for accurately diagnosing and distinguishing the sources and types of resource allocation imbalance. The two-level imbalance classification model mainly identifies two types of bias: intra-node bias and inter-node bias.

[0099] Intra-node imbalance refers to the uneven load distribution between different partitions within the same physical node. Here, a partition refers to multiple task instances or execution processes (e.g., slots) running on a single node (e.g., TaskManager in the Flink framework). When the difference between the load (e.g., CPU utilization) of any two partitions within the same node exceeds a second preset threshold (e.g., 40%), it is determined that a node-type imbalance has occurred. The aforementioned second preset threshold is used to trigger an intervention mechanism in the early stages of resource fragmentation or excessive local load, thereby effectively avoiding performance bottlenecks or resource contention caused by overheating of some partitions within a single node, and achieving early fine-grained control.

[0100] Imbalance between nodes occurs between different physical nodes. When the standard deviation σ of the overall load (such as average CPU utilization) of all nodes exceeds the third preset threshold (e.g., 15%), it is determined that there is a spatial deviation across nodes. The aforementioned third preset threshold refers to the principle of significant difference in statistics and is used to sensitively capture the deviation of the global resource distribution from the equilibrium state. Once the standard deviation exceeds the third preset threshold, it indicates that there is a significant spatial imbalance within the cluster, and a global coordination and redistribution process needs to be initiated to quickly restore the equilibrium state.

[0101] To address the two types of imbalances mentioned above, this invention designs corresponding migration mechanisms. For intra-node imbalances, a same-node partition migration strategy is adopted. This same-node partition migration strategy stipulates that high-load partitions can only be split and migrated between other low-load partitions within the same node. This strictly constrains the migration operation to a single node, completely avoiding network bandwidth overhead and latency caused by cross-node data transmission. Practical verification has shown that this can effectively reduce migration costs by approximately 30%. For inter-node imbalances, a cross-node partition migration strategy is adopted. This cross-node partition migration strategy involves dynamically redistributing partitions across different physical nodes. During this process, partitions with potential resource contention or data dependencies are intentionally distributed across different physical nodes, thereby fundamentally reducing network conflict risks and improving the overall stability and parallel processing efficiency of the cluster.

[0102] In one or more possible embodiments, during partition migration, the data spatial layout of the newly migrated partition may conflict with the distribution of the existing partitions. To quantitatively assess and mitigate such risks, this application proposes a P-collision probability estimation model (multi-factor weighted evaluation model), the calculation formula of which is defined as:

[0103] P_collision = 0.5 × W1 + 0.3 × W2 + 0.2 × W3

[0104] The calculated value of P_collision is used to characterize the risk level of resource or data conflict between the candidate target partition and the original partition. The higher the P_collision value, the greater the possibility of conflict between the candidate target partition and the original partition. W1, W2, and W3 are used to reflect the contribution and sensitivity differences of different dimensions to the collision risk. W1 (the migration cost factor related to the amount of migrated data, with a coefficient of 0.5) is the most important collision inducement and is given the highest weight. The value of W1 is inversely proportional to the "similarity value between the old and new partitions", that is, W1 = 1 / similarity value between the old and new partitions. The similarity value between the old and new partitions refers to the similarity of the capacity of the old and new partitions, or the similarity of the remaining capacity of the old and new partitions. Other factors that can characterize the similarity between old and new partitions, such as capacity similarity values, can also be directly applied, and will not be listed here. The W1 setting mentioned above is used to prioritize the target partition that better matches the capacity requirements of the original partition, thereby improving resource reuse efficiency and effectively avoiding conflict areas that may be caused by capacity mismatch. W2 (load factor related to the busyness of the target node, coefficient 0.3) is used to prioritize the partition with lower overall load as the target partition for migration. By avoiding further concentrating the new load on already high-load partitions, it helps to reduce the probability of local resource contention and partition conflicts from an overall perspective. W3 ( The spatial isolation factor (coefficient 0.2) related to the relative positions of the target partition and the original partition is inversely proportional to the "distance," i.e., W3 = 1 / distance. This factor guides the partition controller to prioritize locations that maintain sufficient physical or logical distance from currently densely assigned similar or highly correlated partitions, minimizing potential collision probabilities by fully utilizing spatial isolation characteristics. The model calculates the P_collision value for each candidate target and always selects the option with the lowest collision probability for migration. For example, if the P_collision value for migrating from original partition 3 to partition 1 is 0.3, while the P_collision value for migrating to partition 2 is 0.4, then the partition controller... The controller will prioritize migrating partition 3 (the original partition) to partition 1 (the target partition). The aforementioned P-collision model provides a quantitative and objective risk assessment basis and decision support for partition migration through multi-factor weighted evaluation, thereby realizing an intelligent and low-risk resource scheduling mechanism. Then, based on the results of the migration, the partition label mapping table is updated, and the label mapping relationship of the original partition in the partition label mapping table is modified to point to the label mapping relationship of the target partition. For example, the mapping target of the original partition 3 is modified to point to the target partition 1, so that the traffic originally sent to partition 3 is rerouted to partition 1 for processing.

[0105] In one or more possible embodiments, the specific process of the above-described partition controller is as follows: Figure 6 As shown, it includes:

[0106] Step 601, Data reception;

[0107] Step 602: Determine whether the above data is raw data stream or raw data stream and traffic distribution characteristics. If the above data is raw data, proceed to step 603; if the above data is raw data and traffic distribution characteristics, proceed to step 604.

[0108] Step 603: Determine that the above data is the original data. Based on the latest partition label mapping table, obtain the partition range of the traffic corresponding to various traffic types in the original data stream. Alternatively, if there is no partition label mapping table, determine the partition corresponding to each traffic item by using a polling method.

[0109] Step 604: Determine that the above data is the original data stream and traffic distribution characteristics, and generate a partition label mapping table based on the traffic distribution characteristics;

[0110] Step 605: Determine whether the difference in load between multiple partitions within a node exceeds a second preset threshold. If yes, proceed to step 606; otherwise, proceed to step 607.

[0111] Step 606: Update the partition label mapping table according to the same node partition migration strategy;

[0112] Step 607: Based on the partition label mapping table, obtain the partition range of traffic corresponding to various traffic types in the original data stream;

[0113] Step 608: Determine whether the load standard deviation corresponding to multiple physical nodes exceeds the third preset threshold. If yes, proceed to step 609; otherwise, proceed to step 6010.

[0114] Step 609: Update the partition label mapping table according to the cross-node partition migration strategy, and obtain the partition range of traffic corresponding to various traffic types in the original data stream;

[0115] Step 6010: Obtain the partition range of traffic corresponding to various traffic types in the original data stream according to the partition label mapping table;

[0116] Step 6011: Route the raw data stream to the corresponding partition;

[0117] In one or more possible embodiments, the method further includes: continuously acquiring the utilization rate of each partition; determining that when the utilization rate of any partition is lower than a sixth preset threshold and the duration exceeds a seventh preset threshold, designating any partition as an idle slot; allocating the traffic contained in any partition to any high-utilization partition; specifically, continuously collecting and calculating the resource utilization rate indicators of each physical partition, and when it is detected that the utilization rate of a certain partition is continuously lower than 10% (sixth preset threshold) and this state is maintained stably for more than 10 minutes (seventh preset threshold), the partition will be determined as an idle slot, and a resource reclamation and integration process will be triggered, including: releasing all computing resources occupied by the idle slot and dynamically migrating the data traffic it carries to the currently high-load partition (i.e., high-pressure node, high-utilization partition); not only realizing the automatic release of idle resources, but also significantly improving the overall resource utilization efficiency of the cluster by merging low-load tasks to high-capacity nodes, effectively avoiding the resource fragmentation problem caused by long-term low load, reducing the operation and maintenance cost of infrastructure, and enhancing the system's adaptability under fluctuating traffic.

[0118] According to the traffic partitioning method provided in this application, a traffic sampler captures data distribution characteristics in real time. A partition controller generates a dynamic allocation strategy based on traffic reports and relies on a custom partitioner combined with a partition label mapping table to achieve precise data routing. A multi-level path label routing mechanism is adopted, replacing traditional key values ​​with labels to achieve autonomous and controllable scheduling of data paths. Furthermore, a differentiated strategy is adaptively adopted for different traffic scales: small traffic is distributed to multiple partitions in a round-robin fashion, while large traffic is directed to specific nodes (or a specific partition range), effectively avoiding the single-point overheating problem caused by KeyBy static hashing and significantly improving system stability. A two-level imbalance threshold mechanism is established: when the load deviation of a node's internal partitions exceeds 40% or the standard deviation of the load between nodes is greater than 15%, partition migration is automatically triggered, and the above mapping table is updated according to the migration results. The migration process is based on the P-collision decision model, which prioritizes target nodes with high capacity similarity, low load density, and strong spatial isolation to minimize migration disturbances. At the same time, the partition utilization rate is continuously monitored, and slots with a utilization rate of less than 10% for more than 10 minutes are automatically released and their traffic is merged into high-pressure nodes, thereby improving the overall resource utilization rate, reducing operation and maintenance costs, and realizing dynamic optimization and efficient management of cluster resources.

[0119] Example 2

[0120] Corresponding to the above-mentioned traffic partitioning method, the present invention also proposes a traffic partitioning device, specifically as follows: Figure 7 As shown, it includes:

[0121] The traffic distribution feature determination module 701 is used to determine the traffic distribution features corresponding to the raw data stream based on the raw data stream acquired in real time; wherein, the traffic distribution features are used to characterize the traffic types contained in the raw data stream and the proportion of each traffic type.

[0122] The partition label mapping table determination module 702 is used to determine the partition label mapping table based on the above traffic distribution characteristics; wherein, the partition label mapping table contains the mapping relationship between each traffic type and the partition range;

[0123] The data routing module 703 is used to route the original data stream to the corresponding partition according to the partition label mapping table and the traffic type corresponding to the original data stream.

[0124] Since the device embodiments of the present invention correspond to the method embodiments described above, details not disclosed in the device embodiments can be referred to in the method embodiments described above, and will not be repeated in the present invention.

[0125] Example 3

[0126] This application also provides an electronic device, including at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the above-described traffic partitioning method.

[0127] like Figure 8 As shown, the device includes a processor 801, a memory 802, a communication interface 803, and a bus 804. The processor 801, memory 802, and communication interface 803 are interconnected via the bus 804.

[0128] Processor 801 is configured to read instructions from memory 802 and execute them, so that at least one processor can execute the traffic partitioning method provided in the above embodiments.

[0129] The memory 802 is used to store various instructions and programs of the xx method provided in the above embodiments.

[0130] The 804 bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be divided into address buses, data buses, control buses, etc. For ease of representation, Figure 8 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0131] The processor 801 can be a central processing unit (CPU), a network processor (NP), a graphics processing unit (GPU), or any combination of CPU, NP, and GPU. It can also be a hardware chip. The aforementioned hardware chip can be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The aforementioned PLD can be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0132] Example 4

[0133] In addition, this application also provides a computer-readable storage medium, such as Figure 9 As shown, the computer storage medium stores a computer program that is used to cause the computer to perform any of the methods described in the above embodiments.

[0134] The memory may include readable media in the form of volatile memory, such as random access memory (RAM) 901 and / or cache memory 902, and may further include read-only memory (ROM) 903.

[0135] The memory may also include a program / utility 905 having a set (at least one) of program modules 904, including but not limited to: an operating system, one or more application programs, other program modules, and program data, each or some combination of these examples may include an implementation of a network environment.

[0136] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0137] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0138] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0139] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0140] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A method of traffic zoning, characterized by, The method comprises: determining a traffic distribution feature corresponding to the original data stream based on the real-time obtained original data stream; wherein the traffic distribution feature is used to represent the traffic types contained in the original data stream and the proportion of each traffic type; determining a partition label mapping table according to the traffic distribution feature; wherein the partition label mapping table contains the mapping relationship between each traffic type and the partition range; routing the original data stream to the corresponding partition according to the partition label mapping table and the traffic type corresponding to the original data stream.

2. The method of claim 1, wherein, The method of determining the partition label mapping table according to the traffic distribution feature comprises: determining the number of partitions allocated to each traffic type according to the proportion of each traffic type; allocating continuous partitions to each traffic type according to the number of partitions allocated to each traffic type; determining the mapping relationship between each traffic type and the partition range according to each traffic type and the continuous partitions allocated to each traffic type, and generating a partition label mapping table.

3. The method of claim 1, wherein, The method further comprises: obtaining the load corresponding to each partition contained in any one physical node; when the difference between the loads corresponding to any two partitions exceeds a second preset threshold, updating the partition label mapping table according to an intra-node partition migration strategy.

4. The method of claim 1, wherein, The method further comprises: obtaining the load standard deviation corresponding to multiple physical nodes; when the load standard deviation exceeds a third preset threshold, updating the partition label mapping table according to a cross-node partition migration strategy.

5. The method according to claim 3 or 4, characterized in that, The intra-node partition migration strategy and the cross-node partition migration strategy are to determine the target partition of the original partition to be migrated according to a multi-factor weighted evaluation model; wherein the multi-factor weighted evaluation model includes at least one of the following evaluation factors: a migration cost factor related to the migration data volume, a load factor related to the busy degree of the target node, and a spatial isolation factor related to the relative position of the target partition and the original partition. The method of updating the partition label mapping table comprises: modifying the label mapping relationship of the original partition in the partition label mapping table to the label mapping relationship pointing to the target partition.

6. The method of claim 1, wherein, The method of determining the traffic distribution feature corresponding to the original data stream based on the real-time obtained original data stream comprises: continuously sampling the original data stream with a preset sampling window; when the sampling data volume reaches a fourth preset threshold, determining the traffic distribution feature corresponding to the sampling data volume, and taking the traffic distribution feature corresponding to the sampling data volume as the traffic distribution feature corresponding to the current real-time obtained original data stream.

7. The method of claim 1, wherein, The method further comprises: continuously obtaining the utilization rate of each partition; when the utilization rate of any one partition is lower than a sixth preset threshold and the duration exceeds a seventh preset threshold, taking the any one partition as an empty slot; allocating the traffic contained in the any one partition to any one high-utilization partition.

8. A flow partitioning device, characterized by The method comprises: a traffic distribution feature determination module, configured to determine a traffic distribution feature corresponding to the original data stream based on the real-time obtained original data stream; wherein the traffic distribution feature is used to represent the traffic types contained in the original data stream and the proportion of each traffic type; The partition label mapping table determination module is configured to determine a partition label mapping table according to the traffic distribution characteristics, wherein the partition label mapping table comprises a mapping relationship between each traffic type and a partition range. The data routing module is configured to route the original data flow to a corresponding partition according to the partition label mapping table and a traffic type corresponding to the original data flow.

9. An electronic device, comprising: The electronic device comprises: at least one processor; and a memory connected to the at least one processor in communication, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform any one of the methods of claims 1-7.

10. A computer storage medium, characterized in that, The computer storage medium stores a computer program, and the computer program is used to enable a computer to perform any one of the methods of claims 1-7.

Images