Flow computing heterogeneous cluster perception scheduling method, device and equipment and storage medium

Abstract

The embodiment of the invention discloses a stream computing heterogeneous cluster perception scheduling method, device and equipment and a storage medium, and the method comprises the steps: collecting the data volume of each key value in a key value set and the processing capability of each node in a node set; initializing an intermediate state based on the data volume and the processing capability; identifying a hot key value set and a non-hot key value set according to a hot key value threshold value and the data volume of each key value; key values in the non-hot key value set are aggregated; scattering key values in the hot key value set; and generating an allocation scheme according to an aggregation and scattering result, and executing task scheduling based on the allocation scheme.

Description

Technical Field

[0001] This application relates to, but is not limited to, the field of cloud data processing technology, and in particular to a stream computing heterogeneous cluster-aware scheduling method, apparatus, device, and storage medium. Background Technology

[0002] With the rapid development of cloud computing and big data technologies, cloud vendors have built large-scale data processing systems to aggregate and statistically analyze service metrics in order to support standardized services (such as Content Delivery Networks (CDNs)). These systems need to process massive amounts of data, such as access logs generated by each user's access request in a CDN service. This data is not only used for internal billing or operations and maintenance, but may also be processed into standard data products for customers. Taking CDN offline log generation as an example, cloud vendors need to extract data for each customer and domain from the log stream, convert it according to a specified format, store it as an intermediate file, and finally aggregate it into a time-granular offline log package as reconciliation vouchers and compliance documentation.

[0003] To process massive amounts of data within a specified timeframe, the industry typically employs streaming big data processing frameworks (such as Spark Streaming, Kafka Streams, and Flink). These frameworks address the parallel processing challenges of large-scale tasks through distributed task decomposition, dynamic resource adjustment, and unified scheduling management. However, these frameworks cannot completely overcome processing bottlenecks caused by business characteristics, especially in terms of data distribution and cluster heterogeneity. Summary of the Invention

[0004] In view of this, embodiments of this application provide at least one stream computing heterogeneous cluster-aware scheduling method, apparatus, device, and storage medium.

[0005] The technical solution of this application embodiment is implemented as follows: On one hand, embodiments of this application provide a stream computing heterogeneous cluster-aware scheduling method, the method comprising: Collect the data volume of each key value in the key-value set and the processing capacity of each node in the node set, where the key-value set includes multiple keys and the node set includes multiple nodes; Based on the data volume and the processing capacity, the intermediate state is initialized, including setting the unassigned key-value set to the key-value set, setting the assigned load of each node to zero, and setting the popular key-value threshold. Based on the popular key value threshold and the amount of data for each key value, a popular key value set and an unpopular key value set are identified, wherein the popular key value set includes key values ​​with a data amount greater than the popular key value threshold, and the unpopular key value set includes key values ​​with a data amount not greater than the popular key value threshold; Aggregate the key values ​​in the set of non-popular key values, including sorting the key values ​​in the set of non-popular key values ​​in ascending order of data volume, and preferentially allocating the sorted key values ​​to nodes with lower processing capabilities until the allocated load of each node does not exceed its processing capability. Dispersing the key values ​​in the set of popular key values ​​includes sorting the key values ​​in the set of popular key values ​​in descending order of data volume, calculating the required number of nodes for each popular key value, and distributing the data of the popular key value to multiple nodes, wherein the nodes are selected in descending order of processing capacity; An allocation scheme is generated based on the aggregation and scattering results, and task scheduling is performed based on the allocation scheme.

[0006] On the other hand, embodiments of this application provide a stream computing heterogeneous cluster-aware scheduling device, the device comprising: The processing module is used to collect the data volume of each key value in the key-value set and the processing capacity of each node in the node set, wherein the key-value set includes multiple key values ​​and the node set includes multiple nodes; Based on the data volume and the processing capacity, the intermediate state is initialized, including setting the unassigned key-value set to the key-value set, setting the assigned load of each node to zero, and setting the popular key-value threshold. Based on the popular key value threshold and the amount of data for each key value, a popular key value set and an unpopular key value set are identified, wherein the popular key value set includes key values ​​with a data amount greater than the popular key value threshold, and the unpopular key value set includes key values ​​with a data amount not greater than the popular key value threshold; Aggregate the key values ​​in the set of non-popular key values, including sorting the key values ​​in the set of non-popular key values ​​in ascending order of data volume, and preferentially allocating the sorted key values ​​to nodes with lower processing capabilities until the allocated load of each node does not exceed its processing capability. Dispersing the key values ​​in the set of popular key values ​​includes sorting the key values ​​in the set of popular key values ​​in descending order of data volume, calculating the required number of nodes for each popular key value, and distributing the data of the popular key value to multiple nodes, wherein the nodes are selected in descending order of processing capacity; The scheduling module is used to generate an allocation scheme based on the aggregation and scattering results, and to execute task scheduling based on the allocation scheme.

[0007] In another aspect, embodiments of this application provide a computer device, including a memory and a processor. The memory stores a computer program that can run on the processor. When the processor executes the program, it implements some or all of the steps in the above-described heterogeneous cluster-aware scheduling method for stream computing.

[0008] In another aspect, embodiments of this application provide a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements some or all of the steps in the above-described stream computing heterogeneous cluster-aware scheduling method.

[0009] In another aspect, embodiments of this application provide a computer program including computer-readable code. When the computer-readable code is run in a computer device, the processor in the computer device executes some or all of the steps in the above-described heterogeneous cluster-aware scheduling method for stream computing.

[0010] In another aspect, embodiments of this application provide a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program. When the computer program is read and executed by a computer, it implements some or all of the steps in the above-described stream computing heterogeneous cluster-aware scheduling method.

[0011] This application embodiment realizes intelligent task scheduling in heterogeneous computing clusters, effectively balances the workload among nodes, reduces the variance of task completion time, improves cluster resource utilization, and reduces the number of small files generated through key-value aggregation, thereby improving the overall efficiency and stability of big data processing pipelines.

[0012] It should be understood that the above general description and the following detailed description are merely exemplary and explanatory, and are not intended to limit the technical solutions of this application. Attached Figure Description

[0013] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with this application and, together with the specification, serve to explain the technical solutions of this application.

[0014] Figure 1 A schematic diagram illustrating the implementation process of a heterogeneous cluster-aware scheduling method for stream computing provided in this application embodiment; Figure 2 A schematic diagram of the composition structure of a heterogeneous cluster sensing and scheduling device for stream computing provided in this application embodiment; Figure 3 This is a schematic diagram of the hardware entity of a computer device provided in an embodiment of this application. Detailed Implementation

[0015] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application are further described in detail below with reference to the accompanying drawings and embodiments. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0016] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0017] The terms “first / second / third” are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that “first / second / third” may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0018] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. The terminology used herein is for descriptive purposes only and is not intended to limit the scope of this application.

[0019] This application provides a heterogeneous cluster-aware scheduling method for stream computing, which can be executed by the processor of a computer device. The computer device can refer to a server, laptop, tablet, desktop computer, smart TV, set-top box, mobile device (such as a mobile phone, portable video player, personal digital assistant, dedicated messaging device, portable gaming device), or other similar computer equipment. Figure 1 This is a schematic diagram illustrating the implementation process of a heterogeneous cluster-aware scheduling method for stream computing provided in an embodiment of this application, as shown below. Figure 1 As shown, the method includes: Step 101: Collect the data volume of each key value in the key-value set and the processing capacity of each node in the node set, wherein the key-value set includes multiple key values ​​and the node set includes multiple nodes.

[0020] In this embodiment, the key-value set refers to a set of unique identifiers used to identify data records. Each key-value corresponds to a specific category or group in the data stream, such as a customer-domain combination in CDN logs. Data volume represents the total number of data records associated with each key-value within a specific time batch, reflecting the processing demand scale of that key-value. The node set refers to the set of available processing units in a computing cluster. Each node represents an independent instance of computing resources, such as a server or virtual task slot. Processing capacity refers to the maximum number of data records that each node can process within a given time window, determined based on node hardware configuration or historical performance metrics.

[0021] The system first extracts a set of key-value pairs from the data stream. By scanning the input data or querying external statistical services, it obtains the number of data records for each key-value pair within the current micro-batch. Simultaneously, the system accesses the cluster management interface or performance monitoring database to read the processing capacity parameters of each node. These parameters can be static configuration values ​​or dynamic evaluation metrics. The system stores the key-value data volume and node processing capacity information as structured data, such as a mapping table from key-value pairs to data volume and a mapping table from nodes to processing capacity, providing an input basis for subsequent scheduling decisions.

[0022] Step 102: Based on the data volume and the processing capacity, initialize the intermediate state, including setting the unassigned key-value set to the key-value set, setting the assigned load of each node to zero, and setting the popular key-value threshold.

[0023] In this embodiment, the intermediate state refers to a temporary data structure used to track the allocation progress during the scheduling process. The unallocated key-value set is the set of key-value identifiers that have not yet been assigned to any node. Allocated load represents the amount of data processing currently undertaken by each node. The hot key-value threshold is a numerical boundary used to distinguish between high-load and low-load key-values.

[0024] The system creates a data structure for an unassigned key-value set, initially containing all key-value identifiers. The system initializes the node load tracing table, setting the assigned load field of all nodes to zero. The system calculates a popular key-value threshold based on cluster configuration parameters or historical performance data; this threshold is typically determined by the product of the node's average processing capacity and a safety factor. These initialization operations establish the starting state of the scheduling algorithm, ensuring consistent initial conditions for subsequent allocation operations.

[0025] Step 103: Based on the popular key value threshold and the data volume of each key value, identify the popular key value set and the non-popular key value set, wherein the popular key value set includes key values ​​with a data volume greater than the popular key value threshold, and the non-popular key value set includes key values ​​with a data volume not greater than the popular key value threshold.

[0026] In this embodiment, the popular key-value set refers to key-value groups whose data volume exceeds the processing threshold, and these key-value groups require special processing strategies. The non-popular key-value set refers to key-value groups whose data volume is within an acceptable range, and is suitable for conventional allocation methods.

[0027] The system iterates through the key-value set, comparing the amount of data for each key-value pair with a popular key-value threshold. The system creates two independent sets: a popular key-value set containing all key-value identifiers with a data amount exceeding the threshold, and a non-popular key-value set containing all key-value identifiers with a data amount equal to or below the threshold. The system preprocesses these two sets using a quicksort algorithm to prepare them for subsequent allocation steps. This classification operation enables differentiated management of high-load and low-load key-value pairs.

[0028] Step 104: Aggregate the key values ​​in the set of non-popular key values, including sorting the key values ​​in the set of non-popular key values ​​in ascending order of data volume, and preferentially allocating the sorted key values ​​to nodes with lower processing capabilities until the allocated load of each node does not exceed its processing capability.

[0029] In this embodiment, aggregation refers to the process of merging multiple low-data-volume key-value pairs and assigning them to the same node. Ascending data order refers to arranging the key-value pairs in ascending order of their data volume.

[0030] The system performs an ascending sorting algorithm on the set of infrequently used key-value pairs, generating a sequence of key-value pairs from smallest to largest. The system selects nodes in ascending order of their processing capacity, starting with the node with the lowest processing capacity. For each selected node, the system iterates through the sorted list of key-value pairs, assigning unassigned key-value pairs to that node sequentially, while simultaneously accumulating the node's assigned load. When a node's remaining processing capacity is insufficient, the system switches to the next node with higher processing capacity to continue the assignment. This process continues until all infrequently used key-value pairs have been assigned, ensuring that low-volume key-value pairs are appropriately aggregated to matching nodes.

[0031] Step 105: Disperse the key values ​​in the popular key value set, including sorting the key values ​​in the popular key value set in descending order of data volume, calculating the required number of nodes for each popular key value, and distributing the data of the popular key value to multiple nodes, wherein the nodes are selected in descending order of processing capacity.

[0032] In this embodiment, "splitting" refers to the operation of splitting high-volume key-value pairs into multiple nodes for joint processing. "Data volume descending order sorting" means arranging key-value pairs in descending order of their data volume. "Number of nodes required" refers to the minimum number of nodes needed to process a single popular key-value pair.

[0033] The system performs a descending sorting algorithm on the popular key-value set, generating a sequence of key-value pairs from largest to smallest. For each popular key-value pair, the system calculates the required number of nodes, which is equal to the key-value data size divided by the average processing capacity of the nodes and rounded up. The system selects nodes in descending order of processing capacity, prioritizing nodes with high processing capacity. The system distributes the data records of the popular key-value pairs evenly across the selected nodes using a hash algorithm, while simultaneously updating the allocated load on these nodes. After each popular key-value pair is allocated, the system removes it from the unallocated set to ensure the allocation process progresses correctly.

[0034] Step 106: Generate an allocation scheme based on the aggregation and scattering results, and perform task scheduling based on the allocation scheme.

[0035] In this embodiment, the allocation scheme refers to a structured description of the mapping relationship between key values ​​and nodes. Task scheduling refers to the execution process of deploying computing tasks to corresponding nodes according to the allocation scheme.

[0036] The system integrates the allocation results and generates a complete key-value-node mapping table. The system encodes the allocation scheme into scheduling instructions and sends them to the computing framework via the cluster management interface. The system starts the task execution engine, configuring the data partitioning strategy and task deployment parameters according to the allocation scheme. The system monitors the task execution status to ensure that all key-value data is processed on the corresponding nodes according to the predetermined scheme.

[0037] This application embodiment realizes intelligent task scheduling in heterogeneous computing clusters, effectively balances the workload among nodes, reduces the variance of task completion time, improves cluster resource utilization, and reduces the number of small files generated through key-value aggregation, thereby improving the overall efficiency and stability of big data processing pipelines.

[0038] Optionally, step 101 includes: Step 1011: Obtain the data volume of each key value from an external statistics system that operates independently of the stream processing framework and provides request statistics based on the key value; or, obtain the data volume of each key value from the internal data source of the stream processing framework and maintain the count state of each key value by inserting aggregation operations in the stream processing pipeline.

[0039] In this embodiment, the data volume for each key-value pair is obtained from an external statistics system. The external statistics system is a computer system that operates independently of the stream processing framework. It provides key-value-based request statistics, such as the number of visits per client-domain, by collecting and analyzing network request data. The system obtains the total number of data entries corresponding to each key-value pair within the current micro-batch by querying the API or database interface of the external statistics system. This data serves as the basic input for perceptual scheduling. Alternatively, the system obtains the data volume for each key-value pair from the internal data source of the stream processing framework. By inserting aggregation operations into the stream processing pipeline, such as using Flink's ProcessFunction to maintain the count state of each key-value pair, and triggering statistical updates at the end of the micro-batch, the system calculates and stores the data volume for each key-value pair in real time.

[0040] Step 1012: Obtain the processing capacity of each node from static configuration information, which is pre-set based on the stress test results when the node is put on the rack, or obtain the processing capacity of each node from a dynamic monitoring system, which collects the real-time performance indicators of the nodes and evaluates their processing capacity.

[0041] In this embodiment, the processing capacity of each node is obtained from static configuration information. Static configuration information is a pre-defined data file based on stress test results when the node is deployed. It stores the processing capacity value of each node, such as the maximum number of data entries that each Task Slot can process under standard testing. The system loads this configuration information at startup, for example, by reading the mapping relationship between `task_slot_id` and `processing_capacity` from a JSON or YAML file; alternatively, the system obtains the processing capacity of each node from a dynamic monitoring system. This dynamic monitoring system is a continuously running computer system that collects real-time performance metrics of nodes, such as CPU utilization and throughput, and uses algorithms to evaluate the current processing capacity of each node. These evaluation results are then fed back to the scheduling system to support dynamic adjustments.

[0042] This application embodiment enables the system to generate an optimized task allocation strategy by accurately acquiring the amount of key-value data and the processing capabilities of nodes, thereby improving the execution efficiency and load balancing of stream processing tasks and reducing latency and small file issues.

[0043] Optionally, step 102 includes: Step 1021: Set the unassigned key-value set to the key-value set, indicating that all key-values ​​are initially unassigned.

[0044] In this embodiment, the unassigned key-value set is a data structure used to store key-value pairs that have not yet been assigned to any node. A key-value pair is a unique identifier for a data record; for example, in CDN log processing, a key-value pair might represent a combination of a customer and a domain name. The system maintains this set to track which key-value pairs have not yet been assigned processing tasks.

[0045] The system initializes an unassigned key-value set, setting it to contain elements from all key-value sets. This indicates that at the start of the task allocation process, all keys and values ​​are in an unassigned state. The system ensures that subsequent steps can perform allocation operations based on this state by iterating through each element in the key-value sets and adding them to the unassigned key-value set. This set serves as an intermediate state variable, used to monitor and update the key-value allocation progress, avoiding duplicate allocations or omissions.

[0046] Step 1022: Set the allocated load of each node to zero, indicating that the node does not bear any data load in the initial state.

[0047] In this embodiment, the allocated load is a numerical metric representing the amount of data currently being processed by a node, typically expressed in terms of the number of data entries or bytes. A node is a logical processing unit in a computing cluster; for example, in the Flink framework, a node corresponds to a Task Slot instance, responsible for executing a specific task.

[0048] The system initializes the allocated load to zero for each node. This means that at the start of the allocation process, all nodes are not processing any data load. The system iterates through the node set, resetting the allocated load attribute of each node to zero to ensure that subsequent allocation algorithms can accurately calculate the remaining processing capacity of each node. This initialization operation is fundamental to load balancing, preventing nodes from allowing historical load to affect the fairness of current task allocation.

[0049] Step 1023: Set a popular key value threshold. The popular key value threshold is determined based on the average or maximum value of the node's processing capacity and is used to distinguish the data size of key values.

[0050] In this embodiment, the popular key-value threshold is a numerical limit used to distinguish the data volume of key-value pairs. Key-value pairs with a data volume exceeding this threshold are marked as popular key-value pairs. Node processing capacity is the number of data entries a node can process per unit of time, reflecting the node's computational performance. The average value is the arithmetic mean of the set of node processing capacities, and the maximum value is the highest value in the set of node processing capacities.

[0051] The system calculates the average or maximum processing capacity of each node and sets a popular key-value threshold based on this. Specifically, the system first collects processing capacity data from all nodes, then calculates the average or selects the maximum value through arithmetic operations. This value is set as the popular key-value threshold to identify keys with excessively large amounts of data in subsequent steps. This threshold serves as a classification criterion, ensuring that popular keys are correctly identified and distributed across multiple nodes for processing, preventing overload of individual nodes.

[0052] Optionally, step 103 includes: Step 1031: Traverse each key value in the key value set and compare the data volume of the key value with the popular key value threshold.

[0053] In this embodiment, the key-value set is a collection containing multiple key-values, each key-value representing a unique identifier in the data stream for partitioning and data processing; the data volume refers to the number of data entries corresponding to each key-value in a specific time batch, reflecting the processing load of the key-value; the popular key-value threshold is a preset numerical limit used to distinguish whether a key-value belongs to the high-load category, and is set based on system configuration or historical performance data.

[0054] The system reads each key-value pair sequentially from the key-value set and retrieves its corresponding data volume value. The system then compares the data volume of each key-value pair with a popular key-value threshold, generating a comparison result. This process ensures that all key-value pairs are evaluated to determine their potential processing needs, providing input data for subsequent classification steps.

[0055] Step 1032: Include key values ​​whose data volume exceeds the popular key value threshold into the popular key value set.

[0056] In this embodiment, the popular key-value set is a storage container used to hold all key-value pairs whose data volume exceeds a threshold. These key-value pairs typically represent high-load tasks and require special handling to avoid latency.

[0057] Based on the comparison results, the system identifies all keys whose data volume exceeds the popular key-value threshold. The system adds these keys to the popular key-value set and updates the set state to reflect the new members. This operation ensures that high-load keys are centrally managed, facilitating subsequent distribution strategies.

[0058] Step 1033: Classify the key values ​​whose data volume is not greater than the popular key value threshold into the non-popular key value set.

[0059] In this embodiment, the non-hot key-value set is a storage container used to hold all key-value pairs whose data volume does not exceed a threshold. These key-value pairs represent low-load tasks and are suitable for aggregation processing to reduce small file issues.

[0060] Based on the comparison results, the system filters out all keys whose data volume does not exceed the popular key value threshold. These keys are then moved to a non-popular key value set, and the set's contents are adjusted to include these low-load entries. This step completes the binary classification of the keys, laying the foundation for subsequent aggregation and load balancing operations.

[0061] This application's embodiments achieve precise partitioning of data load through systematic traversal and classification of key-value pairs, ensuring that high-load key-value pairs are identified and centrally processed, while low-load key-value pairs are aggregated and managed. This optimizes task allocation, reduces processing latency and small file generation, and improves resource utilization efficiency and overall throughput in heterogeneous clusters.

[0062] Optionally, step 104 includes: Step 1041: Sort the key values ​​in the set of non-popular key values ​​in ascending order of data volume to generate a sorted sequence of non-popular key values.

[0063] In this embodiment, the non-popular key-value set refers to a set of key-value pairs whose data volume does not exceed a preset threshold, and the number of requests corresponding to these key-value pairs is relatively small. Data volume represents the number of records corresponding to each key-value pair in a specific batch. Ascending sorting is an algorithmic operation that arranges elements from smallest to largest numerical value. The sorted non-popular key-value sequence is an ordered list generated by a sorting algorithm, where key-value pairs are arranged from smallest to largest data volume.

[0064] The system first accesses the set of infrequently used key-value pairs, extracting the data volume attribute for each key-value pair. The system then applies a sorting algorithm, such as quicksort or mergesort, to sort the key-value pairs in ascending order. The sorting process is based on numerical comparisons of data volume, generating an ordered sequence. The system stores the sorting results in memory or a cache for use in subsequent steps. This operation ensures that key-value pairs are arranged in ascending order of data volume, providing a foundation for load balancing.

[0065] Step 1042: Sort the nodes in the node set in ascending order of processing capacity to generate a sorted node sequence.

[0066] In this embodiment, the node set refers to the collection of all logical task instances in the computing cluster, with each node representing an independent processing unit. Processing capacity represents the upper limit of the number of data entries a node can process within a specific time window, typically based on hardware configuration or historical performance evaluation. Ascending sorting is an algorithmic operation that arranges elements in ascending order of their numerical values. The sorted node sequence is an ordered list generated by a sorting algorithm, where nodes are arranged in ascending order of processing capacity.

[0067] The system retrieves the processing power value of each node from the node set. It then calls a sorting function, such as insertion sort or heap sort, to sort the nodes in ascending order. The sorting process is based on comparisons of processing power values, generating an ordered sequence. The system stores the sorted results in a data structure, such as an array or list. This operation ensures that nodes are arranged in ascending order of processing power, facilitating subsequent steps to prioritize task allocation to nodes with lower processing power.

[0068] Step 1043: Traverse the sorted sequence of nodes. For each node, select unassigned key values ​​from the sorted sequence of non-popular key values ​​and assign them to the node. At the same time, update the assigned load of the node until the assigned load of the node approaches or reaches its processing capacity.

[0069] In this embodiment, the system traverses the sorted sequence of nodes, processing each node sequentially. For the current node, the system checks the sorted sequence of non-hot key-value pairs and selects the unassigned key-value pair with the smallest data volume. The system calculates the sum of the node's allocated load and the key-value pair's data volume; if the sum does not exceed the node's processing capacity, the key-value pair is assigned to the node. The system updates the node's allocated load attribute, increasing the data volume of the key-value pair. The system removes the key-value pair from the unassigned key-value pair set and marks it as assigned. This process is repeated until the node's allocated load approaches or reaches its processing capacity limit, at which point the system moves to the next node to continue allocation.

[0070] This application implements load balancing through a sorting and traversal mechanism. The system prioritizes assigning key-value pairs with smaller data volumes to nodes with lower processing capabilities to avoid resource waste. The system dynamically updates node load to ensure that task allocation matches node capabilities.

[0071] Optionally, step 105 includes: Step 1051: Sort the key values ​​in the popular key value set in descending order of data volume to generate a sorted popular key value sequence.

[0072] In the embodiments of this application, a key value refers to a unique identifier used to identify and group data in a data stream. For example, in CDN access logs, a key value may represent a combination of customer domain names to distinguish different data sources. The data volume represents the total number of data entries contained in a given batch for a specific key value, reflecting the processing load. Descending sorting is an algorithmic operation that arranges elements from largest to smallest to ensure that elements with higher priority are processed first.

[0073] The system first accesses a set of popular key-value pairs, which contains key-value pairs with a data size exceeding a preset threshold. The system extracts the data size attribute for each key-value pair and applies a sorting algorithm, such as quicksort or mergesort, to sort these key-value pairs in descending order. The sorting process is based on numerical comparisons of data size, generating an ordered sequence where the key-value pair with the largest data size is located at the beginning of the sequence. The system outputs the sorted sequence of popular key-value pairs, providing a basis for the processing order of subsequent steps.

[0074] Step 1052: For each popular key value in the sorted popular key value sequence, calculate the number of nodes required to process the popular key value. The number of nodes required is determined based on the ratio of the amount of data of the popular key value to the processing capacity of the nodes.

[0075] In this embodiment, the number of nodes refers to the total number of computing nodes required to process a single popular key-value pair, which is used to ensure that the data load is reasonably distributed; the node processing capacity represents the number of data entries that a single node can process per unit time, which is usually evaluated based on hardware configuration or historical performance data; the ratio is determined by division operation to determine the proportional relationship between two values, which is used to quantify resource requirements.

[0076] The system iterates through the sorted sequence of popular key-value pairs, retrieving the data volume for each popular key-value pair. Simultaneously, the system obtains node processing capacity parameters, which may come from static configuration or dynamic monitoring data. The system performs a division calculation, dividing the data volume of each popular key-value pair by the node processing capacity to obtain the theoretically required number of nodes. If the result is a decimal, the system applies a rounding function to ensure the number of nodes is an integer. The calculation results are stored as attributes for each popular key-value pair to guide subsequent node allocation decisions.

[0077] Step 1053: Sort the nodes in the node set in descending order of processing capacity to generate a sorted node sequence.

[0078] In this embodiment of the application, the node set refers to the group of all available computing nodes in the cluster, and each node represents an independent processing unit; processing capacity is the performance index of a node, which quantifies its data processing rate; descending sorting arranges the nodes from high to low processing capacity, giving priority to the use of high-performance resources.

[0079] The system accesses a set of nodes and reads the processing power value of each node, which may be obtained through stress testing or real-time metric collection. The system applies a sorting algorithm, such as heap sort or insertion sort, to sort all nodes in descending order of processing power. The sorting process ensures that the node with the highest processing power is at the front of the sequence. The system generates a sorted sequence of nodes, which serves as the basis for resource allocation and optimizes task scheduling efficiency.

[0080] Step 1054: Select available nodes from the sorted node sequence, assign data to popular key values, and update the assigned load of the nodes.

[0081] In this embodiment, an available node refers to a computing node that is not currently fully loaded and can accept new tasks; the allocated load represents the amount of data processing that a node is currently undertaking, used to track resource usage status; data allocation is the process of splitting popular key-value data and assigning it to multiple nodes to achieve parallel processing.

[0082] The system initiates the allocation process based on the sorted node sequence and the number of nodes required for popular keys. Starting from the beginning of the node sequence, the system checks the remaining processing capacity of each node (node ​​processing capacity minus allocated load). For each popular key, the system selects the required number of available nodes, ensuring that the total remaining capacity of these nodes can cover the data load. The system splits the data for the popular key into multiple subsets and allocates them to the selected nodes using a hashing or round-robin mechanism. Simultaneously, the system updates the allocated load field of each node, increasing the amount of newly allocated data. After allocation, the system removes the popular key from the pending set and records the allocation result for task execution.

[0083] This application's embodiments implement intelligent distribution of popular key-value pairs, combined with optimized scheduling based on node processing capabilities. The overall process ensures that key-value pairs with large data volumes are distributed across multiple nodes, avoiding single-point overload, while nodes with high processing capabilities are prioritized for heavy-load tasks. This significantly reduces task execution latency, improves the throughput and resource utilization of heterogeneous clusters, and guarantees that data processing tasks are completed within the specified time.

[0084] Optionally, step 106 includes: Step 1061: Based on the results of aggregation and scattering operations, construct an allocation function. The allocation function defines the mapping from each key value to a subset of nodes. The allocation function is used to guide data partitioning operations in the stream processing framework. The allocation scheme is distributed to stream processing tasks in a broadcast state to ensure that all tasks update the allocation information synchronously.

[0085] In this embodiment, the allocation function is a mathematical mapping that associates each key value with a subset of one or more nodes. The key value represents an identifier used for partitioning in the data stream, such as a client domain name or request ID. The node subset refers to a set of logical task instances that process data in a computing cluster, such as a Task Slot in Flink. Stream processing frameworks are software systems used for real-time processing of data streams, such as Apache Flink or Spark Streaming. Broadcast state is a mechanism in stream processing frameworks that allows shared data to be distributed to all parallel task instances, ensuring global consistency. Data partitioning is the process of allocating a data stream to different nodes based on key values ​​to achieve parallel processing.

[0086] The system constructs an allocation function based on the results of aggregation and shuffling operations. This allocation function maps each key-value pair to a subset of nodes, specifically implemented as a mathematical function f: K → P(N), where K is the set of key-value pairs and P(N) is the power set of the node set. The system uses this function to guide data partitioning operations within the stream processing framework, for example, through a custom Partitioner interface in Flink. The allocation scheme is distributed in a broadcast state; the system encapsulates the allocation function as a broadcast stream and sends it to all task instances via the framework's BroadcastState mechanism. Task instances receive and update their local state, ensuring synchronized allocation information and thus applying consistent mapping rules during data stream processing.

[0087] This application's embodiments achieve balanced data load distribution and optimized utilization of heterogeneous nodes by initializing states, aggregating unpopular key values, opening up unpopular key values, and constructing allocation functions. This reduces processing latency, avoids the problem of too many small files, and improves overall throughput and resource efficiency.

[0088] Optionally, step 106 includes: Step 1062: Implement a custom partitioner in the stream processing framework, wherein the custom partitioner routes data to a specified node according to the allocation scheme.

[0089] In this embodiment, a stream processing framework is a software system for real-time processing of continuous data streams. It supports distributed computing and parallel task execution; common examples include Apache Flink or Spark Streaming. A custom partitioner is a user-defined component in the stream processing framework used to distribute data to different computing nodes based on specific logic (such as key-value pairs or allocation schemes).

[0090] The system implements a custom partitioner within the stream processing framework. This partitioner receives an allocation scheme as input, defining the target node to which each key-value or data segment should be routed. The custom partitioner calculates the target index for each data segment based on the node mappings in the allocation scheme by overriding the framework's partitioning interface methods (e.g., implementing the Partitioner interface in Apache Flink). The system takes the key-value pairs from the data stream as input parameters and calls the partitioner methods to return node identifiers, ensuring that data is accurately sent to the specified nodes. This process is executed automatically during the network transmission phase of the data stream through the framework, requiring no manual intervention and achieving efficient data distribution.

[0091] Step 1063: Configure the task affinity policy to prioritize scheduling task instances to physical nodes with matching processing capabilities.

[0092] In this embodiment, the task affinity policy is a configuration rule in the stream processing framework used to control the scheduling preferences of task instances on physical nodes, such as prioritizing the allocation of computationally demanding tasks to high-performance nodes. Physical nodes are the actual server hardware in the cluster, each with specific processing capabilities (such as CPU or memory resources).

[0093] The system configures a task affinity policy, defining matching rules between task instances and physical node attributes by modifying the scheduling configuration file or API parameters of the stream processing framework. Based on the node processing capacity information (such as CPU capacity or historical throughput) in the allocation scheme, the system sets the scheduler to prioritize assigning task instances to physical nodes with matching processing capabilities. For example, in Apache Flink, the system uses resource management components (such as JobManager) to evaluate node resource metrics and applies affinity rules (such as node labels or resource constraints) to allocate tasks. This process is executed automatically during the task submission phase, ensuring that task instances fully utilize node capabilities at runtime and reducing processing latency.

[0094] Step 1064: Request computing resources with specific tags through the scheduler plugin of the stream processing framework to ensure the task execution environment.

[0095] In this embodiment, the scheduler plugin is an extension component of the stream processing framework used to interact with the underlying resource management system (such as Kubernetes) to dynamically request and allocate computing resources. Specific tags are metadata identifiers in the resource management system used to mark computing resources with specific attributes (such as high performance or heterogeneous type), for example, tags for Pods in Kubernetes.

[0096] The system sends resource requests to the underlying resource management system (such as a Kubernetes cluster) through the scheduler plugin of the stream processing framework. These requests contain specific tags (such as `flink.processing-group-type`) to identify the type of computing resources required. The scheduler plugin parses the node requirements in the allocation scheme, generates a resource description file (such as a YAML configuration), and specifies tag matching rules and resource quotas (such as CPU or memory). During task deployment, the system calls the plugin interface, and the resource management system filters available nodes based on the tags and allocates Pods (task managers), ensuring that the task execution environment has the required processing capacity. This process is completed automatically during framework initialization or task expansion, ensuring stable task operation in heterogeneous clusters.

[0097] This application's embodiments ensure data is routed to designated nodes according to the allocation scheme through a custom partitioner, configure a task affinity policy to prioritize matching task instances with the processing capabilities of physical nodes, and use a scheduler plugin to request specific tag resources to guarantee the execution environment. This effectively solves the problems caused by uneven node capabilities and long-tail data distribution in heterogeneous clusters, improves the uniformity of task completion time, reduces the generation of small files, and enhances system throughput and resource utilization.

[0098] Optionally, step 1011 includes: querying the access statistics module in the content delivery network to obtain the number of requests with customer and domain name as key values; the access statistics module aggregates request logs in real time and outputs statistical results; the latency of the statistical results is lower than the processing latency of the stream processing framework, ensuring the timeliness of data volume information.

[0099] In this embodiment, a content delivery network (CDN) is a distributed computer system connected via the internet. It utilizes servers deployed near users to efficiently and reliably transmit files such as music, images, videos, and applications, providing high-performance, scalable, and low-cost network content distribution services. An access statistics module is a component of the CDN, specifically responsible for collecting and aggregating user access logs in real time and generating statistical results, such as the number of requests. A stream processing framework is a big data processing technology used for real-time or near-real-time processing of continuous data streams, such as Spark Streaming or Flink. Processing latency refers to the time interval required from data input to output. Data volume information represents the total number of requests for a specific key-value pair (such as client and domain name) within a given time period. Timeliness refers to the validity and timeliness of data or information, ensuring that it reflects the current state.

[0100] The system queries the access statistics module by calling the content delivery network's application programming interface (API) or directly accessing its database. This module processes inbound request logs in real time, using aggregation algorithms (such as counting or summing) to calculate the number of requests for each client and domain combination, and stores the results in a cache or database. The system verifies the timestamp of the statistical results to ensure its latency is lower than the micro-batch processing interval of the streaming framework, thus guaranteeing that the data volume information is up-to-date for scheduling decisions. Query results are returned in a structured format (such as a list of key-value pairs), which the system loads into memory as input parameters for subsequent task allocation algorithms. The entire process is executed through automated scripts or built-in functions without manual intervention, ensuring high efficiency and consistency.

[0101] This application embodiment significantly reduces task execution latency and file storage overhead by acquiring data volume information in real time, dynamically aggregating unpopular and popular key values, and optimizing allocation based on node processing capabilities.

[0102] Optionally, step 1011 includes: Step 10111: Insert a counting operation after the source operation in the stream processing pipeline, the counting operation using a state backend to maintain key-value counts.

[0103] In this embodiment, a stream processing pipeline is a data processing architecture where data is processed in the form of a continuous stream. Source operations are components in the stream processing pipeline responsible for reading raw data from external systems (such as message queues or log files). Counting operations are computational functions used to count the number of times a specific key-value pair appears in the data stream. The state backend is a storage mechanism in the stream processing framework used to persist and manage intermediate computational states (such as counter values). Key-value counting refers to the cumulative counting of records corresponding to each unique key-value pair in the data stream.

[0104] The system deploys a counting operation module after the source operations in the stream processing pipeline. This module uses a state backend to store the cumulative count for each key-value pair. The state backend can be a distributed database or in-memory storage to ensure the consistency of the counting data during fault recovery. When data flows through the counting operation, the system updates the corresponding counter based on the key-value pair of each record, for example, by incrementing the key-value pair's count through an incremental operation. This process runs continuously without interrupting the data flow, thus maintaining the statistical information of the key-value pairs in real time.

[0105] Step 10112: Output the key-value count result at the end of the micro-batch by using a timer trigger mechanism.

[0106] In this embodiment, the timer triggering mechanism is a feature within the stream processing framework, allowing predefined operations to be automatically executed at specific times or under specific event conditions. Micro-batching is a processing method in stream processing that divides a continuous data stream into fixed time windows or data volume units; each micro-batch represents an independent data processing cycle. The key-value counting result is structured data output from the counting operation, containing each key-value pair and the total number of corresponding records within the micro-batch.

[0107] The system is configured with a timer triggering mechanism that automatically activates at the end of a micro-batch processing cycle. The timer leverages the stream processing framework's built-in time management features, such as event time or processing time windows, to ensure precise triggering at micro-batch boundaries. When the timer is activated, the system retrieves the current counts of all key-value pairs from the state backend, generating a key-value count result set. This result set is output as a data object, such as a list of key-value pairs containing key-value identifiers and integer values, for use in subsequent steps.

[0108] Step 10113: Send the key-value counting results to the coordination service or shared storage for use by the scheduling algorithm.

[0109] In this embodiment, the coordination service is a centralized component in a distributed system used to manage task allocation and state synchronization, such as Apache ZooKeeper or etcd. Shared storage is a persistent storage system accessible to multiple nodes, such as a distributed file system or cloud storage service. The scheduling algorithm is a computational logic used to optimize task allocation and resource scheduling based on input data (such as key-value counts).

[0110] The system sends the key-value counting results to the coordination service or shared storage via a network transmission interface. During transmission, the data is encapsulated using a serialization protocol (such as JSON or Protobuf) to ensure format compatibility. Upon receiving the data, the coordination service stores it in the shared state area for real-time access by the scheduling algorithm module. For example, the system might call the coordination service's API to write data or directly save the results to a specified path in the shared storage. The scheduling algorithm then reads the key-value counting results from this location and uses them as input parameters for task allocation decisions.

[0111] The embodiments of this application ensure the accuracy and fault tolerance of data statistics through counting operations and a status backend, the timer mechanism ensures that the statistical results are output in micro-batch cycles, and the coordination service or shared storage provides an efficient data sharing channel.

[0112] Optionally, step 1012 includes: loading a configuration file when the stream processing application starts, the configuration file containing a mapping relationship between node identifiers and processing capabilities, the processing capabilities being predefined based on node hardware specifications and benchmark test results, the configuration file being stored in text format and parsed through the configuration interface of the stream processing framework.

[0113] In this embodiment, the configuration file is a data file containing system operating parameters and settings, used to store the mapping relationship between node identifiers and processing capabilities, and is usually in text format for easy reading and modification.

[0114] During the startup phase of the stream processing application, the system reads the configuration file from the specified storage location by calling the configuration interface of the stream processing framework, parses the file content to extract the mapping data between node identifiers and processing capabilities, and loads this data into memory to provide node performance benchmark information for subsequent scheduling algorithms.

[0115] This application embodiment enables the system to accurately obtain node processing capability data by pre-loading and parsing configuration files, ensuring that scheduling decisions are based on reliable performance benchmarks, thereby optimizing task allocation and improving the overall processing efficiency and resource utilization of heterogeneous clusters.

[0116] Optionally, step 1012 includes: Step 10121: Collect node performance data through the metrics system of the stream processing framework. The performance data includes CPU utilization, memory utilization, and backpressure metrics.

[0117] In this embodiment, the stream processing framework is a software system for processing continuous data streams. It executes tasks through distributed computing nodes and provides a built-in metrics system to monitor node operating status. CPU utilization refers to the percentage of the node's central processing unit used, reflecting the computing load; memory utilization refers to the proportion of the node's random access memory occupied, indicating data caching pressure; backpressure metric is a measure in the stream processing framework used to measure the imbalance between data production and consumption rates. It is triggered when the downstream processing speed is lower than the upstream data input to avoid system overload.

[0118] The system automatically collects performance data from each compute node through the streaming processing framework's metrics system. Specifically, the metrics system periodically reads CPU utilization values ​​from the node's operating system, obtains memory usage statistics from the memory management module, and monitors backpressure status (such as data backlog length or latency) from the streaming engine's internal queues. This data is formatted into structured metrics (such as key-value pairs or time series) and stored in the framework's local cache or shared storage for access by subsequent steps. The entire process requires no manual intervention, ensuring the real-time nature and consistency of data collection.

[0119] Step 10122: Use an external monitoring service to aggregate performance data and calculate the real-time processing capability of the node.

[0120] In this embodiment, the external monitoring service is a software component independent of the stream processing framework, specifically designed for the centralized collection, storage, and analysis of distributed system metrics. Aggregation refers to the process of merging performance data from multiple nodes into a unified view, eliminating redundancy and improving data consistency. Real-time processing capability is the number of data entries or tasks a node can process per unit of time, derived from performance metrics, and used to quantify the current computational efficiency of a node.

[0121] The system collects performance data and sends it to an external monitoring service. This service receives the data via a network interface and uses aggregation algorithms (such as averaging, percentiles, or weighted calculations) to merge multi-node metrics. Subsequently, the service calculates the real-time processing capacity of each node based on predefined models (such as linear regression or resource utilization formulas), for example, deriving the maximum sustainable throughput by combining CPU utilization and backpressure metrics. The calculation results are standardized into numerical form (such as records processed per second) and stored in the monitoring service's database, ensuring that the data can be queried in real time by the scheduling algorithm.

[0122] Step 10123: Feed back the real-time processing capability to the scheduling algorithm and dynamically adjust the allocation strategy.

[0123] In this embodiment, the scheduling algorithm is the program logic within the stream processing framework that determines how tasks are allocated to nodes; it optimizes resource utilization based on input parameters. Feedback refers to passing the calculated real-time processing capacity as input to the scheduling algorithm. Dynamic adjustment means that the scheduling algorithm automatically modifies the task allocation rules based on new data without restarting the system. The allocation strategy defines the mapping relationship between tasks and nodes, for example, prioritizing the allocation of high-load tasks to high-capacity nodes.

[0124] The system transmits the calculated real-time processing capacity data to the scheduling algorithm module via the application programming interface (API). The scheduling algorithm parses this data and recalculates the allocation strategy based on optimization objectives (such as minimizing task completion time or balancing load), for example, updating task distribution weights or node priority lists. The adjusted strategy takes effect immediately, and the stream processing framework dynamically allocates subsequent tasks to the corresponding nodes according to the new strategy, while the monitoring module continuously verifies the effectiveness of the strategy. This process is executed cyclically to ensure that the allocation strategy always adapts to changes in the cluster state.

[0125] This application's embodiments effectively solve the problem of inconsistent task execution times caused by differences in node configurations by dynamically calculating node processing capabilities based on performance data and feeding this data back to the allocation strategy. This improves cluster throughput, reduces task latency, and lowers operational costs through resource optimization.

[0126] Optionally, CDN offline logs represent a typical big data ETL scenario. For cloud providers, access logs generated by the entire CDN service are collected into a unified, small log stream. At the processing center, a big data stream processing engine extracts logs for each customer and each domain from the log stream. These logs are then converted according to the customer-specified format, and the logs at each time point are stored as intermediate files and uploaded to S3 or HDFS. Finally, a scheduled batch processing task aggregates the intermediate files from multiple time points into the time granularity required by the customer, such as five minutes, one hour, or even one day. Different customers have different latency requirements; therefore, it is desirable for the stream processing steps to complete as promptly as possible. Only by providing low-latency intermediate files can the batch processing task avoid exceeding the customer's expected latency.

[0127] Typically, this type of ETL scenario consists of two tasks (a read task and an output task), connected by a partitioning operation. The partitioning (Shuffle) operation distributes data to specific output tasks based on the key value of each data item, thus making the unordered data stream partially ordered. This mechanism is generally used to construct tasks with appropriate processing capacity, thereby parallelizing the problem being processed.

[0128] Most big data operations need to address the data skew problem caused by long-tail distribution. In this scenario, where the customer-domain name is used as the task key, the CDN usage of different customer-domain names often varies significantly within the same time period. A few key-value pairs generate a large number of requests, while most key-value pairs receive only a few. This leads to two problems: 1. High-volume key-value pairs require longer processing times, and the total processing time is determined by the longest task execution time, thus causing data processing delays.

[0129] 2. A large number of low-usage key-value pairs will generate a large number of small files, increasing the metadata burden on file storage services.

[0130] Furthermore, computing clusters can encounter node fragmentation issues. For example, after a period of use, as nodes fail and are replaced, performance inconsistencies may gradually arise between old and new nodes within the cluster; or, due to cost reduction and efficiency improvements, failed nodes may have to be replaced with older, more existing servers; and heterogeneous nodes may join the cluster due to domestic substitution. If a computationally intensive task is scheduled to a less powerful node, it can cause the task to time out, compromising service availability. Current solutions to these problems include: Solution 1: To address the data skew issue, during the partitioning phase, different keys are randomly distributed to each task to ensure a consistent total processing volume for all tasks.

[0131] Option 2: To address the issue of small files, different processing channels are used for customer businesses with different workloads during the pre-processing phase.

[0132] Solution 3: Address the heterogeneous cluster problem: Separate nodes with different configurations into multiple independent clusters to handle different services, ensuring uniformity within the same cluster.

[0133] However, the above solutions still have limitations. Solution 1 cannot handle cases with severe distribution skew, i.e., where the number of key-value pairs is less than the number of nodes. Solutions 2 and 3 both require modifications to the system architecture, making smooth migration impossible and limiting the horizontal expansion of processing capacity. Therefore, this application proposes a heterogeneous cluster-aware scheduling method for stream computing, which can combine the amount of data processed by key-value pairs with the processing capacity of each node to generate tasks with similar completion times as much as possible.

[0134] The main strategy of the heterogeneous cluster-aware scheduling method for stream computing provided in some embodiments of this application is: under the premise of adapting to the processing capabilities of nodes, popular key values ​​are selected and less popular key values ​​are aggregated. Therefore, before micro-batch processing begins, it is necessary to first obtain the number of data entries corresponding to all key values, and the number of data entries that all nodes can process within the current batch time. Based on the above data, awareness scheduling can be implemented, and a reasonable partitioning strategy can be generated using the following steps: Collect the amount of data corresponding to the key-value pairs, as well as the processing capacity of the nodes, and initialize the intermediate state.

[0135] Aggregate less popular key values. Allocate nodes in descending order of processing capacity, prioritizing nodes with lower processing capacity to process less popular key values, and assign multiple less popular key values ​​to each node until all are allocated.

[0136] Create a hot key value. Mark nodes that exceed the processing capacity of a single machine as hot keys, and use hashing to distribute them across multiple nodes during partitioning.

[0137] To implement the strategy of "targeting high-demand keys and aggregating low-demand keys," and considering the processing capacity of each node to ensure tasks are completed as simultaneously as possible, the following steps are required for each processing batch. First, the input required for planning the task scheme is: Key-value sets: in The number of key values.

[0138] Data volume: per key-value pair Each batch corresponds to a certain number of data entries, represented as Therefore, the amount of data corresponding to all key-value pairs can be represented as a set: ; Node set: in The number of nodes.

[0139] Estimated processing capacity: Node The number of data entries that can be processed within a time window is expressed as follows: .

[0140] The expected output is an allocation scheme, indicating which node handles each key-value pair or a portion thereof. Therefore, it can be represented as an allocation function. .in, It is a set of nodes The power set, i.e. Returns a specific set of nodes. The set is A subset of. In practical implementation, this means planning tasks based on this set: if = 1 means All the data will be processed by a single node.

[0141] if >1 indicates All the data needs to be distributed across multiple nodes.

[0142] Initialization: Before allocation begins, the intermediate state needs to be initialized, and necessary configurations need to be obtained. For each node, the allocated load needs to be calculated. During initialization, for All .use This represents the set of key-value pairs that have not yet been allocated, and is initialized during this process. Additionally, a threshold is needed to identify popular key-value pairs that a single node cannot process, defined as follows: .all key value All included in the popular key-value set The remaining key values ​​are recorded as unpopular key values. ,Right now .

[0143] Aggregate less popular key-value pairs: The purpose of this step is to aggregate key-value pairs that have a small data volume ( The key-value pairs are aggregated and preferentially distributed to nodes with lower processing capacity to achieve load balancing and avoid generating too many small files.

[0144] Get key-value pairs and sorted by data volume Sort in ascending order.

[0145] From processing capacity Traverse in ascending order to obtain nodes ,for Each key value in ,if Not yet allocated, i.e. And the nodes still have spare capacity, that is Then continue to perform the following operations: Will Assigned to :Right now ; Update node load: ; Will Marked as allocated: Unallocated set Remove from ; If during the above process, there are still If a key-value pair fails to be assigned to a low-capacity node (possibly because low-capacity nodes are full), repeating the above attempts will gradually assign them to nodes with higher processing capabilities in ascending order of node capacity. Therefore, traversing nodes in ascending order of processing capability automatically ensures that less popular key-value pairs will eventually be filled by high-capacity nodes.

[0146] Key values ​​for opening the heat dissipation gate: This step involves handling very large amounts of data using popular key values. (Key-value pairs). This key-value data will be distributed across multiple nodes for collaborative processing.

[0147] Get key-value pairs and sorted by data volume Sort in descending order.

[0148] Traversal key-value pairs Calculate the required number of nodes Then each node needs to be loaded. From the perspective of processing capacity Traverse in descending order to obtain nodes If the node still has remaining capacity, that is... If 0, continue performing the following operations until a match is found. Each node.

[0149] Will Assigned to :Right now ; Update node load: (Approximately increased load; how the actual data is precisely and evenly distributed among these nodes will be determined by the underlying hashing mechanism at runtime); Will Marked as allocated: Unallocated set Remove from ; By allocating tasks to each node according to their processing capabilities, the problem of uneven processing power among heterogeneous nodes can be effectively addressed. This ensures that task execution times are similar across nodes while fully utilizing processing power. Furthermore, by aggregating key-value pairs, the number of small files is reduced.

[0150] The above descriptions of heterogeneous cluster-aware scheduling methods for stream computing have been simplified to illustrate the principles and to ensure independence from specific stream processing engines. Therefore, the implementation details of these methods will now be explained using Flink. The implementation principle for Spark Streaming is similar. First, we need to map the theoretical concepts to the actual components of Flink: Node: A logical Task. In Flink, a physical machine (node) can be configured with multiple Task Slots. Each Task Slot can run one or more concurrent instances of Tasks. Therefore, the node described above is actually a concurrent instance of a logical Task capable of independently processing data. The number of nodes discussed above should actually be greater than the number of servers in the actual cluster, equivalent to each physical server being divided into multiple virtual nodes.

[0151] Key-value pairs: Flink still uses Key as the key concept for processing and natively supports Keyed Streams. Its keyBy() operation partitions the data based on the key-value pairs.

[0152] Read and output tasks: corresponding to Flink's Source Operator and Sink Operator.

[0153] Partitioning operation (Shuffle): In Flink, the keyBy() operation introduces a network shuffle into the data stream.

[0154] Initialization: First, it is necessary to quickly count the key-value pairs within the current micro-batch. The corresponding total number of data entries As the foundation of perception-based scheduling, there are two implementation schemes in practice: External data sources. In CDN offline log scenarios, the number of requests per client-domain over a period of time can actually be calculated by other systems, such as Nginx's VTS statistics system, which can use the client and domain as specified keys to count requests. The latency of this system is significantly lower than that of statistics based on detailed logs, so information from external systems can be used directly.

[0155] Internal data source. For application scenarios lacking external statistical methods, this can be achieved by inserting a lightweight aggregation or counter operation early in the stream processing pipeline (e.g., before keyBy()). For example, Flink's ProcessFunction or FlatMapFunction can be used in conjunction with state to maintain a count for each key-value pair, and these statistics can be sent to a coordination service or shared storage at the end of the micro-batch (or triggered by a timer).

[0156] Secondly, it is necessary to estimate the processing capacity of the nodes and obtain the current processing capacity of each Task Slot (or concurrent instance of a logical Task). This can be a pre-configured static value (e.g., obtained from stress testing when the server is deployed) or a value dynamically evaluated based on historical performance. This is crucial for heterogeneous clusters.

[0157] Static configuration: If the cluster has a single purpose, such as only processing CDN offline logs, a small stress test can be performed using the corresponding dataset after the server is deployed. Capacity values ​​for each Task Slot can be passed as configuration parameters when the Flink application starts. For example, a JSON or YAML file containing a mapping of task_slot_id:processing_capacity can be loaded when the Job Manager starts.

[0158] Dynamic evaluation: The real-time processing capability of a Task Slot can be dynamically evaluated by monitoring Flink Task Manager metrics such as CPU utilization, throughput, and backpressure. These metrics can be exposed by the Flink MetricsSystem and collected by external monitoring systems (such as Prometheus), then fed back to the coordination service.

[0159] The algorithm's input and output are processed as follows: First, the allocation strategy generated based on the collected information needs to be promptly distributed to Flink's Source Tasks and Shuffle phase. Therefore, Flink's BroadcastState is used to send the allocation strategy as a broadcast stream to all relevant Tasks (e.g., Source Tasks or custom partitioners). All Tasks can access and update this broadcast state.

[0160] After synchronizing, a custom partitioner is needed, and new keys should be constructed using a hash value from the split keys to distribute the data. In CDN offline log scenarios, the request ID is used as the distribution hash value for a single data entry by default; for other scenarios, millisecond-level timestamps or a round-robin approach can be used. Importantly, these strategies need to be synchronized to the partitioner beforehand to ensure that the input keys are recognizable. Therefore, a custom partitioner needs to be implemented. Flink allows users to implement custom Partitioners. <key>The interface has only one method, int partition(KEYkey, int numPartitions), which takes a key value and the total parallelism, and returns an integer indicating the index of the Task Slot where the key value should be sent.

[0161] Finally, to associate specific tasks with corresponding physical servers, a custom affinity configuration strategy needs to be implemented. By default, the Flink JobManager requests TaskSlots fairly, without requesting TaskSlots from a specific NodePool based on the Key type processed internally by the Task. Therefore, a custom Flink scheduler plugin needs to be implemented that can identify the target machine type specified by the Task operator (e.g., Processing_Group_ID) after keyBy and request the Pod (TaskManager) with the corresponding flink.processing-group-type tag from the Kubernetes scheduler.

[0162] 1. By considering the actual processing capabilities of nodes when constructing tasks, it is possible to complete computing tasks within a specified time in a large-scale heterogeneous cluster.

[0163] 2. By aggregating less common key-value pairs, we can adapt to actual processing capabilities while reducing the generation of small files and lowering the metadata maintenance pressure of object storage.

[0164] Based on the foregoing embodiments, this application provides a stream computing heterogeneous cluster awareness scheduling device. The device includes various units and modules included in each unit, which can be implemented by a processor in a computer device; of course, it can also be implemented by specific logic circuits. In the implementation process, the processor can be a central processing unit (CPU), a microprocessor unit (MPU), a digital signal processor (DSP), or a field programmable gate array (FPGA), etc.

[0165] Figure 2 This application provides a schematic diagram of the composition structure of a heterogeneous cluster-aware scheduling device for stream computing, as shown in the embodiments of this application. Figure 2 As shown, the stream computing heterogeneous cluster awareness scheduling device 20 includes: The processing module 201 is used to collect the data volume of each key value in the key-value set and the processing capacity of each node in the node set, wherein the key-value set includes multiple key values ​​and the node set includes multiple nodes; Based on the data volume and the processing capacity, the intermediate state is initialized, including setting the unassigned key-value set to the key-value set, setting the assigned load of each node to zero, and setting the popular key-value threshold. Based on the popular key value threshold and the amount of data for each key value, a popular key value set and an unpopular key value set are identified, wherein the popular key value set includes key values ​​with a data amount greater than the popular key value threshold, and the unpopular key value set includes key values ​​with a data amount not greater than the popular key value threshold; Aggregate the key values ​​in the set of non-popular key values, including sorting the key values ​​in the set of non-popular key values ​​in ascending order of data volume, and preferentially allocating the sorted key values ​​to nodes with lower processing capabilities until the allocated load of each node does not exceed its processing capability. Dispersing the key values ​​in the set of popular key values ​​includes sorting the key values ​​in the set of popular key values ​​in descending order of data volume, calculating the required number of nodes for each popular key value, and distributing the data of the popular key value to multiple nodes, wherein the nodes are selected in descending order of processing capacity; The scheduling module 202 is used to generate an allocation scheme based on the aggregation and scattering results, and to execute task scheduling based on the allocation scheme.

[0166] Optionally, the processing module 201 is further configured to: The data volume for each key-value pair can be obtained from an external statistics system that operates independently of the stream processing framework and provides request statistics based on the key-value pair. Alternatively, the data volume for each key-value pair can be obtained from an internal data source of the stream processing framework, and the count status of each key-value pair can be maintained by inserting aggregation operations into the stream processing pipeline. The processing capacity of each node can be obtained from static configuration information, which is pre-set based on the stress test results when the node is put on the rack, or the processing capacity of each node can be obtained from a dynamic monitoring system, which collects the real-time performance indicators of the nodes and evaluates their processing capacity.

[0167] Optionally, the processing module 201 is further configured to: Set the set of unassigned key-value pairs to the set of key-value pairs, indicating that all key-value pairs are initially unassigned; Setting the allocated load of each node to zero means that the node does not bear any data load in the initial state; A popular key-value threshold is set, which is determined based on the average or maximum value of the node's processing capacity, and is used to distinguish the data size of key values.

[0168] Optionally, the processing module 201 is further configured to: Iterate through each key in the key-value set and compare the amount of data for each key with the popular key-value threshold; Keys with a data volume greater than the aforementioned popular key value threshold are included in the popular key value set; Key values ​​whose data volume is not greater than the popular key value threshold are classified into the non-popular key value set.

[0169] Optionally, the processing module 201 is further configured to: Sort the key values ​​in the set of non-popular key values ​​in ascending order of data volume to generate a sorted sequence of non-popular key values. Sort the nodes in the node set in ascending order of processing power to generate a sorted node sequence. Traverse the sorted sequence of nodes. For each node, select unassigned key values ​​from the sorted sequence of non-hot key values ​​and assign them to the node. At the same time, update the assigned load of the node until the assigned load of the node approaches or reaches its processing capacity.

[0170] Optionally, the processing module 201 is further configured to: Sort the key values ​​in the popular key value set in descending order of data volume to generate a sorted popular key value sequence; For each popular key value in the sorted popular key value sequence, calculate the number of nodes required to process that popular key value. The number of nodes required is determined based on the ratio of the amount of data in the popular key value to the processing capacity of the nodes. Sort the nodes in the node set in descending order of processing power to generate a sorted node sequence. Select available nodes from the sorted sequence of nodes, assign data to popular key-value pairs, and update the assigned load of the nodes.

[0171] Optionally, the processing module 201 is further configured to: Based on the results of aggregation and scattering operations, an allocation function is constructed. The allocation function defines the mapping from each key-value pair to a subset of nodes. The allocation function is used to guide data partitioning operations in the stream processing framework. The allocation scheme is distributed to stream processing tasks in a broadcast state to ensure that all tasks update the allocation information synchronously.

[0172] Optionally, the scheduling module 202 is further configured to: Implement a custom partitioner in the stream processing framework, which routes data to a specified node according to the allocation scheme; Configure a task affinity policy to prioritize scheduling task instances to physical nodes with matching processing capabilities; The scheduler plugin of the stream processing framework requests computing resources with specific tags to ensure the task execution environment.

[0173] Optionally, the processing module 201 is further configured to: Query the access statistics module in the content delivery network to obtain the number of requests using the client and domain name as keys; The access statistics module aggregates request logs in real time and outputs statistical results. The latency of the statistical results is lower than that of the streaming framework, ensuring the timeliness of the data information.

[0174] Optionally, the processing module 201 is further configured to: A counting operation is inserted after the source operation in the stream processing pipeline, and the counting operation uses a state backend to maintain a key-value count; The key-value count result is output at the end of the micro-batch by using a timer trigger mechanism; The key-value counting results are sent to the coordination service or shared storage for use by the scheduling algorithm.

[0175] Optionally, the processing module 201 is further configured to: A configuration file containing a mapping between node identifiers and processing capabilities is loaded when the stream processing application starts. The processing capacity is predefined based on node hardware specifications and benchmark test results; The configuration file is stored in text format and parsed through the configuration interface of the stream processing framework.

[0176] Optionally, the processing module 201 is further configured to: Node performance data is collected through a metrics system of the stream processing framework, including CPU utilization, memory usage, and backpressure metrics. Use external monitoring services to aggregate performance data and calculate the real-time processing capabilities of nodes; The real-time processing capability is fed back to the scheduling algorithm to dynamically adjust the allocation strategy.

[0177] This application embodiment realizes intelligent task scheduling in heterogeneous computing clusters, effectively balances the workload among nodes, reduces the variance of task completion time, improves cluster resource utilization, and reduces the number of small files generated through key-value aggregation, thereby improving the overall efficiency and stability of big data processing pipelines.

[0178] The descriptions of the apparatus embodiments above are similar to those of the method embodiments above, and have similar beneficial effects. In some embodiments, the functions or modules included in the apparatus provided in this application can be used to perform the methods described in the method embodiments above. For technical details not disclosed in the apparatus embodiments of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0179] If the technical solution of this application involves personal information, the product using this technical solution has clearly informed the user of the personal information processing rules and obtained the user's voluntary consent before processing the personal information. If the technical solution of this application involves sensitive personal information, the product using this technical solution has obtained the user's separate consent before processing the sensitive personal information, and also meets the requirement of "express consent". For example, at personal information collection devices such as cameras, clear and prominent signs are set up to inform users that they have entered the scope of personal information collection and that personal information will be collected. If an individual voluntarily enters the collection scope, it is deemed that they have agreed to the collection of their personal information; or on the personal information processing device, the personal information processing rules are clearly informed through signs / information, and authorization is obtained through pop-up information or by asking the individual to upload their personal information; wherein, the personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the types of personal information processed.

[0180] It should be noted that, in the embodiments of this application, if the above-described heterogeneous cluster-aware scheduling method for stream computing is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiments of this application, or the part that contributes to the related technology, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, mobile hard drives, read-only memory (ROM), magnetic disks, or optical disks. Thus, the embodiments of this application are not limited to any specific hardware, software, or firmware, or any combination of hardware, software, and firmware.

[0181] This application provides a computer device including a memory and a processor. The memory stores a computer program that can run on the processor. When the processor executes the program, it implements some or all of the steps in the above-described method.

[0182] This application provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements some or all of the steps in the above-described method. The computer-readable storage medium can be transient or non-transient.

[0183] This application provides a computer program including computer-readable code, wherein when the computer-readable code is executed in a computer device, a processor in the computer device performs some or all of the steps in the above-described method.

[0184] This application provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program. When the computer program is read and executed by a computer, it implements some or all of the steps in the above-described method. This computer program product can be implemented specifically through hardware, software, or a combination thereof. In some embodiments, the computer program product is specifically embodied as a computer storage medium; in other embodiments, the computer program product is specifically embodied as a software product, such as a software development kit (SDK), etc.

[0185] It should be noted that the descriptions of the various embodiments above tend to emphasize the differences between them, while their similarities or commonalities can be referred to interchangeably. The descriptions of the above embodiments of the device, storage medium, computer program, and computer program product are similar to the descriptions of the above method embodiments and have similar beneficial effects. For technical details not disclosed in the embodiments of the device, storage medium, computer program, and computer program product of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0186] It should be noted that, Figure 3 This is a schematic diagram of a hardware entity of a computer device in an embodiment of this application, such as... Figure 3 As shown, the hardware entity of the computer device 700 includes: one or more processors 701, a communication interface 702, and a memory 703, wherein: Processor 701 typically controls the overall operation of computer device 700.

[0187] Communication interface 702 enables computer devices to communicate with other terminals or servers over a network.

[0188] The memory 703 is configured to store instructions and applications executable by the processor 701, and can also cache data to be processed or already processed (e.g., image data, audio data, voice communication data, and video communication data) in the processor 701 and various modules in the computer device 700. It can be implemented using flash memory or random access memory (RAM). Data transfer between the processor 701, the communication interface 702, and the memory 703 can be performed via bus 704. Only one processor is shown in the figure; each processor 100 includes one or more cores.

[0189] It should be noted that the computer device may include multiple processors 701, and each processor 701 can interact with each other through aggregated communication methods such as all-to-all, all-gather, or all-reduce. The processors 701 may be central processing units (CPUs), graphics processing units (GPUs), embedded neural network processing units (NPUs), tensor processing units (TPUs), data processing units (DPUs), accelerated processing units (APUs), floating-point processing units (FPUs), or application-specific integrated circuits (ASICs). The processors may also be single-core or multi-core processors. The processor may consist of a CPU and hardware chips. The hardware chips may be ASICs, PLDs, or combinations thereof. The PLDs may be complex programmable logic devices (CPLDs), FPGAs, generic array logic (GALs), or any combination thereof. The processor can also be implemented using logic devices with built-in processing logic, such as FPGAs or digital signal processors (DSPs).

[0190] The communication interface 702 can be a wired interface or a wireless interface, used to communicate with other modules or devices. The wired interface can be an Ethernet interface, a local interconnect network (LIN), etc., and the wireless interface can be a cellular network interface or a wireless LAN interface, etc.

[0191] Memory 703 can be non-volatile memory, such as read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Memory 703 can also be volatile memory, which can be random access memory (RAM) used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synclink dynamic random access memory (SLDRAM), and direct rambus RAM (DRRAM), direct rambus DRAM (DRDRAM), and rambus DRAM.

[0192] The 704 bus can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus can be divided into address bus, data bus, control bus, etc.

[0193] It should be understood that the phrase "one embodiment" or "an embodiment" throughout the specification means that a specific feature, structure, or characteristic related to the embodiment is included in at least one embodiment of this application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. It should be understood that in the various embodiments of this application, the sequence numbers of the above steps / processes do not imply a sequential order of execution; the execution order of each step / process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application. The sequence numbers of the above embodiments of this application are merely descriptive and do not represent the superiority or inferiority of the embodiments.

[0194] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0195] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection between devices or units can be electrical, mechanical, or other forms.

[0196] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units. They may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.

[0197] In addition, each functional unit in the various embodiments of this application can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0198] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the above method embodiments. The aforementioned storage medium includes various media that can store program code, such as mobile storage devices, read-only memory (ROM), magnetic disks, or optical disks.

[0199] Alternatively, if the integrated units described above are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, or the part that contributes to related technologies, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROM, magnetic disks, or optical disks.

[0200] The above description is merely an embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.< / key>

Claims

1. A stream computing heterogeneous cluster-aware scheduling method, characterized in that, The method includes: Collect the data volume of each key value in the key-value set and the processing capacity of each node in the node set, where the key-value set includes multiple keys and the node set includes multiple nodes; Based on the data volume and the processing capacity, the intermediate state is initialized, including setting the unassigned key-value set to the key-value set, setting the assigned load of each node to zero, and setting the popular key-value threshold. Based on the popular key value threshold and the amount of data for each key value, a popular key value set and an unpopular key value set are identified, wherein the popular key value set includes key values ​​with a data amount greater than the popular key value threshold, and the unpopular key value set includes key values ​​with a data amount not greater than the popular key value threshold; Aggregate the key values ​​in the set of non-popular key values, including sorting the key values ​​in the set of non-popular key values ​​in ascending order of data volume, and preferentially allocating the sorted key values ​​to nodes with lower processing capabilities until the allocated load of each node does not exceed its processing capability. Dispersing the key values ​​in the set of popular key values ​​includes sorting the key values ​​in the set of popular key values ​​in descending order of data volume, calculating the required number of nodes for each popular key value, and distributing the data of the popular key value to multiple nodes, wherein the nodes are selected in descending order of processing capacity; An allocation scheme is generated based on the aggregation and scattering results, and task scheduling is performed based on the allocation scheme.

2. The method according to claim 1, characterized in that, The steps of collecting the data volume of each key value in the key-value set and the processing capacity of each node in the node set include: The data volume for each key-value pair can be obtained from an external statistics system that operates independently of the stream processing framework and provides request statistics based on the key-value pair. Alternatively, the data volume for each key-value pair can be obtained from an internal data source of the stream processing framework, and the count status of each key-value pair can be maintained by inserting aggregation operations into the stream processing pipeline. The processing capacity of each node can be obtained from static configuration information, which is pre-set based on the stress test results when the node is put on the rack, or the processing capacity of each node can be obtained from a dynamic monitoring system, which collects the real-time performance indicators of the nodes and evaluates their processing capacity.

3. The method according to claim 1, characterized in that, The step of initializing the intermediate state based on the data volume and the processing capability includes: Set the set of unassigned key-value pairs to the set of key-value pairs, indicating that all key-value pairs are initially unassigned; Setting the allocated load of each node to zero means that the node does not bear any data load in the initial state; A popular key-value threshold is set, which is determined based on the average or maximum value of the node's processing capacity, and is used to distinguish the data size of key values.

4. The method according to claim 1, characterized in that, The step of identifying popular key-value sets and non-popular key-value sets based on the popular key-value threshold and the data volume of each key-value includes: Iterate through each key in the key-value set and compare the amount of data for each key with the popular key-value threshold; Keys with a data volume greater than the aforementioned popular key value threshold are included in the popular key value set; Key values ​​whose data volume is not greater than the popular key value threshold are classified into the non-popular key value set.

5. The method according to claim 1, characterized in that, The step of aggregating the key values ​​in the set of non-popular key values ​​includes: Sort the key values ​​in the set of non-popular key values ​​in ascending order of data volume to generate a sorted sequence of non-popular key values. Sort the nodes in the node set in ascending order of processing power to generate a sorted node sequence. Traverse the sorted sequence of nodes. For each node, select unassigned key values ​​from the sorted sequence of non-hot key values ​​and assign them to the node. At the same time, update the assigned load of the node until the assigned load of the node approaches or reaches its processing capacity.

6. The method according to claim 1, characterized in that, The step of breaking down the key values ​​in the popular key-value set includes: Sort the key values ​​in the popular key value set in descending order of data volume to generate a sorted popular key value sequence; For each popular key value in the sorted popular key value sequence, calculate the number of nodes required to process that popular key value. The number of nodes required is determined based on the ratio of the amount of data in the popular key value to the processing capacity of the nodes. Sort the nodes in the node set in descending order of processing power to generate a sorted node sequence. Select available nodes from the sorted sequence of nodes, assign data to popular key-value pairs, and update the assigned load of the nodes.

7. The method according to claim 1, characterized in that, The step of generating an allocation scheme based on the results of aggregation and dispersion includes: Based on the results of aggregation and scattering operations, an allocation function is constructed. The allocation function defines the mapping from each key-value pair to a subset of nodes. The allocation function is used to guide data partitioning operations in the stream processing framework. The allocation scheme is distributed to stream processing tasks in a broadcast state to ensure that all tasks update the allocation information synchronously.

8. The method according to claim 1, characterized in that, The step of performing task scheduling based on the allocation scheme includes: Implement a custom partitioner in the stream processing framework, which routes data to a specified node according to the allocation scheme; Configure a task affinity policy to prioritize scheduling task instances to physical nodes with matching processing capabilities; The scheduler plugin of the stream processing framework requests computing resources with specific tags to ensure the task execution environment.

9. The method according to claim 2, characterized in that, The step of obtaining the amount of data for each key value from an external statistical system includes: Query the access statistics module in the content delivery network to obtain the number of requests using the client and domain name as keys; The access statistics module aggregates request logs in real time and outputs statistical results. The latency of the statistical results is lower than that of the streaming framework, ensuring the timeliness of the data information.

10. The method according to claim 2, characterized in that, The step of obtaining the amount of data for each key-value pair from the internal data source of the stream processing framework includes: A counting operation is inserted after the source operation in the stream processing pipeline, and the counting operation uses a state backend to maintain a key-value count; The key-value count result is output at the end of the micro-batch by using a timer trigger mechanism; The key-value counting results are sent to the coordination service or shared storage for use by the scheduling algorithm.

11. The method according to claim 2, characterized in that, The step of obtaining the processing capacity of each node from static configuration information includes: A configuration file containing a mapping between node identifiers and processing capabilities is loaded when the stream processing application starts. The processing capacity is predefined based on node hardware specifications and benchmark test results; The configuration file is stored in text format and parsed through the configuration interface of the stream processing framework.

12. The method according to claim 2, characterized in that, The step of obtaining the processing capacity of each node from the dynamic monitoring system includes: Node performance data is collected through a metrics system of the stream processing framework, including CPU utilization, memory usage, and backpressure metrics. Use external monitoring services to aggregate performance data and calculate the real-time processing capabilities of nodes; The real-time processing capability is fed back to the scheduling algorithm to dynamically adjust the allocation strategy.

13. A stream computing heterogeneous cluster sensing and scheduling device, characterized in that, Applied to a client, the device includes: The processing module is used to collect the data volume of each key value in the key-value set and the processing capacity of each node in the node set, wherein the key-value set includes multiple key values ​​and the node set includes multiple nodes; Based on the data volume and the processing capacity, the intermediate state is initialized, including setting the unassigned key-value set to the key-value set, setting the assigned load of each node to zero, and setting the popular key-value threshold. Based on the popular key value threshold and the amount of data for each key value, a popular key value set and an unpopular key value set are identified, wherein the popular key value set includes key values ​​with a data amount greater than the popular key value threshold, and the unpopular key value set includes key values ​​with a data amount not greater than the popular key value threshold; Aggregate the key values ​​in the set of non-popular key values, including sorting the key values ​​in the set of non-popular key values ​​in ascending order of data volume, and preferentially allocating the sorted key values ​​to nodes with lower processing capabilities until the allocated load of each node does not exceed its processing capability. Dispersing the key values ​​in the set of popular key values ​​includes sorting the key values ​​in the set of popular key values ​​in descending order of data volume, calculating the required number of nodes for each popular key value, and distributing the data of the popular key value to multiple nodes, wherein the nodes are selected in descending order of processing capacity; The scheduling module is used to generate an allocation scheme based on the aggregation and scattering results, and to execute task scheduling based on the allocation scheme.

14. A computer device comprising a memory and a processor, the memory storing a computer program executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the stream computing heterogeneous cluster-aware scheduling method according to any one of claims 1 to 12.

15. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the steps of the stream computing heterogeneous cluster-aware scheduling method according to any one of claims 1 to 12.

Images