An adaptive computing power segmentation scheduling method for a large model inference task

Abstract

The present application relates to the field of data processing, and particularly relates to a self-adaptive computing power segmentation scheduling method for large model inference tasks, which comprises the following steps: synchronously collecting features of multiple to-be-inferred tasks and computing power cluster equipment common dimensions; clustering the tasks, setting multiple batch size schemes and dividing the task batches according to the clustering; for each scheme, matching the tasks with the equipment in batch order, calculating the matching degree of the batches, and determining the weight by the ratio of the feature weight to the equipment task same dimension feature; after the matching is completed, calculating the balance degree of the equipment computing power and memory utilization, combining the matching degree to obtain a scheme evaluation index; comparing the scheme evaluation indexes of all processing modes, and selecting the highest one and its matching result as the final scheme. The present application can better meet the task requirements and reduce the processing time of the tasks on the equipment by comprehensively considering multiple dimensional features and multiple processing modes, thereby reducing the service delay.

Description

Technical Field

[0001] This invention relates to the field of data processing. More specifically, this invention relates to an adaptive computational power partitioning and scheduling method for large model inference tasks. Background Technology

[0002] As large-scale model inference needs become more prevalent in dynamic scenarios such as cloud and edge computing, the rapid expansion of model size has caused the computation and memory requirements of a single inference to far exceed the physical limits of a single accelerator (such as a GPU). Therefore, it is necessary to rely on computing power partitioning techniques such as tensor parallelism, pipelined parallelism, and dynamic batch processing to distribute the model across multiple devices for collaborative computation.

[0003] Currently, partitioning strategies such as tensor parallelism and pipelined parallelism are typically statically configured during deployment, which cannot adapt to real-time changes in user request tasks and fluctuating hardware resources such as computing power, memory, and bandwidth. For example, they may cause idle computing power during periods of low demand, while during peak periods, they may cause latency due to insufficient resources; and they lack flexibility for different network conditions and heterogeneous device clusters.

[0004] Therefore, existing technologies cannot achieve "on-demand" and "precise matching" of underlying computing resources for inference tasks. How to design a mechanism that can perceive task requirements and resource status in real time and dynamically and adaptively select the optimal computing power allocation strategy and scheduling scheme has become a key technical challenge for improving the efficiency of large model inference, reducing operating costs, and ensuring service reliability. This is also the core problem that this invention aims to solve. Summary of the Invention

[0005] To address the technical problems of low resource utilization efficiency and service latency caused by the inability of static partitioning strategies to adapt to dynamically changing inference loads and resource states, this invention provides the following technical solution.

[0006] An adaptive computational power partitioning and scheduling method for large-scale model inference tasks includes: Simultaneously collect features of common dimensions from multiple inference tasks and various devices in the computing power cluster, including accuracy requirements, memory requirements, computing power requirements, and bandwidth requirements; Based on the task characteristics, cluster analysis is performed on the reasoning task. According to the clustering results, multiple batch size schemes are set. For each scheme, the task is divided into multiple batches. For each scheme, tasks and devices are matched in batch order to calculate the maximum sum of matching weights between tasks and devices under preset processing methods, which is the goal of completing the matching and obtains the matching degree of that batch. The calculation of the matching weights includes calculating the weight of each feature under preset processing methods and the ratio of the same dimension feature between any device and any task. After all batches are matched, the computing power and memory utilization of each device are counted to calculate the device balance under the scheme. Combining the matching degree of each batch with the device balance under the scheme, the evaluation index of matching according to the scheme is calculated. Compare the evaluation indicators corresponding to all schemes under all processing methods, and select the scheme with the highest evaluation indicator and its matching result as the final matching scheme.

[0007] Preferably, the clustering uses the k-means algorithm.

[0008] Preferably, the process of obtaining the matching weights includes: Based on any processing method, for each dimension of features, calculate the ratio of the feature value corresponding to any device to the feature value corresponding to any task. If the ratio is greater than 1, take 1 as the matching score of the feature in that dimension; if the ratio is less than 1, take the ratio itself as the matching score of the feature in that dimension. The calculated matching score is multiplied by the feature weight of the corresponding dimension to obtain the weighted matching score. The weighted matching scores of all dimensions are summed to obtain the matching weight between any device and any task under this processing method.

[0009] Preferably, the process of obtaining the matching weights includes: Based on any processing method, for each dimension of features, calculate the ratio of the feature value corresponding to any device to the feature value corresponding to any task, and multiply the ratio by the correction factor of the corresponding feature to obtain the correction ratio. If the correction ratio is greater than 1, then 1 is taken as the matching score of the feature in that dimension; if the correction ratio is less than 1, then the correction ratio itself is taken as the matching score of the feature in that dimension. The calculated matching score is multiplied by the feature weight of the corresponding dimension to obtain the weighted matching score. The weighted matching scores of all dimensions are summed to obtain the matching weight between any device and any task under this processing method.

[0010] Preferably, the step of matching tasks with devices in batches further includes: after the current batch is matched, updating the feature values ​​of each feature of the device, and performing the matching of the next batch based on the updated features.

[0011] Preferably, the equalization of the device includes: The computing power balance and memory balance are calculated based on the computing power and memory utilization of each device. The computing power balance and memory balance are added together and the average is taken to obtain the device balance. The calculation methods for computing power balance and memory balance are the same.

[0012] Preferably, the process of obtaining the computing power balance includes: Calculate the absolute value of the difference in computing power utilization between any two devices; calculate the mean of the computing power utilization of all computing power devices and calculate a normalization factor based on the total number of pairwise combinations of devices; calculate the ratio of the absolute value of the difference in computing power utilization between any two devices to the normalization factor; and iterate through all devices to obtain the sum of the ratios of the absolute value of the difference in computing power utilization between all pairs of devices to the normalization factor. Subtract the sum of these ratios from 1 to obtain the computing power balance of the device.

[0013] Preferably, the process of obtaining the computing power balance includes: Calculate the mean and standard deviation of the computing power utilization rate of all computing power devices. Use the ratio of the standard deviation to the mean of the calculated computing power utilization rate as the coefficient of variation. Subtract the coefficient of variation from 1 as the computing power balance of the devices.

[0014] Preferably, the process of obtaining the evaluation indicators includes: Obtain the normalized value of the sum of matching degrees of all batches under each batch size scheme, and then multiply it by the equipment balance degree under that batch size scheme. The result of the product is used as the evaluation index of matching under that batch size scheme.

[0015] Preferably, the final matching scheme specifically includes: The corresponding processing method, batch size, and the device that carries each inference task.

[0016] The present invention has the following beneficial effects: 1. Simultaneously collect multi-dimensional features of the tasks to be inferred and the computing cluster devices, including accuracy, memory, computing power, and bandwidth requirements, comprehensively considering the matching factors between tasks and devices. Based on these features, perform cluster analysis and set multiple batch size schemes to match tasks with devices in batch order, aiming to maximize the sum of matching weights. This allows tasks to be more rationally allocated to devices, avoiding resource idleness or over-occupancy, and effectively improving the overall resource utilization efficiency of the computing cluster.

[0017] 2. By comprehensively considering multi-dimensional features and various processing methods, matching weights, matching degrees, device balance, and evaluation metrics are calculated to select the optimal solution for task and device matching. This refined matching method can better meet task requirements, reduce task processing time on devices, and thus reduce service latency. Attached Figure Description

[0018] Figure 1 This is a flowchart of steps S1-S5 in an adaptive computing power partitioning and scheduling method for a large model inference task according to an embodiment of the present invention. Detailed Implementation

[0019] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are some embodiments of the present invention, but not all embodiments.

[0020] This invention is applicable to various scenarios requiring efficient and flexible scheduling of large model inference tasks across heterogeneous and dynamically changing computing power clusters. Based on real-time task load and resource status, it can automatically select the optimal computing power partitioning and task allocation strategy across various heterogeneous computing power scenarios, such as cloud data centers, edge computing nodes, and hybrid cloud environments. This enables refined and adaptive scheduling of cluster resources, significantly improving the overall utilization efficiency of computing resources and ensuring the performance and stability of inference services.

[0021] Reference Figure 1 An adaptive computing power partitioning and scheduling method for large model inference tasks includes steps S1-S5, as detailed below: S1: Collect task characteristics of multiple inference tasks and device characteristics of each device in the computing power cluster.

[0022] By acquiring real-time feature information of the task to be inferred and the resource status of computing devices, a data foundation is provided for subsequent adaptive partitioning and scheduling decisions. The specific implementation process is as follows: For each large model task to be inferred, its resource requirements are obtained by parsing its model structure file or preset configuration file: accuracy requirements, memory requirements, computing power requirements, and bandwidth requirements.

[0023] Accordingly, the real-time characteristics of each device in the computing power cluster are obtained synchronously: available precision, available memory, available computing power, and available bandwidth.

[0024] It should be noted that the features of the above-mentioned tasks and devices correspond one-to-one according to their properties, so the features given above can be called features of common dimensions.

[0025] For example, based on common empirical values, hardware specifications, and typical workload characteristics in the field of large model inference, assume that an AI application platform simultaneously receives four different types of large model inference tasks, and the available computing power cluster contains four GPUs.

[0026] The characteristics of the above four tasks are as follows: Task A (Large Language Model Chat): Requires FP16 precision, high memory requirements (20GB), high computing power requirements, suitable for tensor parallel processing, and high bandwidth requirements (e.g., 40Gbps).

[0027] Task B (Wensheng Graph Model): Requires FP16 precision, extremely high memory requirements (32GB), high computing power requirements, suitable for pipelined parallel processing, and moderate bandwidth requirements (e.g., 25Gbps).

[0028] Task C (Speech Recognition Model): Requires INT8 precision, low memory requirements (8GB), medium computing power requirements, suitable for dynamic batch processing, and low bandwidth requirements (e.g., 5Gbps).

[0029] Task D (Visual Understanding Model): Requires FP16 accuracy, medium memory requirement (15GB), medium computing power requirement, may perform simple model parallelism, and medium bandwidth requirement (e.g., 15Gbps).

[0030] Accordingly, the real-time characteristics of the four GPU devices are as follows: Device 1: Supports FP16, 24GB of available memory, 75TFLOPS of available computing power, and 50Gbps of available bandwidth.

[0031] Device 2: Supports FP16, 22GB of available memory, 78TFLOPS of available computing power, and 50Gbps of available bandwidth.

[0032] Device 3: Supports INT8 / FP16, 10GB of available memory, 45TFLOPS of available computing power, and 20Gbps of available bandwidth.

[0033] Device 4: Supports FP16, 30GB of available memory, 70TFLOPS of available computing power, and 30Gbps of available bandwidth.

[0034] S2: Perform cluster analysis on the reasoning task based on task characteristics. Based on the clustering results, set multiple batch size schemes. For each scheme, divide the task into multiple batches.

[0035] Currently, when matching tasks and devices for inference, all tasks and devices are considered simultaneously, resulting in high computational complexity. This can easily lead to uneven resource allocation due to diverse task requirements, and all tasks must be rematched if the process fails.

[0036] To reduce computational complexity and speed up matching, the approach has shifted from overall matching to batch matching. This allows for multiple adjustments, resulting in more balanced resource utilization and better fault tolerance, as a failure in one batch does not affect other batches. However, batch matching may lead to some batches having high demands for a single feature, consuming excessive computing power for that single feature and impacting the matching of subsequent batches and the overall performance.

[0037] Therefore, based on the characteristics of the real-time tasks to be reasoned, all tasks are clustered, for example using the k-means clustering algorithm, to divide these tasks into different clusters. For instance, 100 real-time tasks to be reasoned are automatically divided into 3 clusters: cluster 1, cluster 2, and cluster 3, corresponding to heavy tasks, medium tasks, and light tasks, respectively.

[0038] After the clustering is completed, different batch size schemes are set, such as matching according to batches of 10, 20, or 30. Furthermore, when deciding which tasks to include in a batch, tasks are extracted from each cluster proportionally. The more tasks a cluster contains, the more tasks each batch selects from that cluster, thus ensuring a balance in the demand for different features across batches.

[0039] For example, assuming the batch size is 10, cluster 1 has 30 tasks to be inferred, so this batch will extract 3 tasks from cluster 1. Cluster 2 has 50 tasks, so 5 tasks will be extracted from cluster 2. Cluster 3 has 20 tasks, so 2 tasks will be extracted from cluster 3.

[0040] Then, based on the above operations, determine the number of tasks selected from each batch and each cluster under different batch size schemes.

[0041] In summary, by using clustering and proportional sampling, we ensure that each batch contains tasks with varying resource requirements, and each batch of tasks consumes different types and amounts of memory, computing power, and bandwidth simultaneously. This means that a certain resource will not be completely exhausted in the early stages, preserving diverse resource options for subsequent batches.

[0042] In addition, after the tasks for each batch were selected, the batches of each size scheme were sequentially numbered to facilitate subsequent analysis.

[0043] S3: For each scheme, tasks and devices are matched in batch order to calculate the goal of maximizing the sum of matching weights between tasks and devices under multiple preset processing methods, and obtain the matching degree of the batch; after all batches are matched, the computing power and memory utilization of each device are counted to calculate the device balance under the scheme.

[0044] Different parallel processing methods (tensor parallelism, pipelined parallelism, and dynamic batch processing) have different requirements for device features. Tensor parallelism requires synchronization of gradients and parameters between devices, and bandwidth bottlenecks directly limit scalability, making device bandwidth more important. Pipeline parallelism requires storing complete model layers at each stage, and a large number of intermediate activation values ​​need to be cached, making device memory more important. Determining the feature weights for different processing methods allows for a better match between the task and the device to the actual needs of the processing method, improving processing efficiency and performance. For example, when processing large neural networks with tensor parallelism, choosing a device with low bandwidth will lead to delays in gradient and parameter synchronization, affecting training speed and convergence; however, prioritizing devices with high bandwidth based on the feature weights of tensor parallelism can avoid this problem.

[0045] First, the above processing methods include: tensor parallelism, pipelined parallelism, and dynamic batch processing.

[0046] Secondly, the importance of each feature of the device varies under different processing methods (tensor parallelism, pipelined parallelism, and dynamic batch processing). The weight values ​​of each feature—precision, memory, bandwidth, and computing power—are given for each of the three processing methods.

[0047] For example, the feature weights under tensor parallelism are: Precision weights: =0.18, Memory weight: =0.22, Bandwidth weight: =0.35, Computing power weight: =0.25.

[0048] The feature weights under pipeline parallelism are: Precision weight: =0.12, Memory weight: =0.38 bandwidth weight: =0.25, Computing power weight: =0.25.

[0049] The feature weights under dynamic batch processing are: Precision weight: =0.23, Memory weight: =0.42, Bandwidth weight: =0.10, Computing power weight: =0.25.

[0050] Next, after obtaining the weights of all features under different processing methods, since the weights themselves only represent "importance" and cannot directly reflect the "actual matching situation".

[0051] Therefore, it is also necessary to conduct a quantifiable compatibility assessment of specific equipment and task pairs, and further calculate the matching weights between tasks and equipment.

[0052] In one embodiment, taking the batch size 10 exemplarily given in S2 above as an example, starting from the first batch, for one of the processing methods, for each dimension of features, the ratio of the feature value corresponding to any device to the feature value corresponding to any task is calculated. If the ratio is greater than 1, then 1 is taken as the matching score of the feature of that dimension, indicating that the device capability fully meets the task requirements; if the ratio is less than 1, then the ratio itself is taken as the matching score of the feature of that dimension, indicating that the device capability partially meets the task requirements.

[0053] Furthermore, the matching score calculated above is multiplied by the feature weight of the corresponding dimension to obtain the weighted matching score. The weighted matching scores of all dimensions are summed to obtain the matching weight between any device and any task under this processing method.

[0054] This allows us to obtain the matching weights between each device and each task in the first batch. These matching weights can then be used to construct a weight matrix, with rows representing devices and columns representing tasks.

[0055] Following the one-to-one matching principle, find the maximum sum of matching weights for all pairs in the weight matrix, and use this maximum sum of matching weights as the matching degree of the batch. At the same time, you can also get the specific matching scheme, that is, which devices are paired with which tasks in the batch.

[0056] It should be noted that precision features need to be encoded first. For example, FP64, FP32, FP16, INT8, and INT4 are encoded as 1, 0.8, 0.6, 0.4, and 0.2, respectively.

[0057] In another embodiment, in addition to considering the basic dimensional characteristics of devices and tasks, it is also necessary to consider the device topology and memory fragmentation to more accurately reflect the actual available resource capabilities of the device and avoid matching deviations caused by memory fragmentation or inefficient network topology.

[0058] Taking the batch size of 10 given in S2 above as an example, starting from the first batch, for one of the processing methods, for each dimension of features, calculate the ratio of the feature value corresponding to any device to the feature value corresponding to any task, and then multiply the ratio by the correction factor of the corresponding feature to obtain the correction ratio.

[0059] For memory characteristics, the correction factor is the fragmentation factor, calculated as the ratio of the largest available contiguous memory block to the total free memory, reflecting the impact of memory fragmentation on available memory. The value ranges from 0 to 1, with higher values ​​indicating lower fragmentation. For bandwidth characteristics, the correction factor is bandwidth utilization efficiency, which depends on the device's topology (fully connected, tree, or bus). Empirical values ​​are set as follows: 0.9 for fully connected, 0.7 for tree, and 0.5 for bus. For computing power and precision characteristics, the corresponding correction factors are all set to 1, indicating that these two characteristics are unaffected by the correction.

[0060] If the correction ratio is greater than 1, then 1 is taken as the matching score of the feature in that dimension, indicating that the device capability fully meets the task requirements; if the correction ratio is less than 1, then the correction ratio itself is taken as the matching score of the feature in that dimension, indicating that the device capability partially meets the task requirements.

[0061] Furthermore, the matching score calculated above is multiplied by the feature weight of the corresponding dimension to obtain the weighted matching score. The weighted matching scores of all dimensions are summed to obtain the matching weight between any device and any task under this processing method.

[0062] Then, the first batch of matching is completed according to the above operations, and the matching degree of this batch is obtained.

[0063] It should be noted that after completing the first batch of matching, the feature values ​​of each feature of the device need to be updated, and the next batch of matching is based on the updated features. In other words, subsequent matching only considers the remaining available resources of the device.

[0064] Finally, based on the above operations, matching can be completed for batches of size 10, and the matching degree of each batch can be obtained. Similarly, matching for other batch sizes and the matching degree of each batch can be completed.

[0065] After each task is assigned, it is necessary to further evaluate whether the resource usage of each device is balanced in order to avoid situations where some devices are overloaded while others are idle.

[0066] It should be noted that task execution typically requires both computing power and memory, which directly determine task execution efficiency. Therefore, it is necessary to further evaluate whether the load on computing power and memory is evenly distributed across all computing devices to calculate the balance of all computing devices.

[0067] In one embodiment, the computing power utilization rate of each device is statistically analyzed from the results of the matching process described above. The absolute value of the difference in computing power utilization rate between any two devices is calculated. In addition, considering that the above is a dual calculation, the mean of the computing power utilization rate of all computing devices is calculated, and a normalization factor is calculated by combining the total number of pairwise combinations of devices. Then, the ratio of the absolute value of the difference in computing power utilization rate between any two devices to the normalization factor is calculated. All devices are traversed to obtain the sum of the ratios of the absolute value of the difference in computing power utilization rate between all pairs of devices to the normalization factor.

[0068] The sum of these ratios is further subtracted from 1 to obtain the computing power balance of the equipment, thereby achieving positive transformation of the indicator and intuitively converting the degree of difference into the degree of balance.

[0069] The normalization factor mentioned above is expressed by the following relation: In the formula, As the normalization factor, The total number of computing devices. This represents the average computing power utilization rate of computing devices.

[0070] in, This is a correction factor to ensure that the balance is 1 when the computing power utilization of all devices is exactly the same; This indicates the total number of times the devices are combined in pairs. As a normalized number of devices, the balance index can be compared across systems of different sizes.

[0071] In another embodiment, the mean and standard deviation of the computing power utilization rate of all computing power devices are calculated. The ratio of the calculated standard deviation to the mean is used as the coefficient of variation, and the difference between 1 and the coefficient of variation is used as the computing power balance of the devices. By dividing the standard deviation by the mean, the influence of dimensions and the magnitude of the mean is eliminated, directly reflecting the balance ratio.

[0072] Then, the memory balance is calculated similarly based on the above calculation of computing power balance.

[0073] The average of the computing power balance and memory balance is calculated to obtain the computing power balance of the devices.

[0074] Finally, the balance of computing power across all batch size schemes can be obtained.

[0075] S4: Combining the matching degree of each batch with the balance of equipment under this scheme, the evaluation index for matching according to this scheme is calculated.

[0076] The above S3 calculations yield the balance of computing power under all batch size schemes and the matching degree of each batch under each scheme.

[0077] The matching degree mentioned above reflects the degree of matching between the task and the device, that is, the fit between the task requirements (such as computing resources, memory, etc.) and the device capabilities. The higher the matching degree, the higher the task execution efficiency is usually (for example, the task is more likely to be assigned to the appropriate device, reducing waiting time or resource waste).

[0078] The aforementioned load balance measures the load balancing among computing devices. A higher load balance indicates more similar utilization rates among devices, avoiding the problem of some devices being overloaded while others are idle, thereby improving overall system efficiency and resource utilization.

[0079] However, if there is only high matching degree but low balance degree (i.e., the task is well matched but the device load is unbalanced), it is easy to cause system bottlenecks, resource waste or delays; if there is only high balance degree but low matching degree (i.e., the device load is balanced but the task allocation is unreasonable), it is easy to cause the task to not be assigned to the optimal device, resulting in a decrease in execution efficiency.

[0080] Therefore, by combining the above-mentioned balance and matching degree, a comprehensive and fair evaluation standard can be constructed to ensure that the selected batch size can achieve efficient task matching and maintain a balanced equipment load, thereby improving the overall utilization efficiency of computing resources.

[0081] In one embodiment, the normalized value of the sum of matching degrees for all batches under each batch size scheme is obtained (normalization makes the matching degrees under different batch size schemes comparable), and then multiplied by the equipment balance degree under that batch size scheme. The product is used as the evaluation index of matching under that batch size scheme. Similarly, the evaluation index of matching under all batch size schemes is obtained.

[0082] It is important to emphasize that all the above operations are analyzed based on a single processing method. Therefore, the same evaluation index can be obtained for all batch size schemes under other processing methods.

[0083] In another embodiment, considering the relative change in balance before and after matching, the difference between the balance after matching and the balance before matching is calculated for each batch size scheme. This difference is then divided by the balance of the devices under that batch size scheme to obtain the relative rate of change in balance. The relative rate of change in balance is added to the balance, and then multiplied by the normalized value of the sum of the matching degrees of all batches under that batch size scheme. The product is used as the evaluation index for matching under that batch size scheme. This operation makes the evaluation index more comprehensive, focusing not only on the final result of matching but also on the optimization effect of the matching process on system balance, thereby helping to select batch sizes and matching schemes that can comprehensively improve system performance.

[0084] S5: Compare the evaluation indicators of all solutions under all processing methods, and select the solution with the highest evaluation indicator and its matching result as the final matching solution.

[0085] Through all the operations in S3 and S4 above, we finally obtain the evaluation index for matching all batch size schemes under all processing methods (tensor parallelism, pipelined parallelism, and dynamic batch processing), and then select the optimal parallel strategy combination, batch size, and task-device matching scheme corresponding to the maximum evaluation index.

[0086] More specifically, the matching scheme mentioned above specifies the parallel strategy to be used, the batch size to be adopted, and the specific computing devices to which each inference task is assigned for execution.

[0087] In summary, S1-S5, as a complete solution, can maximize the resource utilization and system stability of the entire computing cluster while ensuring task execution efficiency.

[0088] It should be noted that those skilled in the art can make various modifications and improvements without departing from the inventive concept, and these all fall within the scope of protection of this invention. Therefore, the scope of protection of this patent should be determined by the appended claims.

Claims

1. An adaptive computing power partitioning and scheduling method for large-scale model inference tasks, characterized in that, include: Simultaneously collect features of common dimensions from multiple inference tasks and various devices in the computing power cluster, including accuracy requirements, memory requirements, computing power requirements, and bandwidth requirements; Cluster analysis is performed on the reasoning tasks based on task characteristics. Based on the clustering results, multiple batch size schemes are set. Each batch size scheme includes the total number of reasoning tasks extracted in each batch and the number of clusters extracted. For each scheme, tasks and devices are matched in batch order to calculate the matching weight between tasks and devices under each preset processing method. The process of obtaining the matching weight includes: based on any processing method, for each dimension of features, calculating the ratio of the feature value corresponding to any device to the feature value corresponding to any task. If the ratio is greater than 1, then 1 is taken as the matching score of the feature of that dimension; if the ratio is less than 1, then the ratio itself is taken as the matching score of the feature of that dimension. The calculated matching score is multiplied by the feature weight of the corresponding dimension to obtain the weighted matching score. The weighted matching scores of all dimensions are summed to obtain the matching weight between any device and any task in each batch under this processing method. These matching weights are then constructed into a weight matrix, with rows representing devices and columns representing tasks. Following the one-to-one matching principle, the maximum sum of matching weights for all pairs in the weight matrix is ​​found, and this maximum sum of matching weights is taken as the matching degree of each batch. After all batches are matched, the computing power and memory utilization of each device are calculated to obtain the device balance under this scheme. Combining the matching degree of each batch with the device balance under this scheme, the evaluation index for matching according to this scheme is calculated. Compare the evaluation indicators corresponding to all schemes under all processing methods, and select the scheme with the highest evaluation indicator and its matching result as the final matching scheme.

2. The adaptive computing power partitioning and scheduling method for large model inference tasks according to claim 1, characterized in that, The clustering uses the k-means algorithm.

3. The adaptive computing power partitioning and scheduling method for large model inference tasks according to claim 1, characterized in that, The process of obtaining the matching weights includes: Based on any processing method, for each dimension of features, calculate the ratio of the feature value corresponding to any device to the feature value corresponding to any task, and multiply the ratio by the correction factor of the corresponding feature to obtain the correction ratio. If the correction ratio is greater than 1, then 1 is taken as the matching score of the feature in that dimension; if the correction ratio is less than 1, then the correction ratio itself is taken as the matching score of the feature in that dimension. The calculated matching score is multiplied by the feature weight of the corresponding dimension to obtain the weighted matching score. The weighted matching scores of all dimensions are summed to obtain the matching weight between any device and any task under this processing method.

4. The adaptive computing power partitioning and scheduling method for large model inference tasks according to claim 1, characterized in that, The process of matching tasks with devices in batches also includes: after the current batch is matched, updating the feature values ​​of each feature of the device, and performing the matching of the next batch based on the updated features.

5. The adaptive computing power partitioning and scheduling method for large model inference tasks according to claim 1, characterized in that, The balance of the device includes: The computing power balance and memory balance are calculated based on the computing power and memory utilization of each device. The computing power balance and memory balance are added together and the average is taken to obtain the device balance. The calculation methods for computing power balance and memory balance are the same.

6. The adaptive computing power partitioning and scheduling method for large model inference tasks according to claim 5, characterized in that, The process of obtaining the computing power balance includes: Calculate the absolute value of the difference in computing power utilization between any two devices; calculate the mean of the computing power utilization of all computing power devices and calculate a normalization factor based on the total number of pairwise combinations of devices; calculate the ratio of the absolute value of the difference in computing power utilization between any two devices to the normalization factor; and iterate through all devices to obtain the sum of the ratios of the absolute value of the difference in computing power utilization between all pairs of devices to the normalization factor. Subtract the sum of these ratios from 1 to obtain the computing power balance of the device.

7. The adaptive computing power partitioning and scheduling method for large model inference tasks according to claim 5, characterized in that, The process of obtaining the computing power balance includes: Calculate the mean and standard deviation of the computing power utilization rate of all computing power devices. Use the ratio of the standard deviation to the mean of the calculated computing power utilization rate as the coefficient of variation. Subtract the coefficient of variation from 1 as the computing power balance of the devices.

8. The adaptive computing power partitioning and scheduling method for large model inference tasks according to claim 1, characterized in that, The process of obtaining the evaluation indicators includes: Obtain the normalized value of the sum of matching degrees of all batches under each batch size scheme, and then multiply it by the equipment balance degree under that batch size scheme. The result of the product is used as the evaluation index of matching under that batch size scheme.

9. The adaptive computing power partitioning and scheduling method for large model inference tasks according to claim 1, characterized in that, The final matching scheme specifically includes: The corresponding processing method, batch size, and the device that carries each inference task.

Images