Power scheduling large model inference method and device for computing card cluster and medium

Abstract

The application discloses a kind of power scheduling large model inference methods, device and medium for power card cluster, belong to power card load balancing technical field, specifically as follows: the overall sequence length and peak video memory requirement of current inference request are obtained, and the available video memory capacity of each power card, inference task adaptation coefficient and comprehensive load score;According to overall sequence length and inference task adaptation coefficient, calculate sequence length matching degree, to generate pre-allocation weight;Exclude the power card corresponding to the available video memory capacity less than peak video memory requirement, and according to pre-allocation weight and comprehensive load score, the remaining power card is comprehensively evaluated to determine target power card;Current inference request is issued to target power card for inference, to generate inference result.Therefore, by implementing the present application, inference task can be allocated to hardware adaptation and the power card with optimal current load under the premise of meeting video memory constraints for execution, to improve the inference efficiency of power card cluster.

Description

Technical Field

[0001] This invention relates to the field of computing power card load balancing technology, and in particular to a large-scale power dispatching model inference method, device and medium for computing power card clusters. Background Technology

[0002] With the deepening of the construction of new power systems, business scenarios such as power grid operation analysis, load forecasting, fault diagnosis, and dispatch plan generation are becoming increasingly complex. Traditional decision-making methods based on mechanistic models and human experience are no longer sufficient to meet the requirements of real-time performance and accuracy. Artificial intelligence technologies, represented by large language models, offer a new technical path for the intelligent transformation of the power dispatching field due to their powerful semantic understanding and generation capabilities. By encoding multi-source heterogeneous information such as power grid operating status, historical load data, and equipment parameters into text sequences, large power dispatching models can achieve a deep understanding of complex power grid situations and directly generate high-value inference results such as load forecast values, dispatching operation instructions, and anomaly handling suggestions, providing interpretable decision support for dispatchers. However, power dispatching operations have significant domain characteristics: the sequence length of inference requests varies greatly, and the computational power and memory requirements of inference tasks fluctuate dynamically with changes in sequence length. Deploying large-scale power dispatch models based on computing card clusters has become the mainstream approach. However, computing cards generally exhibit significant heterogeneity: different models, and even cards of the same model, differ in peak computing power, memory capacity and bandwidth, and inter-card communication latency. This further increases the complexity of resource allocation during inference. Ensuring the accuracy of inference results while enabling efficient execution of large-scale model inference on heterogeneous computing card clusters with low-latency responses is a critical technical challenge that needs to be addressed in the current construction of AI infrastructure for power dispatch.

[0003] Existing inference deployment solutions mostly employ static strategies such as round-robin, random, or simple hashing to distribute requests. While some solutions can collect CPU or memory utilization data to adjust weights, they still suffer from the following drawbacks: First, they cannot predict the actual computational load and peak memory usage of requests, leading to long-sequence tasks being assigned to cards with insufficient memory, causing swapping in and out or even failure, while short-sequence tasks are assigned to cards with coarse computational granularity, resulting in wasted computing power, limited cluster throughput, and increased latency. Second, they cannot perceive the heterogeneous differences in computing power, memory bandwidth, and communication latency between computing cards and adapt to different length sequences accordingly, causing inference latency to be limited by the performance bottleneck nodes. These shortcomings are further amplified in high-concurrency scenarios, severely restricting the availability and reliability of online inference services. Summary of the Invention

[0004] This invention provides a method, apparatus, and medium for power scheduling large model inference for computing power card clusters. It can allocate inference tasks to computing power cards with hardware compatibility and optimal current load, while satisfying memory constraints, thereby improving the inference efficiency of computing power card clusters.

[0005] This invention provides a large-scale power dispatching model inference method for computing power card clusters, including: Obtain the overall sequence length and peak memory requirement of the current inference request, and obtain the available memory capacity, inference task adaptation coefficient, and overall load score of each computing card in the target computing card cluster; wherein, the inference task adaptation coefficient and the overall load score are pre-generated based on the static performance benchmark and hardware operating status of each computing card; Based on the overall sequence length and the adaptation coefficient of each inference task, the sequence length matching degree between the current inference request and each computing card is calculated, and the pre-allocated weight of each computing card is generated based on the sequence length matching degree. The computing cards with available video memory capacity less than the peak video memory requirement are screened out, and the remaining computing cards are comprehensively evaluated based on the pre-allocated weight and the comprehensive load score. The target computing card is determined based on the evaluation results. The current inference request is sent to the target computing card so that the target computing card can perform power dispatching large-scale model inference based on the current inference request and generate inference results.

[0006] This invention provides a quantitative data foundation for subsequent computing card matching by acquiring the overall sequence length and peak memory requirement of inference requests, as well as the available memory capacity of each computing card, pre-generated inference task adaptation coefficients, and comprehensive load scores. Simultaneously, by setting the inference task adaptation coefficients and comprehensive load scores to be generated based on the static performance benchmarks and hardware operating states of each computing card, these two parameters are generated based on the inherent hardware attributes and actual operating states of the computing cards, rather than being fixed values ​​set artificially. This provides physically grounded quantitative data for subsequent matching decisions. By combining the overall sequence length with the inference task adaptation coefficients of each computing card to calculate the sequence length matching degree and generate pre-allocated weights, the task sequence length characteristics can be quantitatively matched with the hardware adaptation preferences of the computing cards, avoiding mismatches between inference tasks and computing cards based on sequence length. By comprehensively evaluating and determining the target computing card among computing cards whose memory capacity meets peak memory requirements based on pre-allocated weights and overall load scores, this approach simultaneously satisfies the requirements of three dimensions: hard memory constraints, hardware adaptation preferences, and current load balancing. Among the adapted computing cards, the one with the best overall conditions is selected for inference. By sending inference requests to the target computing card to execute power dispatching model inference and generate inference results, scheduling decisions can be transformed into specific inference outputs, enabling online inference services for power dispatching models. Compared to existing technologies that fail to perceive heterogeneous differences in computing cards and the problem of blind matching due to dynamic task requirements, this application can allocate inference tasks to computing cards with hardware adaptation and optimal current load, while meeting memory constraints, thus improving the inference efficiency of computing card clusters.

[0007] Furthermore, the inference task adaptation coefficient and the overall load score are pre-generated based on the static performance benchmark and hardware operating status of each computing card, specifically: The static performance benchmark and hardware operation status data of each computing card are fused and processed to construct a dual-mode performance profile of each computing card; wherein, the parameters of the dual-mode performance profile include the current available computing power, the current available video memory bandwidth, the current available communication transmission power, and the current service queue pressure factor. The currently available computing power, the currently available video memory bandwidth, the currently available communication transmission capacity, and the current service queue pressure factor are weighted and aggregated to generate the inference task adaptation coefficient and comprehensive load score for each computing card.

[0008] This invention constructs a dual-mode performance profile by fusing the static performance benchmarks and hardware operating status data of each computing card, and generates inference task adaptation coefficients and comprehensive load scores by weighting and aggregating the parameters in the profile. This enables the adaptation coefficients and load scores to simultaneously reflect the inherent hardware capabilities and real-time operating status of the computing card, providing a quantitative basis for matching subsequent tasks with computing cards.

[0009] Further, the step of calculating the sequence length matching degree between the current inference request and each computing power card based on the overall sequence length and the adaptation coefficient of each inference task, and generating the pre-allocated weights of each computing power card based on the sequence length matching degree, includes: For any inference task adaptation coefficient, the overall sequence length and the inference task adaptation coefficient are substituted into the matching function for calculation to obtain the sequence length matching degree between the current inference request and the computing power card corresponding to the inference task adaptation coefficient. The initial pre-allocated weight of the computing power card is generated based on the sequence length matching degree, and the initial pre-allocated weight is corrected based on the current business queue pressure factor corresponding to the computing power card to obtain the pre-allocated weight of the computing power card.

[0010] This invention calculates the sequence length matching degree and generates pre-allocated weights by substituting the overall sequence length and inference task adaptation coefficient into the matching function, and then corrects the pre-allocated weights according to the business queue pressure factor. This enables the pre-allocated weights to reflect both hardware adaptation preferences and the current queue congestion, thus avoiding the allocation of tasks to computing cards that have accumulated too much power.

[0011] Further, the process of filtering out computing cards with available video memory capacity less than the peak video memory requirement, and comprehensively evaluating the remaining computing cards based on the pre-allocated weights and the overall load score, and determining the target computing card based on the evaluation results, includes: The computing cards with available video memory capacity less than the peak video memory requirement are filtered out to obtain a set of candidate computing cards. For any computing card in the candidate computing card set, the comprehensive load score of the computing card is weighted according to the pre-allocated weight of the computing card to generate the weighted load score of the computing card; The computing card with the smallest weighted load score in the candidate computing card set is selected as the target computing card.

[0012] This invention enables the matching of tasks and computing cards to simultaneously meet the requirements of three dimensions: hard constraints on video memory, hardware adaptation preferences, and current load balancing. By filtering out computing cards whose available video memory capacity does not meet the peak video memory requirements, and then weighting the comprehensive load score according to the pre-allocated weights, the computing card with the smallest weighted load score is selected as the target.

[0013] Further, the step of sending the current inference request to the target computing card, so that the target computing card performs power dispatching large-scale model inference based on the current inference request and generates inference results, includes: The current inference request is sent to the target computing card so that the target computing card allocates video memory storage space for the current inference request according to the peak video memory requirement of the current inference request. Load a hardware affinity operator library adapted to the hardware architecture characteristics of the target computing card, and perform power scheduling large model inference based on the video memory storage space and the hardware affinity operator library to generate inference results.

[0014] This invention, through sending inference requests to the target computing card and allocating video memory space thereon, loads a hardware affinity operator library adapted to the hardware architecture of the target computing card to perform inference, thereby enabling the inference execution process to be deeply adapted to the hardware characteristics, reducing video memory fragmentation and execution overhead, and improving inference execution efficiency.

[0015] Furthermore, obtaining the overall sequence length and peak memory requirement of the current inference request includes: Obtain the request attribute parameters of the current inference request; wherein, the request attribute parameters include input length, expected output length, and model type identifier; The overall sequence length is obtained by adding the input length and the expected output length. The requested attribute parameters are input into a pre-trained prediction model, so that the prediction model can predict the task volume based on the requested attribute parameters and output the peak memory requirement.

[0016] This invention, through obtaining the input length, expected output length, and model type identifier of the inference request, adds the input length and expected output length to obtain the overall sequence length, and inputs the request attribute parameters into the pre-trained prediction model to output peak memory requirements. This enables quantitative prediction of the computational resource consumption of the inference task, providing a quantitative basis for matching subsequent tasks with computing cards.

[0017] Furthermore, the method for obtaining the static performance benchmark and the hardware operating status data includes: The performance benchmark tests are performed on each of the aforementioned computing cards, and the static performance benchmarks of each computing card are determined based on the test results; wherein, the static performance benchmarks include floating-point operation capability per unit time, peak memory bandwidth, and architecture type information; The hardware operating status of each computing card is collected in real time; wherein, the hardware operating status includes the computing core occupancy, video memory occupancy, instantaneous video memory read / write bandwidth, data transmission throughput and transmission latency of the inter-card interconnection link, length of the pending request queue, and number of inference tasks completed per unit time.

[0018] This invention establishes an offline performance benchmark for each computing card by conducting performance benchmark tests on each computing card and collecting hardware operating status in real time. This provides a dynamic data source for online performance profiling, enabling the dual-mode performance profiling to simultaneously reflect the inherent capabilities of the hardware and its real-time operating status.

[0019] Another embodiment of the present invention provides a power dispatching large model inference device for computing power card clusters, comprising: the power dispatching large model inference device being used to execute the steps of a power dispatching large model inference method for computing power card clusters as described in the present invention.

[0020] Another embodiment of the present invention provides a terminal device, including: a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, it implements the steps of the power dispatching large model inference method for computing power card clusters as described in the present invention.

[0021] Another embodiment of the present invention also provides a computer-readable storage medium item, including: a stored computer program, which, when the computer program is running, controls the device where the computer-readable storage medium is located to perform the steps of a power dispatching large model inference method for computing card clusters as described in the present invention. Attached Figure Description

[0022] Figure 1 This is a flowchart illustrating an embodiment of the power dispatching large model inference method for computing power card clusters provided by the present invention. Figure 2 This is a flowchart illustrating another embodiment of the power dispatching large model inference method for computing power card clusters provided by the present invention. Figure 3 This is a schematic diagram of an embodiment of the power dispatching large model inference device for computing power card clusters provided by the present invention. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings of the embodiments. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0024] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the application; the terms “comprising” and “having”, and any variations thereof, in the specification, claims, and foregoing description of the drawings are intended to cover non-exclusive inclusion.

[0025] In the description of the embodiments of this application, technical terms such as "first" and "second" are used only to distinguish different objects and should not be construed as indicating or implying relative importance or implicitly specifying the number, specific order, or primary and secondary relationship of the indicated technical features. In the description of the embodiments of this application, "multiple" means two or more, unless otherwise explicitly defined.

[0026] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0027] In the description of the embodiments in this application, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this document generally indicates that the preceding and following related objects have an "or" relationship.

[0028] In the description of the embodiments of this application, the term "multiple" refers to two or more (including two), similarly, "multiple sets" refers to two or more (including two sets), and "multiple pieces" refers to two or more (including two pieces).

[0029] See Figure 1 To address the problem of blind matching caused by the inability of existing technologies to perceive heterogeneous differences in computing power cards and dynamic task requirements, an embodiment of the present invention provides a large-scale power scheduling model inference method for computing power card clusters, including steps S101 to S104: Step S101: Obtain the overall sequence length and peak memory requirement of the current inference request, and obtain the available memory capacity, inference task adaptation coefficient and comprehensive load score of each computing card in the target computing card cluster; wherein, the inference task adaptation coefficient and the comprehensive load score are pre-generated based on the static performance benchmark and hardware operating status of each computing card.

[0030] It should be noted that obtaining the overall sequence length and peak memory requirement of the current inference request, and obtaining the available memory capacity, inference task adaptation coefficient, and comprehensive load score of each computing card in the target computing card cluster, means: for the current inference request to be processed, obtaining the sum of the input text length and the expected output text length of the request as the overall sequence length, and obtaining the peak memory usage required to execute the request through a pre-trained prediction model; at the same time, obtaining the current remaining memory capacity of each computing card participating in inference in the cluster, the pre-generated inference task adaptation coefficient used to characterize the execution efficiency preference of each computing card for inference tasks with different sequence lengths, and the comprehensive load score used to characterize the current comprehensive load level of each computing card.

[0031] Preferably, obtaining the overall sequence length and peak memory requirement of the current inference request includes: Obtain the request attribute parameters of the current inference request; wherein, the request attribute parameters include input length, expected output length, and model type identifier; The overall sequence length is obtained by adding the input length and the expected output length. The requested attribute parameters are input into a pre-trained prediction model, so that the prediction model can predict the task volume based on the requested attribute parameters and output the peak memory requirement.

[0032] In one embodiment, a natural language inference request from a user is received, such as "based on the current real-time status of the provincial power grid, predict the renewable energy absorption capacity within the next 15 minutes". The token length (i.e., input length) of the input prompt is calculated by a token segmenter, the maximum expected token length (i.e., expected output length) is obtained from the default configuration, and the model type identifier corresponding to the inference request (such as a large power dispatch model with 7B, 13B, or 70B parameters) is extracted.

[0033] Furthermore, the input length and the expected output length are added together to obtain the overall text sequence length of the current inference request. The request attribute parameters are then input into an offline pre-trained prediction model to predict task complexity. The prediction model combines the parameters of the large-scale power dispatch model itself with the base memory usage corresponding to fixed activation values, the key-value cache memory overhead for a single text unit, and the overall text sequence length of the current inference request to calculate the peak memory requirements during the inference task execution. In addition, the prediction model can also combine the input length and the expected output length of the current inference request to comprehensively calculate the overall computational scale required for the current inference request, i.e., the expected computational load.

[0034] Preferably, the method for obtaining the static performance benchmark and the hardware operating status data includes: The performance benchmark tests are performed on each of the computing cards, and the static performance benchmark of each computing card is determined based on the test results; wherein, the static performance benchmark includes floating-point operation capability per unit time, peak memory bandwidth and architecture type information; The hardware operating status of each computing card is collected in real time; wherein, the hardware operating status includes the computing core occupancy, video memory occupancy, instantaneous video memory read / write bandwidth, data transmission throughput and transmission latency of the inter-card interconnection link, length of the pending request queue, and number of inference tasks completed per unit time.

[0035] In one embodiment, standardized, operator-level performance benchmark tests can be performed on all computing cards in the computing card cluster during the cluster initialization phase or during low-load maintenance periods. Based on the test results, the static performance benchmarks of each computing card are determined, including: floating-point operation capability per unit time (reflecting its theoretical peak computing power), peak memory bandwidth (reflecting its theoretical maximum data throughput capability), and architecture type information (used to index the corresponding optimization strategy set).

[0036] Furthermore, a local hardware status acquisition unit can be deployed on the host machine of each computing card to collect the hardware operating status of each computing card in real time, including: computing core occupancy, video memory occupancy, instantaneous video memory read / write bandwidth status, data transmission and reception throughput and transmission latency of inter-card interconnection links, length of pending request queue, and number of inference tasks completed per unit time.

[0037] Preferably, the inference task adaptation coefficient and the comprehensive load score are pre-generated based on the static performance benchmark and hardware operating status of each computing card, specifically: The static performance benchmark and hardware operation status data of each computing card are fused and processed to construct a dual-mode performance profile of each computing card; wherein, the parameters of the dual-mode performance profile include the current available computing power, the current available video memory bandwidth, the current available communication transmission power, and the current service queue pressure factor. The currently available computing power, the currently available video memory bandwidth, the currently available communication transmission capacity, and the current service queue pressure factor are weighted and aggregated to generate the inference task adaptation coefficient and comprehensive load score for each computing card.

[0038] In one embodiment, the offline static performance benchmark of the computing card is fused with the real-time collected hardware operating status to construct a dual-mode performance profile for each computing card. The profile parameters include: currently available computing power, currently available video memory bandwidth, currently available communication transmission capacity, and current service queue pressure factor.

[0039] Furthermore, based on the weighted aggregation of parameters in the dual-mode performance profile, the real-time comprehensive load score and inference task adaptation coefficient of each computing card are obtained. Specifically, the currently available computing power, the currently available memory bandwidth, and the currently available communication transmission capacity are assigned a first set of weights and weighted aggregation to generate the inference task adaptation coefficient; the currently available computing power, the currently available memory bandwidth, the currently available communication transmission capacity, and the current service queue pressure factor are assigned a second set of weights and weighted aggregation to generate the comprehensive load score. The inference task adaptation coefficient is used for sequence length-aware pre-allocation correction, reflecting the computing card's preference and friendliness towards tasks of different sequence lengths. For example, computing cards with large memory capacity and high bandwidth have a higher adaptation coefficient for inference tasks with long sequence lengths; conversely, computing cards with fine-grained computing unit scheduling but limited memory capacity have a higher adaptation coefficient for inference tasks with short sequence lengths.

[0040] Step S102: Calculate the sequence length matching degree between the current inference request and each computing power card based on the overall sequence length and the adaptation coefficient of each inference task, and generate the pre-allocated weight of each computing power card based on the sequence length matching degree.

[0041] It should be noted that calculating the sequence length matching degree between the current inference request and each computing card based on the overall sequence length and the adaptation coefficient of each inference task, and generating the pre-allocated weight of each computing card based on the sequence length matching degree means: taking the overall sequence length of the current inference request as input, combining the inference task adaptation coefficient of each computing card, and calculating the sequence length matching degree between the current inference request and each computing card through the matching degree calculation function. This matching degree is used to measure the execution efficiency of each computing card for the current sequence length inference task. Then, according to the matching degree of each computing card, a corresponding pre-allocated weight is generated for each computing card. The higher the matching degree, the larger the pre-allocated weight, indicating that the computing card has a higher adaptation priority when executing the current length inference task.

[0042] Preferably, the step of calculating the sequence length matching degree between the current inference request and each computing power card based on the overall sequence length and the adaptation coefficient of each inference task, and generating the pre-allocated weights of each computing power card based on the sequence length matching degree, includes: For any inference task adaptation coefficient, the overall sequence length and the inference task adaptation coefficient are substituted into the matching function for calculation to obtain the sequence length matching degree between the current inference request and the computing power card corresponding to the inference task adaptation coefficient. The initial pre-allocated weight of the computing power card is generated based on the sequence length matching degree, and the initial pre-allocated weight is corrected based on the current business queue pressure factor corresponding to the computing power card to obtain the pre-allocated weight of the computing power card.

[0043] In one embodiment, the sequence length matching degree between each computing power card and the current inference request is determined based on the inference task adaptation coefficient corresponding to each computing power card: ; in, The sequence length matching degree; The length of the entire sequence; For the reasoning task adaptation coefficient; For the matching function, methods such as table lookup matching, linear interpolation, or kernel function operations can be used. The higher the sequence length matching degree, the higher the expected running efficiency of the computing card in executing inference tasks of the current length.

[0044] Furthermore, an initial pre-allocated weight is generated for each computing card based on the sequence length matching degree. This initial pre-allocated weight is then corrected by combining it with the current service queue pressure factor corresponding to the computing card, resulting in a corrected pre-allocated weight. Specifically, the initial pre-allocated weight is calculated using a negative correlation function with the service queue pressure factor to obtain the corrected pre-allocated weight. The negative correlation function is used to reduce the pre-allocated weight as the service queue pressure factor increases. In addition, schedulable load-aware task units can be generated as initial preferences for subsequent scheduling decisions. These load-aware task units include: inference requests, expected computational load, peak memory requirements, the expected maximum latency of the inference request (QoS constraint), and pre-allocated weights.

[0045] Step S103: Screen out computing cards with available video memory capacity less than the peak video memory requirement, and comprehensively evaluate the remaining computing cards according to the pre-allocated weight and the comprehensive load score, and determine the target computing card based on the evaluation results.

[0046] It should be noted that the process of filtering out computing cards with available video memory capacity less than the peak video memory requirement, and comprehensively evaluating the remaining computing cards based on the pre-allocated weights and the comprehensive load score, and determining the target computing card based on the evaluation results, means: first, filtering out computing cards with insufficient available video memory capacity to support the current inference task, ensuring that all remaining computing cards meet the hard constraint of the peak video memory requirement; then, comprehensively evaluating each remaining computing card based on its pre-allocated weights and comprehensive load score; the evaluation results are used to determine the overall suitability of each computing card; and finally, determining the target computing card from the remaining computing cards based on the evaluation results.

[0047] Preferably, the step of filtering out computing cards with available video memory capacity less than the peak video memory requirement, and comprehensively evaluating the remaining computing cards based on the pre-allocated weights and the comprehensive load score, and determining the target computing card based on the evaluation results, includes: The computing cards with available video memory capacity less than the peak video memory requirement are filtered out to obtain a set of candidate computing cards. For any computing card in the candidate computing card set, the comprehensive load score of the computing card is weighted according to the pre-allocated weight of the computing card to generate the weighted load score of the computing card. The computing card with the smallest weighted load score in the candidate computing card set is selected as the target computing card.

[0048] In one embodiment, all load-aware task units generated by inference requests can be stored in a global task pool, and a global dynamic priority queue can be built in the global task pool. The load-aware task units are first sorted in ascending order according to the expected maximum latency (the smaller the expected maximum latency, the earlier it is placed), and then sorted in descending order according to the expected computational load (the larger the expected computational load, the earlier it is placed).

[0049] Furthermore, the computing cards in the computing card cluster can be organized into logical rings according to physical topology (such as rack or switch port distance). Select a computing card as the ring head, generate an initial scheduling authorization certificate, including: the current timestamp and incrementing sequence number, the comprehensive load score of each computing card and the number of tasks to be scheduled; and transmit the initial scheduling authorization certificate unidirectionally along the logical ring direction.

[0050] Furthermore, the computing power card holding the scheduling authorization certificate can extract the load-aware task unit at the head of the global dynamic priority queue (this load-aware task unit can be regarded as the current inference request generated through steps S101 and S102). The number of tasks extracted at one time is dynamically adapted according to the holding time of the current scheduling authorization certificate. For each extracted load-aware task unit, the following allocation operation is performed: combining the pre-allocated weight of the load-aware task unit and the comprehensive load score of each computing power card, the computing power card with the smallest weighted load score and whose available video memory capacity can meet the peak video memory requirements of the load-aware task unit is selected as the target computing power card for task allocation (i.e., step S103); wherein, the weighted load score is obtained by multiplying the pre-allocated weight by the corresponding comprehensive load score; the load-aware task unit is sent to the local request queue of the target computing power card, and the identifier of the corresponding target computing power card and the task allocation timestamp are recorded in the load-aware task unit; if the length of the local task queue of the target computing power card exceeds the preset threshold, the cluster load migration mechanism can be triggered. After completing the above scheduling decision, the computing power card holding the scheduling authorization certificate forwards the scheduling authorization certificate to its pre-agreed successor card on the logical ring and enters a waiting state until it receives the certificate again.

[0051] Specifically, before each computing card passes scheduling authorization credentials to subsequent computing cards, the upper limit for holding scheduling authorization credentials for the next time is dynamically adjusted based on the cluster's global inference request arrival rate and average inference task execution latency. ; in, Maximum holding time; A fixed benchmark holding period; The number of requests arriving per unit of time; The target processing rate is set at, for example, 80% of the maximum throughput. When the request arrival rate increases, the holding time of scheduling authorization credentials is shortened to accelerate the transfer of scheduling authorization credentials between computing cards and improve scheduling response efficiency; when the cluster is idle, the holding time of scheduling authorization credentials is extended to reduce unnecessary communication overhead during the scheduling process.

[0052] Step S104: The current inference request is sent to the target computing card so that the target computing card can perform power scheduling large model inference according to the current inference request and generate inference results.

[0053] It should be noted that sending the current inference request to the target computing power card so that the target computing power card can perform power dispatching large-scale model inference based on the current inference request and generate inference results means: allocating the current inference request to the target computing power card, and having the target computing power card perform power dispatching large-scale model inference calculation based on the current inference request, outputting the corresponding inference results, and completing the inference service response.

[0054] Preferably, the step of sending the current inference request to the target computing card, so that the target computing card performs power dispatching large-scale model inference based on the current inference request and generates inference results, includes: The current inference request is sent to the target computing card so that the target computing card allocates video memory storage space for the current inference request according to the peak video memory requirement of the current inference request. Load a hardware affinity operator library adapted to the hardware architecture characteristics of the target computing card, and perform power scheduling large model inference based on the video memory storage space and the hardware affinity operator library to generate inference results.

[0055] In one embodiment, based on the pre-identified hardware architecture characteristics of the target computing card, a hard affinity operator library adapted to the hardware architecture characteristics is loaded. A multi-precision hybrid operation mode is adopted, merging operations such as query, key-value matrix multiplication, scaling, and masking into a single kernel, and consolidating consecutive operation steps. This reduces the frequency of computing kernel startup, optimizes the GPU memory read / write interaction process, and improves the execution efficiency of inference operations. During the execution of the inference task, based on the peak GPU memory demand of the load-aware task unit, contiguous GPU memory storage space is pre-allocated for the current task for inference caching. After the inference task is completed, the occupied GPU memory space is marked as reusable, reducing GPU memory fragmentation and lowering the resource overhead caused by repeated GPU memory allocation and release.

[0056] Specifically, during the inference process of the large-scale power dispatch model, the actual execution time and peak memory usage are recorded and integrated to generate real-time execution data. The actual execution time and peak memory usage of the inference task are compared and verified with the previously predicted computational scale and peak memory requirements. Prediction deviations are statistically analyzed, and the fitting coefficient of the task complexity prediction model is corrected to improve the accuracy of subsequent task resource requirement predictions. Based on real-time execution data, core scheduling parameters such as the competition attenuation coefficient, scheduling weight calculation parameters, and token forwarding frequency threshold of the dual-mode performance profile of the computing power card are dynamically adjusted. For the actual operating efficiency differences between long and short sequence tasks, the computing power card inference task adaptation coefficient is optimized to enhance the matching accuracy between tasks and computing power cards. Global indicators such as the overall cluster inference task execution latency, resource utilization, and failure frequency are summarized, and the task allocation rules, load migration triggering conditions, and failover strategies of the two-level scheduling architecture are iteratively optimized to achieve continuous adaptive optimization of cluster scheduling performance. Extremely abnormal inference task execution data is filtered out to avoid parameter misadjustments caused by hardware instantaneous failures or sudden task impacts, ensuring the stability and reliability of the scheduling strategy iteration process.

[0057] This invention provides a quantitative data foundation for subsequent computing card matching by acquiring the overall sequence length and peak memory requirement of inference requests, as well as the available memory capacity of each computing card, pre-generated inference task adaptation coefficients, and comprehensive load scores. Simultaneously, by setting the inference task adaptation coefficients and comprehensive load scores to be generated based on the static performance benchmarks and hardware operating states of each computing card, these two parameters are generated based on the inherent hardware attributes and actual operating states of the computing cards, rather than being fixed values ​​set artificially. This provides physically grounded quantitative data for subsequent matching decisions. By combining the overall sequence length with the inference task adaptation coefficients of each computing card to calculate the sequence length matching degree and generate pre-allocated weights, the task sequence length characteristics can be quantitatively matched with the hardware adaptation preferences of the computing cards, avoiding mismatches between inference tasks and computing cards based on sequence length. By comprehensively evaluating and determining the target computing card among computing cards whose memory capacity meets peak memory requirements based on pre-allocated weights and overall load scores, the requirements of three dimensions—hard memory constraints, hardware adaptation preferences, and current load balancing—can be simultaneously met. Among the adapted computing cards, the one with the best overall conditions is selected for inference. By sending inference requests to the target computing card to execute power dispatching model inference and generate inference results, scheduling decisions can be transformed into specific inference outputs, realizing online inference services for the power dispatching model.

[0058] Optionally, in this embodiment of the invention, the inference task adaptation coefficient and the comprehensive load score are pre-generated based on the static performance benchmark and hardware operating status of each computing card, specifically: The static performance benchmark and hardware operation status data of each computing card are fused and processed to construct a dual-mode performance profile of each computing card; wherein, the parameters of the dual-mode performance profile include the current available computing power, the current available video memory bandwidth, the current available communication transmission power, and the current service queue pressure factor. The currently available computing power, the currently available video memory bandwidth, the currently available communication transmission capacity, and the current service queue pressure factor are weighted and aggregated to generate the inference task adaptation coefficient and comprehensive load score for each computing card.

[0059] This invention constructs a dual-mode performance profile by fusing the static performance benchmarks and hardware operating status data of each computing card, and generates inference task adaptation coefficients and comprehensive load scores by weighting and aggregating the parameters in the profile. This enables the adaptation coefficients and load scores to simultaneously reflect the inherent hardware capabilities and real-time operating status of the computing card, providing a quantitative basis for matching subsequent tasks with computing cards.

[0060] Optionally, in this embodiment of the invention, the step of calculating the sequence length matching degree between the current inference request and each of the computing power cards based on the overall sequence length and the adaptation coefficient of each inference task, and generating the pre-allocated weights of each computing power card based on the sequence length matching degree, includes: For any inference task adaptation coefficient, the overall sequence length and the inference task adaptation coefficient are substituted into the matching function for calculation to obtain the sequence length matching degree between the current inference request and the computing power card corresponding to the inference task adaptation coefficient. The initial pre-allocated weight of the computing power card is generated based on the sequence length matching degree, and the initial pre-allocated weight is corrected based on the current business queue pressure factor corresponding to the computing power card to obtain the pre-allocated weight of the computing power card.

[0061] This invention calculates the sequence length matching degree and generates pre-allocated weights by substituting the overall sequence length and inference task adaptation coefficient into the matching function, and then corrects the pre-allocated weights according to the business queue pressure factor. This enables the pre-allocated weights to reflect both hardware adaptation preferences and the current queue congestion, thus avoiding the allocation of tasks to computing cards that have accumulated too much power.

[0062] Optionally, in this embodiment of the invention, the step of filtering out computing cards with available video memory capacity less than the peak video memory requirement, and comprehensively evaluating the remaining computing cards according to the pre-allocated weights and the comprehensive load score, and determining the target computing card based on the evaluation results, includes: The computing cards with available video memory capacity less than the peak video memory requirement are filtered out to obtain a set of candidate computing cards. For any computing card in the candidate computing card set, the comprehensive load score of the computing card is weighted according to the pre-allocated weight of the computing card to generate the weighted load score of the computing card. The computing card with the smallest weighted load score in the candidate computing card set is selected as the target computing card.

[0063] This invention enables the matching of tasks and computing cards to simultaneously meet the requirements of three dimensions: hard constraints on video memory, hardware adaptation preferences, and current load balancing. By filtering out computing cards whose available video memory capacity does not meet the peak video memory requirements, and then weighting the comprehensive load score according to the pre-allocated weights, the computing card with the smallest weighted load score is selected as the target.

[0064] Optionally, in this embodiment of the invention, the step of sending the current inference request to the target computing power card, so that the target computing power card performs power dispatching large-scale model inference according to the current inference request and generates inference results, includes: The current inference request is sent to the target computing card so that the target computing card allocates video memory storage space for the current inference request according to the peak video memory requirement of the current inference request. Load a hardware affinity operator library adapted to the hardware architecture characteristics of the target computing card, and perform power scheduling large model inference based on the video memory storage space and the hardware affinity operator library to generate inference results.

[0065] This invention, through sending inference requests to the target computing card and allocating video memory space thereon, loads a hardware affinity operator library adapted to the hardware architecture of the target computing card to perform inference, thereby enabling the inference execution process to be deeply adapted to the hardware characteristics, reducing video memory fragmentation and execution overhead, and improving inference execution efficiency.

[0066] Optionally, in this embodiment of the invention, obtaining the overall sequence length and peak memory requirement of the current inference request includes: Obtain the request attribute parameters of the current inference request; wherein, the request attribute parameters include input length, expected output length, and model type identifier; The overall sequence length is obtained by adding the input length and the expected output length. The requested attribute parameters are input into a pre-trained prediction model, so that the prediction model can predict the task volume based on the requested attribute parameters and output the peak memory requirement.

[0067] This invention, through obtaining the input length, expected output length, and model type identifier of the inference request, adds the input length and expected output length to obtain the overall sequence length, and inputs the request attribute parameters into the pre-trained prediction model to output peak memory requirements. This enables quantitative prediction of the computational resource consumption of the inference task, providing a quantitative basis for matching subsequent tasks with computing cards.

[0068] Optionally, in this embodiment of the invention, the method for obtaining the static performance benchmark and the hardware operating status data includes: The performance benchmark tests are performed on each of the aforementioned computing cards, and the static performance benchmarks of each computing card are determined based on the test results; wherein, the static performance benchmarks include floating-point operation capability per unit time, peak memory bandwidth, and architecture type information; The hardware operating status of each computing card is collected in real time; wherein, the hardware operating status includes the computing core occupancy, video memory occupancy, instantaneous video memory read / write bandwidth, data transmission throughput and transmission latency of the inter-card interconnection link, length of the pending request queue, and number of inference tasks completed per unit time.

[0069] This invention establishes an offline performance benchmark for each computing card by conducting performance benchmark tests on each computing card and collecting hardware operating status in real time. This provides a dynamic data source for online performance profiling, enabling the dual-mode performance profiling to simultaneously reflect the inherent capabilities of the hardware and its real-time operating status.

[0070] like Figure 2 As shown, based on the above-described method embodiments, another embodiment of the power dispatching large-scale model inference method for computing power card clusters is provided, including steps S1 to S6: Step S1: Perform real-time perception of computing power status of the computing power card cluster; wherein, step S1 is used to obtain the inference task adaptation coefficient and comprehensive load score mentioned in step S101; Step S2 involves parsing user requests and extracting load features, as well as predicting task computation load and video memory requirements; wherein, step S2 is used to obtain the overall sequence length and peak video memory requirements mentioned in step S101. Step S3: Perform sequence length-aware pre-allocation correction; wherein, step S3 is used to obtain the pre-allocated weights mentioned in step S102; Step S4: Construct a global dynamic priority queue and perform logical ring initialization and scheduling authorization certificate issuance; wherein, step S4 is used to determine the computing power card to execute step S103; Step S5: The task is sent to the local queue of the target card and hardware affinity inference acceleration is performed; wherein, performing step S5 is equivalent to performing step S104. Step S6: Collect actual performance data of the task and perform feedback closed-loop adaptive parameter optimization.

[0071] This invention provides a quantitative data foundation for subsequent computing card matching by acquiring the overall sequence length and peak memory requirement of inference requests, as well as the available memory capacity of each computing card, pre-generated inference task adaptation coefficients, and comprehensive load scores. Simultaneously, by setting the inference task adaptation coefficients and comprehensive load scores to be generated based on the static performance benchmarks and hardware operating states of each computing card, these two parameters are generated based on the inherent hardware attributes and actual operating states of the computing cards, rather than being fixed values ​​set artificially. This provides physically grounded quantitative data for subsequent matching decisions. By combining the overall sequence length with the inference task adaptation coefficients of each computing card to calculate the sequence length matching degree and generate pre-allocated weights, the task sequence length characteristics can be quantitatively matched with the hardware adaptation preferences of the computing cards, avoiding mismatches between inference tasks and computing cards based on sequence length. By comprehensively evaluating and determining the target computing card among computing cards whose memory capacity meets peak memory requirements based on pre-allocated weights and overall load scores, the requirements of three dimensions—hard memory constraints, hardware adaptation preferences, and current load balancing—can be simultaneously met. Among the adapted computing cards, the one with the best overall conditions is selected for inference. By sending inference requests to the target computing card to execute power dispatching model inference and generate inference results, scheduling decisions can be transformed into specific inference outputs, realizing online inference services for the power dispatching model.

[0072] like Figure 3 As shown, based on the above method embodiments, corresponding apparatus embodiments are provided; One embodiment of the present invention provides a power dispatching large model inference device for computing power card clusters, including: a data acquisition module 301, a sequence length matching module 302, a load balancing module 303, and a model inference module 304; The data acquisition module 301 is used to acquire the overall sequence length and peak memory requirement of the current inference request, and to acquire the available memory capacity, inference task adaptation coefficient and comprehensive load score of each computing card in the target computing card cluster; wherein, the inference task adaptation coefficient and the comprehensive load score are pre-generated based on the static performance benchmark and hardware operating status of each computing card. The sequence length matching module 302 is used to calculate the sequence length matching degree between the current inference request and each computing power card based on the overall sequence length and the adaptation coefficient of each inference task, and to generate the pre-allocated weight of each computing power card based on the sequence length matching degree. The load balancing module 303 is used to filter out computing cards whose available video memory capacity is less than the peak video memory requirement, and to comprehensively evaluate the remaining computing cards according to the pre-allocated weight and the comprehensive load score, and to determine the target computing card according to the evaluation results. The model inference module 304 is used to send the current inference request to the target computing card, so that the target computing card can perform power scheduling large model inference according to the current inference request and generate inference results.

[0073] Optionally, in this embodiment of the invention, the data acquisition module 301 includes: a profile construction submodule and a computing power perception submodule; The profile construction submodule is used to fuse the static performance benchmark and hardware operation status data of each computing card to construct a dual-mode performance profile of each computing card; wherein, the parameters of the dual-mode performance profile include the current available computing power, the current available video memory bandwidth, the current available communication transmission power, and the current service queue pressure factor. The computing power awareness submodule is used to perform weighted aggregation of the currently available computing power, the currently available video memory bandwidth, the currently available communication transmission power, and the current service queue pressure factor to generate the inference task adaptation coefficient and comprehensive load score for each computing power card.

[0074] This invention constructs a dual-mode performance profile by fusing the static performance benchmarks and hardware operating status data of each computing card, and generates inference task adaptation coefficients and comprehensive load scores by weighting and aggregating the parameters in the profile. This enables the adaptation coefficients and load scores to simultaneously reflect the inherent hardware capabilities and real-time operating status of the computing card, providing a quantitative basis for matching subsequent tasks with computing cards.

[0075] Optionally, in this embodiment of the invention, the sequence length matching module 302 includes: a matching degree calculation submodule and a weight generation submodule; The matching degree calculation submodule is used to calculate the sequence length matching degree between the current inference request and the computing power card corresponding to the inference task adaptation coefficient by substituting the overall sequence length and the inference task adaptation coefficient into the matching function for any inference task adaptation coefficient. The weight generation submodule is used to generate the initial pre-allocated weight of the computing power card based on the sequence length matching degree, and to correct the initial pre-allocated weight based on the current business queue pressure factor corresponding to the computing power card to obtain the pre-allocated weight of the computing power card.

[0076] This invention calculates the sequence length matching degree and generates pre-allocated weights by substituting the overall sequence length and inference task adaptation coefficient into the matching function, and then corrects the pre-allocated weights according to the business queue pressure factor. This enables the pre-allocated weights to reflect both hardware adaptation preferences and the current queue congestion, thus avoiding the allocation of tasks to computing cards that have accumulated too much power.

[0077] Optionally, in this embodiment of the invention, the load balancing module 303 includes: a video memory comparison submodule, a score weighting submodule, and a target computing power card determination submodule; The memory comparison submodule is used to filter out computing cards whose available memory capacity is less than the peak memory requirement, and obtain a set of candidate computing cards. The score weighting submodule is used to weight the comprehensive load score of any computing card in the candidate computing card set according to the pre-allocated weight of the computing card, and generate the weighted load score of the computing card. The target computing power card determination submodule is used to select the computing power card with the smallest weighted load score in the candidate computing power card set as the target computing power card.

[0078] This invention enables the matching of tasks and computing cards to simultaneously meet the requirements of three dimensions: hard constraints on video memory, hardware adaptation preferences, and current load balancing. By filtering out computing cards whose available video memory capacity does not meet the peak video memory requirements, and then weighting the comprehensive load score according to the pre-allocated weights, the computing card with the smallest weighted load score is selected as the target.

[0079] Optionally, in this embodiment of the invention, the model inference module 304 includes: a video memory allocation submodule and a model inference submodule; The video memory allocation submodule is used to send the current inference request to the target computing card, so that the target computing card allocates video memory storage space for the current inference request according to the peak video memory requirement of the current inference request. The model inference submodule is used to load a hardware affinity operator library adapted to the hardware architecture features of the target computing card, and to perform power dispatching large model inference and generate inference results based on the video memory storage space and the hardware affinity operator library.

[0080] This invention, through sending inference requests to the target computing card and allocating video memory space thereon, loads a hardware affinity operator library adapted to the hardware architecture of the target computing card to perform inference, thereby enabling the inference execution process to be deeply adapted to the hardware characteristics, reducing video memory fragmentation and execution overhead, and improving inference execution efficiency.

[0081] Optionally, in this embodiment of the invention, the data acquisition module 301 further includes: a parameter acquisition submodule, a length calculation submodule, and a task volume prediction submodule; The parameter acquisition submodule is used to acquire the request attribute parameters of the current inference request; wherein, the request attribute parameters include input length, expected output length and model type identifier; The length calculation submodule is used to add the input length and the expected output length to obtain the overall sequence length; The task volume prediction submodule is used to input the request attribute parameters into a pre-trained prediction model, so that the prediction model can predict the task volume based on the request attribute parameters and output the peak memory requirement.

[0082] This invention, through obtaining the input length, expected output length, and model type identifier of the inference request, adds the input length and expected output length to obtain the overall sequence length, and inputs the request attribute parameters into the pre-trained prediction model to output peak memory requirements. This enables quantitative prediction of the computational resource consumption of the inference task, providing a quantitative basis for matching subsequent tasks with computing cards.

[0083] Optionally, in this embodiment of the invention, the data acquisition module 301 further includes: a performance benchmark testing submodule and a hardware operating status acquisition submodule; The performance benchmark testing submodule is used to perform performance benchmark tests on each of the computing cards and determine the static performance benchmark of each computing card based on the test results; wherein, the static performance benchmark includes floating-point operation capability per unit time, peak memory bandwidth and architecture type information; The hardware operation status acquisition submodule is used to collect the hardware operation status of each computing card in real time; wherein, the hardware operation status includes computing core occupancy, video memory occupancy, instantaneous video memory read / write bandwidth status, data transmission and reception throughput and transmission latency of inter-card interconnection links, length of pending request queue and number of inference tasks completed per unit time.

[0084] This invention establishes an offline performance benchmark for each computing card by conducting performance benchmark tests on each computing card and collecting hardware operating status in real time. This provides a dynamic data source for online performance profiling, enabling the dual-mode performance profiling to simultaneously reflect the inherent capabilities of the hardware and its real-time operating status.

[0085] It is understood that the above-described device embodiments correspond to the method embodiments of the present invention, and can implement the power dispatching large model inference method for computing card clusters provided by any of the above-described method embodiments of the present invention.

[0086] In this embodiment of the invention, the data acquisition module 301 acquires the overall sequence length and peak memory requirement of the inference request, as well as the available memory capacity of each computing card, the pre-generated inference task adaptation coefficient, and the comprehensive load score, providing a quantitative data foundation for subsequent computing card matching. Simultaneously, the data acquisition module 301 sets the inference task adaptation coefficient and comprehensive load score to be generated based on the static performance benchmark and hardware operating status of each computing card. This ensures that the two parameters are generated based on the inherent hardware attributes and actual operating status of the computing cards, rather than fixed values ​​set manually, providing physically grounded quantitative data for subsequent matching decisions. The sequence length matching module 302 combines the overall sequence length with the inference task adaptation coefficient of each computing card to calculate the sequence length matching degree and generate pre-allocated weights. This quantitatively matches the task sequence length characteristics with the hardware adaptation preferences of the computing cards, avoiding mismatches between inference tasks and computing cards based on sequence length. The load balancing module 303 comprehensively evaluates and determines the target computing card among computing cards whose memory capacity meets the peak memory requirements, based on pre-allocated weights and overall load scores. This simultaneously satisfies the requirements of three dimensions: hard memory constraints, hardware adaptation preferences, and current load balancing. Among the adapted computing cards, the card with the best overall conditions is selected for inference. The model inference module 304 sends inference requests to the target computing card to execute the power dispatching large-scale model inference and generate inference results. This enables the dispatching decisions to be transformed into specific inference outputs, realizing online inference services for the power dispatching large-scale model.

[0087] It should be noted that the device embodiments described above are merely illustrative, and some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Furthermore, in the accompanying drawings of the device embodiments provided by this invention, the connection relationships between modules indicate that they have communication connections, which can specifically be implemented as one or more communication buses or signal lines. Those skilled in the art can understand and implement this without any creative effort.

[0088] Based on the above embodiment of a power dispatching large model inference method for computing power card clusters, another embodiment of the present invention provides a terminal device, which includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, it implements a power dispatching large model inference method for computing power card clusters according to any embodiment of the present invention.

[0089] For example, in this embodiment, the computer program can be divided into one or more modules, which are stored in the memory and executed by the processor to complete the present invention. The one or more modules may be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of the computer program in the terminal device.

[0090] The terminal device may be a desktop computer, laptop, handheld computer, or cloud server, etc. The terminal device may include, but is not limited to, a processor and a memory.

[0091] The processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor. The processor is the control center of the terminal device, connecting all parts of the terminal device via various interfaces and lines.

[0092] Based on the above-described method embodiments, another embodiment of the present invention provides a computer-readable storage medium including a stored computer program, wherein, when the computer program is running, it controls the device where the computer-readable storage medium is located to execute the power dispatching large model inference method for computing card clusters described in any of the above-described method embodiments of the present invention.

[0093] The modules / units integrated in the device / terminal equipment, if implemented as software functional units and sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the above embodiments of the present invention can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc.

[0094] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A power dispatching large-scale model inference method for computing power card clusters, characterized in that, include: Obtain the overall sequence length and peak memory requirement of the current inference request, and obtain the available memory capacity, inference task adaptation coefficient, and overall load score of each computing card in the target computing card cluster; wherein, the inference task adaptation coefficient and the overall load score are pre-generated based on the static performance benchmark and hardware operating status of each computing card; Based on the overall sequence length and the adaptation coefficient of each inference task, the sequence length matching degree between the current inference request and each computing card is calculated, and the pre-allocated weight of each computing card is generated based on the sequence length matching degree. The computing cards with available video memory capacity less than the peak video memory requirement are screened out, and the remaining computing cards are comprehensively evaluated based on the pre-allocated weight and the comprehensive load score. The target computing card is determined based on the evaluation results. The current inference request is sent to the target computing card so that the target computing card can perform power dispatching large-scale model inference based on the current inference request and generate inference results.

2. The power dispatching large-scale model inference method for computing power card clusters as described in claim 1, characterized in that, The inference task adaptation coefficient and the overall load score are pre-generated based on the static performance benchmarks and hardware operating status of each computing card, specifically: The static performance benchmark and hardware operation status data of each computing card are fused and processed to construct a dual-mode performance profile of each computing card; wherein, the parameters of the dual-mode performance profile include the current available computing power, the current available video memory bandwidth, the current available communication transmission power, and the current service queue pressure factor. The currently available computing power, the currently available video memory bandwidth, the currently available communication transmission capacity, and the current service queue pressure factor are weighted and aggregated to generate the inference task adaptation coefficient and comprehensive load score for each computing card.

3. The power dispatching large-scale model inference method for computing power card clusters as described in claim 2, characterized in that, The step of calculating the sequence length matching degree between the current inference request and each computing card based on the overall sequence length and the adaptation coefficient of each inference task, and generating the pre-allocated weights of each computing card based on the sequence length matching degree, includes: For any inference task adaptation coefficient, the overall sequence length and the inference task adaptation coefficient are substituted into the matching function for calculation to obtain the sequence length matching degree between the current inference request and the computing power card corresponding to the inference task adaptation coefficient. The initial pre-allocated weight of the computing power card is generated based on the sequence length matching degree, and the initial pre-allocated weight is corrected based on the current business queue pressure factor corresponding to the computing power card to obtain the pre-allocated weight of the computing power card.

4. The power dispatching large-scale model inference method for computing power card clusters as described in claim 1, characterized in that, The process of filtering out computing cards with available video memory capacity less than the peak video memory requirement, and comprehensively evaluating the remaining computing cards based on the pre-allocated weights and the overall load score, and determining the target computing card based on the evaluation results, includes: The computing cards with available video memory capacity less than the peak video memory requirement are filtered out to obtain a set of candidate computing cards. For any computing card in the candidate computing card set, the comprehensive load score of the computing card is weighted according to the pre-allocated weight of the computing card to generate the weighted load score of the computing card; The computing card with the smallest weighted load score in the candidate computing card set is selected as the target computing card.

5. The power dispatching large-scale model inference method for computing power card clusters as described in claim 1, characterized in that, The step of sending the current inference request to the target computing card, so that the target computing card can perform power dispatching large-scale model inference based on the current inference request and generate inference results, includes: The current inference request is sent to the target computing card so that the target computing card allocates video memory storage space for the current inference request according to the peak video memory requirement of the current inference request. Load a hardware affinity operator library adapted to the hardware architecture characteristics of the target computing card, and perform power scheduling large model inference based on the video memory storage space and the hardware affinity operator library to generate inference results.

6. The power dispatching large-scale model inference method for computing power card clusters as described in claim 1, characterized in that, The process of obtaining the overall sequence length and peak memory requirement of the current inference request includes: Obtain the request attribute parameters of the current inference request; wherein, the request attribute parameters include input length, expected output length, and model type identifier; The overall sequence length is obtained by adding the input length and the expected output length. The requested attribute parameters are input into a pre-trained prediction model, so that the prediction model can predict the task volume based on the requested attribute parameters and output the peak memory requirement.

7. A power dispatching large-scale model inference method for computing power card clusters as described in claims 1-6, characterized in that, The method for obtaining the static performance benchmark and the hardware operating status data includes: The performance benchmark tests are performed on each of the aforementioned computing cards, and the static performance benchmarks of each computing card are determined based on the test results; wherein, the static performance benchmarks include floating-point operation capability per unit time, peak memory bandwidth, and architecture type information; The hardware operating status of each computing card is collected in real time; wherein, the hardware operating status includes the computing core occupancy, video memory occupancy, instantaneous video memory read / write bandwidth, data transmission throughput and transmission latency of the inter-card interconnection link, length of the pending request queue, and number of inference tasks completed per unit time.

8. A power dispatching large-scale model inference device for computing power card clusters, characterized in that, include: The power dispatching large model inference device is used to execute the power dispatching large model inference method for computing card clusters as described in any one of claims 1-7.

9. A terminal device, characterized in that, include: The processor, the memory, and the computer program stored in the memory and configured to be executed by the processor, wherein when the processor executes the computer program, it implements a power dispatching large model inference method for computing power card clusters as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, include: A stored computer program, wherein, when the computer program is executed, the device containing the computer-readable storage medium is controlled to execute a power dispatching large model inference method for computing card clusters as described in any one of claims 1-7.

Images