Processing method and device for a model

Abstract

The application provides a model processing method and device, the method at least includes: based on a plurality of hardware parameters of an electronic device running a target model, determining a first instantaneous resource potential; the first instantaneous resource potential is used to represent the instantaneous hardware remaining capacity of the electronic device for running the target model; based on the first instantaneous resource potential, a first adjustment factor for adjusting the batch size of the target model is determined; the first instantaneous resource potential is positively correlated with the first adjustment factor; based on the first adjustment factor, the target batch size of the target model in the current scheduling period is determined; in the current scheduling period, the target model performs model inference based on the target batch size to match the hardware remaining capacity of the electronic device.

Description

Technical Field

[0001] This application relates to model processing techniques, and more particularly to model processing methods and devices. Background Technology

[0002] In large-scale model inference scenarios, the inference performance of the model is crucial. Related technologies primarily focus on the utilization of GPU memory for model inference, failing to fully exploit the device's potential and resulting in low resource utilization. Summary of the Invention

[0003] This application provides at least one method and apparatus for processing a model.

[0004] The technical solution of this application is implemented as follows: Firstly, this application provides a method for processing a model, the method comprising: Based on multiple hardware parameters of the electronic device running the target model, the first instantaneous resource potential is determined; the first instantaneous resource potential is used to characterize the instantaneous remaining hardware capacity of the electronic device for running the target model. Based on the first instantaneous resource potential, a first adjustment factor is determined for adjusting the batch size of the target model; the first instantaneous resource potential is positively correlated with the first adjustment factor; The target batch size of the target model in the current scheduling cycle is determined based on the first adjustment factor; Under the current scheduling cycle, the target model performs model inference based on the target batch size to match the remaining hardware capacity of the electronic device.

[0005] Secondly, this application provides a model processing apparatus, comprising at least: The first determining unit is used to determine the first instantaneous resource potential based on multiple hardware parameters of the electronic device running the target model; the first instantaneous resource potential is used to characterize the instantaneous remaining hardware capacity of the electronic device for running the target model. The second determining unit is used to determine a first adjustment factor for adjusting the batch size of the target model based on the first instantaneous resource potential; the first instantaneous resource potential is positively correlated with the first adjustment factor; The third determining unit is used to determine the target batch size of the target model in the current scheduling cycle based on the first adjustment factor; The processing unit is used to perform model inference based on the target batch size in the current scheduling cycle to match the remaining hardware capacity of the electronic device.

[0006] Thirdly, this application provides an electronic device comprising multiple hardware components and a processor, on which a target model runs; the processor is configured to: determine a first instantaneous resource potential based on multiple hardware parameters; the first instantaneous resource potential characterizes the instantaneous comprehensive hardware remaining capacity of the electronic device for running the target model; determine a first adjustment factor for adjusting the batch size of the target model based on the first instantaneous resource potential; the first instantaneous resource potential is positively correlated with the first adjustment factor; determine a target batch size of the target model in the current scheduling period based on the first adjustment factor; and in the current scheduling period, the target model performs model inference based on the target batch size to match the processing capacity of the electronic device.

[0007] Fourthly, this application provides a computer-readable storage medium storing a computer program or instructions that, when executed by a processor, implement the method provided in the first aspect.

[0008] Fifthly, this application provides a computer program product, which includes a computer program or instructions that, when executed by a processor, implement the method provided in the first aspect. Attached Figure Description

[0009] Figure 1 A schematic diagram of a first optional process for processing a model provided in an embodiment of this application; Figure 2 A schematic diagram of a second optional process for processing the model provided in an embodiment of this application; Figure 3 A schematic diagram of a third optional process for processing the model provided in an embodiment of this application; Figure 4 A schematic diagram of a fourth optional process for processing the model provided in an embodiment of this application; Figure 5 A fifth optional flowchart illustrating the model processing method provided in the embodiments of this application; Figure 6 A sixth optional flowchart illustrating the model processing method provided in the embodiments of this application; Figure 7 A schematic diagram of a seventh optional process for processing the model provided in the embodiments of this application; Figure 8 An eighth optional flowchart illustrating the model processing method provided in the embodiments of this application; Figure 9 A ninth optional flowchart illustrating the model processing method provided in the embodiments of this application; Figure 10This is a schematic diagram of an optional process for the Token-Rate Policy Algorithm (TR-Policy Algorithm) module provided in the embodiments of this application; Figure 11 This is a schematic diagram of an optional structure of the processing apparatus for the model provided in the embodiments of this application.

[0010] It should be noted that the terms "first" and "second" mentioned above are only used to distinguish between different options and do not represent the degree of superiority or inferiority of the options or their priority in the implementation process. Detailed Implementation

[0011] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the specific technical solutions of the application will be further described in detail below with reference to the accompanying drawings of the embodiments of this application. The following embodiments are used to illustrate this application, but are not intended to limit the scope of this application.

[0012] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0013] In the following description, the terms "first," "second," and "third" are used only to distinguish different objects and do not represent a specific order of objects, nor are they constituting a chronological order. It is understood that "first," "second," and "third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0014] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0015] To facilitate understanding, some technical terms will be explained first.

[0016] Instantaneous Resource Potential (IRP) is a quantitative indicator of the remaining hardware capacity of an electronic device that can be used to run a target model at a given moment. This indicator is obtained by collecting and fusing multiple key hardware parameters (such as CPU instantaneous frequency, memory bandwidth, bus bandwidth, and power margin), and then performing weighted normalization. It characterizes the computing, memory access, and communication potential that an electronic device can dynamically utilize at any given moment. A higher IRP value indicates that the electronic device has stronger hardware resource margins at the current moment, enabling it to support larger-scale batch processing or more aggressive task scheduling strategies.

[0017] Expert Network: In the Hybrid Expert (MoE) architecture, an expert network is a collection of multiple independent sub-models. Each expert network has the same structure but independent parameters and is responsible for processing different parts of the input data. During inference, parts of the expert network are sparsely activated based on the input features, thereby achieving high-capacity and high-efficiency model inference. Different experts may exhibit different resource bottleneck types during network runtime, such as computationally intensive, memory-intensive, or communication-intensive.

[0018] It should be noted that the embodiments of this application can be applied to various inference models, such as models of MoE (Hybrid Expert) architecture or other general inference models.

[0019] To address the aforementioned issues, embodiments of this application provide a model processing method, apparatus, device, storage medium, and computer program product. In practical applications, the model processing method can be implemented by a model processing apparatus, and the functional entities within the model processing apparatus can be collaboratively implemented using the hardware resources of electronic devices, such as processors and other computing and communication resources.

[0020] Firstly, embodiments of this application provide a method for processing a model. The following description uses an electronic device as the executing entity.

[0021] refer to Figure 1 The processing methods for the model shown may include, but are not limited to, those shown. Figure 1 S101 to S104 are shown.

[0022] S101. The electronic device determines the first instantaneous resource potential based on multiple hardware parameters of the electronic device running the target model.

[0023] In this embodiment, the executing entity is an electronic device that runs the target model. Specifically, it can be a server or high-performance computing node equipped with GPU, CPU, memory, and power management electronic devices. For example, the electronic device can be various desktop workstations.

[0024] The first instantaneous resource potential is used to characterize the instantaneous remaining hardware capacity of an electronic device for running a target model. For example, it is calculated based on multiple hardware parameters to determine how much additional "effective computing power margin" the hardware can squeeze out without triggering power overload, bus congestion, or memory bottlenecks.

[0025] Multiple hardware parameters: These refer to multiple dynamic indicators that reflect the underlying physical limits of a device in real time. This application does not limit the content of these multiple hardware parameters, and they can be configured according to actual needs. For example, multiple hardware parameters may include, but are not limited to, the following: the instantaneous operating frequency of the CPU of the electronic device, the instantaneous transfer rate of the memory of the electronic device, the instantaneous bandwidth of the bus of the electronic device, the instantaneous remaining power of the power supply of the electronic device, etc.

[0026] The electronic device acquires multiple key instantaneous data points at high frequency through the underlying driver interface, namely, several hardware parameters: CPU instantaneous frequency (reflecting the current remaining computing clock cycles), memory instantaneous transfer rate (reflecting the remaining data flow rate of video memory / main memory), and bus instantaneous bandwidth (reflecting the remaining channel width for data transfer); simultaneously, it obtains the instantaneous remaining power supply as a hard constraint. Then, based on these multiple hardware parameters, the first instantaneous resource potential is determined.

[0027] For example, electronic devices collect multiple key hardware parameters in real time through built-in system monitoring interfaces (such as Linux's sysfs, NVML, Windows performance counters, etc.). These hardware parameters include, but are not limited to: the instantaneous average operating frequency of the CPU / accelerator core (h_Freq), the instantaneous available bandwidth of the system's DDR memory (h_MemBW), the instantaneous idle throughput potential of the PCIe bus (h_PCIE), and the instantaneous available headroom of the power supply system or thermal design power (TDP) (h_PowerHeadroom). These hardware parameters reflect the dynamic state of the electronic device at the current moment, rather than its static specifications. For example, if the rated power of the power supply is 1200W and the current actual consumption is 720W, then h_PowerHeadroom = (1200-720) / 1200 = 0.4, indicating that the electronic device still has 40% power headroom, which can be used for overclocking or increasing the load. In addition, h_Freq can be the ratio of the current frequency to the maximum frequency. For example, if the current frequency is 1.8GHz and the maximum frequency is 2.0GHz, then h_Freq = 0.9. h_MemBW can be the ratio of current bandwidth to peak bandwidth. For example, if the current bandwidth is 30GB / s and the peak bandwidth is 64GB / s, then h_MemBW = 0.46875. h_PCIE is calculated based on the ratio of the current transfer rate to the PCIe Gen speed limit. For example, if the current rate is 10GB / s and the Gen4 limit is 16GB / s, then h_PCIE = 0.625.

[0028] There is a direct mapping relationship between the first instantaneous resource potential and hardware parameters. For example, when h_Freq increases (e.g., from 0.8 to 0.9) and λ_freq is large, the first instantaneous resource potential will increase significantly, indicating that the system's computing power potential is enhanced; when h_MemBW decreases (e.g., from 0.5 to 0.3), even if other parameters remain unchanged, the first instantaneous resource potential will decrease, reflecting the exacerbation of memory access bottlenecks.

[0029] In a typical high-load scenario where a user is playing a game while triggering the AI ​​super-resolution model, the electronic device collects four instantaneous parameters through the underlying driver: the CPU's current frequency is 2.2GHz (0.8GHz margin), the memory transfer rate is 18GB / s (7GB / s margin), the bus bandwidth is 14GB / s (only 6GB / s margin), and the remaining power consumption is only 2.8W. The electronic device assessment found that its initial instantaneous potential is low.

[0030] The electronic device initiates a hardware parameter acquisition module every scheduling cycle (e.g., every 10ms) to collect and fuse multiple hardware parameters in real time, generating the first instantaneous resource potential value for the current moment. The hardware parameter acquisition module operates through lightweight system calls, minimizing its impact on the target model's inference latency. By constructing a unified first instantaneous resource potential index, the electronic device, for the first time, transforms heterogeneous underlying engineering metrics (such as frequency, bandwidth, and power) into quantifiable inputs understandable to the upper-level scheduler, overcoming the bottleneck of the current lack of hardware potential awareness. This transformation mechanism enables scheduling decisions to move beyond blind conservatism, accurately determining whether the system has the capability to expand batch size or optimize expert allocation based on millisecond-level dynamic hardware changes, providing a scientific basis for subsequent dynamic adjustments.

[0031] S102. The electronic device determines a first adjustment factor for adjusting the batch size of the target model based on the first instantaneous resource potential.

[0032] The first instantaneous resource potential is positively correlated with the first adjustment factor.

[0033] Batch size: In deep learning model inference, it refers to the number of input samples processed in a single forward pass. For example, BatchSize=4 indicates the number of tokens processed by the model in one scheduling cycle.

[0034] First adjustment factor: This is a dimensionless scaling factor (such as 0.5, 1.0, 1.5) used to multiply by the baseline batch size to dynamically adapt to the real-time state of the hardware.

[0035] Positive correlation means that the higher the "first instantaneous resource potential" (the more spare hardware), the larger the adjustment factor, and the larger the batch size the model is allowed to use; conversely, when resources are scarce, the adjustment factor shrinks and the batch size is reduced.

[0036] The electronic device inputs the calculated "first instantaneous resource potential" into a preset mapping function (such as a linear scaling factor, piecewise step function, or lookup table). Since the two are positively correlated, the higher the potential value, the larger the "first adjustment factor" of the mapping output. Subsequently, the electronic device multiplies the model's default baseline batch size (which can be a historical batch size or a fixed batch size set based on experience) by this factor to obtain the dynamic batch size actually used in the current frame or current round. Then, the inference engine packages the input data according to this dynamic value and sends it to the model for forward computation. This mechanism ensures that: when the bus is unobstructed and power is abundant, the parallel computing power of the GPU is fully utilized to improve processing efficiency through high throughput; when the bus is congested or power is tight, it actively adjusts to smaller batches to reduce instantaneous memory usage and computational power consumption, avoiding the electronic device from triggering temperature control throttling or task timeouts due to resource contention.

[0037] Continuing the previous example, the phone's current "initial instantaneous resource potential" is assessed as only supporting a lightweight model (bus margin 6GB / s, power budget 2.8W). The electronic device's default mapping strategy is: a factor of 1.5 when the potential is above 80%, 1.2 when it's between 60% and 80%, and 1.0 when it's below 60%. At this point, the potential assessment value is only 55%, so the output "first adjustment factor" is 1.0, meaning the model maintains the baseline batch size (e.g., Batch Size = 4). However, assuming the user exits the game at this moment, the electronic device detects a sudden increase in bus margin to 18GB / s and a return to 8W of power, with the potential jumping to 90%. The mapping function then outputs an adjustment factor of 1.5, and the model batch size is dynamically adjusted to 6 (4 × 1.5). At this point, the GPU's Tensor Core computing power is fully utilized, increasing overall throughput by 50%, while the device temperature remains controllable due to sufficient power budget, perfectly achieving a positive feedback loop of "the more abundant the resources, the more efficient the execution."

[0038] After acquiring the initial instantaneous resource potential, the adjustment factor calculation module immediately calculates the first adjustment factor according to preset rules and uses it as one of the key decision outputs. The calculation process of the adjustment factor calculation module has low computational complexity, completing within milliseconds, meeting real-time scheduling requirements. By introducing the first adjustment factor, a smooth transition from hardware potential to batch size adjustment is achieved, avoiding potential abrupt changes or oscillations that might occur when directly using the initial instantaneous resource potential value for batch size calculation. For example, when the initial instantaneous resource potential fluctuates slightly, the change in the first adjustment factor is constrained by rules, ensuring the stability of batch size adjustment. Simultaneously, the positive correlation design of the first adjustment factor ensures that electronic devices can actively explore throughput potential in high-potential states, and can promptly shrink when low-potential or latency is critical, effectively balancing the contradiction between throughput and latency.

[0039] S103. The electronic device determines the target batch size of the target model in the current scheduling cycle based on the first adjustment factor.

[0040] Current scheduling cycle: The smallest time slice unit (e.g., 100ms, 1 second, or 10 frames) for electronic device resource evaluation and model configuration updates. Hardware parameters fluctuate at high frequency, and electronic devices do not adjust the model every microsecond. Instead, they follow a fixed cycle of "sampling-decision-execution," which is the basic scheduling beat.

[0041] Target Batch Size: The minimum batch size (e.g., 2, 4, 6, 8) used by the model during the actual inference execution within the current scheduling period. It is the final execution value obtained after rounding and boundary clipping after applying the "first adjustment factor" to the "baseline batch size".

[0042] The electronic device has a built-in timer or frame counter with a defined fixed "scheduling period" (e.g., an update triggered every 200ms or every 15 frames). At the beginning of each scheduling period, the electronic device retrieves the "first adjustment factor" calculated in the previous step and multiplies it by the model's preset "baseline batch size" (e.g., 4) to obtain an initial calculated value. Subsequently, considering GPU memory alignment and hardware operator limitations (e.g., requiring the batch size to be a power of 2 or not exceeding the maximum value of 16), the electronic device performs integer rounding and upper / lower limit pruning on this value, ultimately producing the "target batch size" for that period. Throughout the following period, the inference engine strictly locks this size to execute all inference tasks; until the next period arrives, the electronic device recalculates the potential and refreshes the target batch size based on the latest hardware state, thereby achieving periodic synchronization between model behavior and hardware state.

[0043] S104. Under the current scheduling cycle, the target model of the electronic device performs model inference based on the target batch size to match the remaining hardware capacity of the electronic device.

[0044] Model inference: The process of feeding packaged input data (such as images, text tokens) into a neural network, performing forward propagation calculations, and producing results.

[0045] Matching with remaining hardware capacity: This is a dynamic balance state, not requiring the batch size to exactly "fully" utilize all hardware resources. Rather, it means that within the current cycle, the memory usage, bus throughput pressure, and power consumption requirements of the inference task are strictly constrained within the safe envelope defined by the "first instantaneous resource potential." This manifests as: not triggering memory overflow, not causing prolonged bus congestion (sharp increase in latency), and not causing instantaneous power consumption to exceed the remaining power supply budget (triggering emergency hardware throttling).

[0046] Once the target batch size is locked at the start of the current scheduling cycle, the inference engine uses this fixed size to continuously perform inference calculations on the model within that cycle. Each inference operation consumes CPU clock cycles, memory bandwidth, and power, and the total amount consumed falls precisely within the "potential" budget assessed in the previous stage (e.g., bus utilization is controlled within 80% of remaining bandwidth, and instantaneous power does not exceed the power supply margin). The electronic device hardware performance monitoring module continues to collect parameters without interrupting the execution of the current cycle until its end. When the next cycle begins, the electronic device reassesses the "new remaining hardware capacity" and dynamically refreshes the batch size. This forms a negative feedback loop of "assessment → locking → execution → reassessment," ensuring that the model runs within the hardware's "comfort zone" in the long term, neither wasting computing power nor overdrawing physical limits.

[0047] The current scheduling cycle (200ms) has locked the target batch size at 6, and hardware evaluation shows a bus margin of 18GB / s and a power budget of 8W. Within this cycle, the inference engine executed 5 consecutive model inferences (processing 6 4K frames each time). Verification by the underlying PMU (Power Management Unit) showed that the actual instantaneous peak bus throughput was 11GB / s (far less than the 18GB / s limit), and the actual peak power consumption was 2W (far less than the 8W budget). The entire inference process was smooth and stutter-free, and GPU utilization remained at a healthy level. This illustrates the concept of "matching"—the hardware has ample remaining capacity, the batch size is increased to fully utilize parallel computing power, but the actual consumption is still safely constrained within the potential envelope. After the 200ms cycle ends, assuming the electronic device detects a decrease in battery power leading to a reduction in the remaining power budget to 5W, the batch size will be adjusted back during the next cycle's re-evaluation. This ensures that even with battery aging, the model can continue to run stably through dynamic adaptation, without the electronic device suddenly blacking out and restarting due to power overdraft.

[0048] The data processing method provided in this application embodiment achieves refined dynamic scheduling of target model inference by constructing a closed-loop feedback control system of perception → decision → execution for electronic devices. Specifically, firstly, multiple hardware parameters are collected in real time through a hardware parameter acquisition module to construct a first instantaneous resource potential, quantifying the current remaining hardware capacity of the electronic device; secondly, based on the positive correlation between the first instantaneous resource potential and a first adjustment factor, the first adjustment factor is dynamically calculated to guide batch size adjustment; finally, the target model performs inference based on the target batch size, achieving dynamic matching with the remaining hardware capacity of the electronic device. The data processing method provided in this application embodiment not only breaks through the static assumption of hardware potential in traditional schedulers, but also transforms multiple hardware parameters (such as CPU frequency, memory bandwidth, power margin, etc.) into upper-level scheduling decision factors for the first time, ensuring that while maximizing the Token Rate, the utilization efficiency of hardware resources is improved, and the processing capacity of the model is matched with the hardware potential. The model processing capacity of this embodiment can be matched with the fine-grained hardware resources of the electronic device, rather than a simple and general single-parameter matching. It has a more refined hardware matching granularity, improves hardware utilization, and can further explore utilization efficiency, thereby maximizing processing efficiency.

[0049] The following section explains the process by which the electronic device in S101 determines the resource potential of the first instant based on multiple hardware parameters of the electronic device running the target model.

[0050] In some embodiments, reference Figure 2 The process may include, but is not limited to, S201 and S202 described below.

[0051] S201. Electronic equipment collects multiple hardware parameters of the electronic equipment.

[0052] The multiple hardware parameters include at least one or more of the following: the instantaneous operating frequency of the central processing unit (CPU) of the electronic device, the instantaneous transfer rate of the memory of the electronic device, the instantaneous bandwidth of the bus of the electronic device, and the instantaneous remaining power of the power supply of the electronic device.

[0053] Hardware parameters: These refer to the CPU's current actual operating clock speed (GHz), the data exchange rate between memory and CPU (GB / s), the instantaneous bandwidth of data transmission on the electronic device bus (GB / s), and the remaining power (W) that the battery / power supply can safely output at this time. These four parameters respectively reflect "computing power," "data supply capacity," "data transport capacity," and "energy budget."

[0054] CPU instantaneous operating frequency: The actual clock speed (GHz) of the current CPU core, which determines the instantaneous peak speed of logical operations.

[0055] Instantaneous memory transfer rate: The actual read / write speed (GB / s) at which data is exchanged between RAM and CPU, which determines how fast data is fed to the CPU.

[0056] Bus instantaneous bandwidth: The instantaneous throughput (GB / s) of the current electronic device bus (such as the data channel between the CPU and GPU, peripherals), which determines whether the entire data pipeline is congested.

[0057] Instantaneous remaining power: The “safe extra watts” (W) that the current battery or power supply circuit can still provide instantaneously. This is the physical ceiling for all performance improvements (limited by heat dissipation and discharge capacity).

[0058] Electronic devices first capture raw readings of CPU instantaneous frequency, memory transfer rate, bus bandwidth, and remaining power supply at the same time through underlying hardware monitoring drivers.

[0059] S202, The electronic device performs weighted and normalized processing on multiple hardware parameters to obtain the first instantaneous resource potential.

[0060] Weighting: Each of the four parameters is multiplied by a coefficient between 0 and 1, with a total weight of 1, to express the preference of different tasks for hardware resources (e.g., AI inference relies more on bandwidth, while games rely more on GPU frequency).

[0061] Normalization: Divide physical values ​​with different dimensions (such as 2.2GHz, 18GB / s) by their respective physical upper limits (such as 3.0GHz, 25GB / s) to convert them into a uniform "proportional fraction" between 0 and 1, eliminating unit differences for mathematical calculations.

[0062] First instantaneous resource potential: The four normalized scores are weighted and summed to obtain a comprehensive score between 0 and 1, which represents the current overall remaining safety margin of the hardware.

[0063] Because these four values ​​have completely different dimensions (GHz, GB / s, W), they cannot be directly added. The electronic device then calls a preset normalization function: dividing each measured value by its hardware-calibrated physical peak value (e.g., current CPU 2.2GHz ÷ peak 3.0GHz = 0.73), uniformly mapping it to a dimensionless proportional value between 0 and 1. This step transforms all complex physical quantities into a "percentage of current performance used," clearing the dimension barrier for subsequent fusion.

[0064] After normalization, the electronic device retrieves the corresponding weight matrix from a preset strategy library based on the type of the target model currently running (such as an AI super-resolution model or video codec). For example, for high-throughput AI tasks, a high weight of 0.4 is assigned to the bus bandwidth, while a low weight of only 0.1 is assigned to the CPU. Subsequently, the electronic device multiplies the four normalized scale values ​​by their respective weights and sums them up to obtain a weighted composite score between 0 and 1. This score is the "first instantaneous resource potential," which strips away all physical units and serves as the sole mathematical input for subsequent calculations of the batch size adjustment factor.

[0065] The effectiveness of this technology lies in fusing multidimensional hardware signals with vastly different physical meanings and dimensions into a stable, continuous, and unitless single quantitative indicator (potential score). This not only eliminates the magnitude differences between various hardware dimensions, allowing subtle fluctuations in CPU frequency and drastic changes in bus bandwidth to be compared fairly under the same "ruler," but also enables resource evaluation logic to precisely adapt to the hardware preferences of different tasks by dynamically adjusting the weight matrix (making AI inference focus more on bandwidth bottlenecks and games focus more on GPU collaboration). This provides upper-level scheduling algorithms with a precise decision-making basis that reflects both the "physical baseline (power)" and the "characteristics of the current task."

[0066] The formula for calculating this instantaneous resource potential index is: Pt = Σ n i=1 λ i ·Normalize(h i ).

[0067] Among them, h i Represents the i-th hardware metric (such as h_Freq, h_MemBW, etc.), Normalize(h i ) represents normalizing the i-th hardware metric to the interval [0,1] (e.g., (current value - minimum value) / (maximum value - minimum value)), λ iLet be the weight coefficient of the corresponding indicator, and satisfy Σλ i =1.

[0068] Weighting coefficient λ i The setup method is as follows: 1. Benchmarking: Under a fixed batch size, artificially limit various hardware indicators using system tools (such as reducing the instantaneous operating frequency of the CPU of the electronic device or the instantaneous transfer rate of the memory of the electronic device). 2. Sensitivity Calculation: Measure Token Rate T for each hardware metric h. i Partial derivatives | T / h i |, that is, the sensitivity; 3. Weight Normalization: Use sensitivity as the initial weight and normalize the initial weight, λ. i =Sensitivity i / Σ n j=1 Sensitivity j .

[0069] For example, in a MoE inference scenario, if it is found that the instantaneous transfer rate parameter of the electronic device's memory has the greatest impact on the Token Rate (highest sensitivity), then h_MemBW is given a higher weight λ_mem; conversely, if the instantaneous operating frequency parameter of the electronic device's CPU has a greater impact, then λ_freq is increased. This quantitative analysis based on model and hardware bottlenecks ensures that the Resource Potential Assessment (IRP) metric accurately reflects the current system's effective resource potential for that specific model.

[0070] In practical implementation, assuming the collected values ​​are h_Freq=0.85, h_MemBW=0.78, h_PCIE=0.62, and h_PowerHeadroom=0.4, after normalization, they are denoted as H1=0.85, H2=0.78, H3=0.62, and H4=0.4 respectively. Then, according to the preset weights λ=[0.3, 0.4, 0.2, 0.1], Pt=0.3×0.85+0.4×0.78+0.2×0.62+0.1×0.4=0.747.

[0071] The process of determining the target batch size of the target model in the current scheduling cycle based on the first adjustment factor in S103 is described below.

[0072] In some embodiments, reference Figure 3 The process may include, but is not limited to, S301 to S304 described below.

[0073] S301. The electronic device determines the first batch size by multiplying the first adjustment factor by the historical batch size of the target model.

[0074] Historical batch size: The actual batch size value used in the previous scheduling cycle, which serves as the "baseline anchor point" for the current adjustment to ensure the continuity of the adjustment.

[0075] First batch size: The theoretically calculated value obtained by multiplying the adjustment factor by the historical batch size, without any hardware boundary verification, may exceed the hardware's tolerance.

[0076] The electronic device first reads the historical batch size (e.g., locked at 4 in the previous cycle) from the memory cache, which was actually executed and recorded in the previous scheduling cycle. Then, it multiplies the "first adjustment factor" (e.g., 1.5) calculated in the previous step with this historical batch size to obtain a purely theoretical value without any physical constraints. This product is defined as the "first batch size," which mathematically represents "the theoretical scaling range of the batch size under the current hardware potential variation trend." However, it does not consider hard physical limitations such as memory capacity and operator alignment; it is purely a quantitative mapping result based on resource potential.

[0077] S302. Obtain the maximum and minimum batch sizes that the electronic device can support.

[0078] Maximum / Minimum Batch Size: Hard boundaries determined by hardware memory capacity, GPU / NPU operator alignment requirements (such as requiring a power of 2), or limitations of the model itself. Exceeding the batch size will cause inference to fail.

[0079] The second batch of dimensions: feasible predicted values ​​obtained by substituting theoretical values ​​into hard boundaries and performing "amplitude clipping", which have been ensured to fall within the hardware safety range.

[0080] Before performing any numerical pruning, the electronic device sends a query request to the underlying hardware abstraction layer or model configuration manager to obtain the "maximum batch size" and "minimum batch size" that the current electronic device can support in the existing operating environment. The maximum batch size is typically determined by the available video memory size and the parallel computing depth of the GPU / NPU (exceeding this limit will trigger a memory overflow crash); the minimum batch size is determined by the statistical validity of the model's batch normalization layer or the minimum parallel granularity requirement of the operator library (below this value will cause a sharp deterioration in operator execution efficiency). These two boundary values ​​are the absolute "hard benchmarks" for all subsequent pruning operations.

[0081] S303. Based on the first batch size, the largest batch size, and the smallest batch size, the electronic device predicts the second batch size that the target model can process in the current scheduling cycle.

[0082] The electronic device substitutes the "first batch size" calculated in the first step into the closed interval defined by the "maximum batch size" and "minimum batch size" obtained in the second step, and performs a hard clipping operation. The specific logic is as follows: if the first batch size is greater than the maximum value, it is forcibly truncated to the maximum value; if it is less than the minimum value, it is forcibly increased to the minimum value; if it is exactly within the interval, the original value is retained. The value output after this clipping operation is defined as the "second batch size." This value is ensured to fall 100% within the hardware physical limits, theoretically eliminating the risk of inference crashes due to memory overruns or operator mismatches. However, at this point, the potential for excessive changes between adjacent cycles has not yet been considered.

[0083] S304. The target batch size for electronic equipment shall be determined based on at least the second batch size.

[0084] Target batch size: The final execution value locked in the current scheduling cycle after applying additional smoothing strategies (such as amplitude limiting step size, jitter filtering) to the feasible prediction value.

[0085] After obtaining a physically feasible "second batch size," the electronic device can directly lock it as the final execution value. Alternatively, it can determine the target batch size based on the second batch size, taking into account other influencing factors such as expert load balancing (Imbalance) and the current request queue length, forming a multi-layered decision-making loop. In the MoE architecture, the final value of the target batch size not only affects throughput but also indirectly determines the expert allocation strategy. t This is because larger batch sizes typically mean higher parallelism, requiring more appropriate expert mapping to avoid resource contention.

[0086] This process maintains the temporal continuity of resource scheduling through continuous recursion of "historical benchmark multiplication factor," ensures that each decision does not exceed the hardware limits through "maximum / minimum hard threshold," and filters out erroneous adjustments caused by instantaneous spikes and glitches in hardware parameters through "smoothing filtering." The combined effect of these three mechanisms makes the batch size adjustment both agile in response to macroscopic changes in the remaining hardware capacity (such as a steady increase after exiting a large game) and resilient in resisting microscopic noise (such as millisecond-level random fluctuations in CPU frequency). Ultimately, it achieves a coupling relationship between model inference load and hardware physical state that is both closely followed and smoothly transitioned.

[0087] Using the scenario after exiting the game on a mobile phone: The historical batch size in the previous scheduling cycle (200ms ago) was 4. The current hardware potential leap causes the adjustment factor to be calculated as 1.5. The electronic device first calculates the first batch size = 4 × 1.5 = 6. Then, it checks the hardware configuration table and finds that the absolute maximum batch size supported by the current video memory capacity is 16, and the minimum is 1. Since 6 is between 1 and 16, the second batch size is determined to be 6. Considering that if it is locked directly at 6, it will only increase by 2 (an increase of 50%) compared to 4 in the previous cycle, within the step size threshold, the electronic device will not impose additional speed limits, and the final target batch size will be locked at 6. However, assuming that the hardware potential suddenly changes to a factor of 3.0, the first batch size = 4 × 3.0 = 12. Although it does not exceed the maximum boundary of 16, the electronic device detects that the single jump from 4 to 12 is 8 (an increase of 200%), which exceeds the preset single maximum step size limit (such as the limit of 4). Therefore, the smoothing strategy intervenes, and the target batch size for this cycle is temporarily set to 8 (4+4), and the next cycle will decide whether to continue to climb to 12 based on the latest potential. This operation effectively prevents excessive increases in batch size due to instantaneous bus bandwidth recovery (hardware spike), avoids emergency frequency reduction caused by instantaneous power consumption spikes, and ensures "gradual" resource scheduling.

[0088] The process of determining the target batch size of the electronic equipment in S304 is described below, based at least on the second batch size.

[0089] In some embodiments, reference Figure 4 The process may include, but is not limited to, S401 to S403 described below.

[0090] S401, The electronic device obtains the set maximum acceptable reference latency for the user and the current latency of the target model operation.

[0091] The maximum acceptable reference latency for users: The experience threshold preset by developers or users (such as a single inference not exceeding 50ms) is the "red line" to ensure a smooth experience and the ultimate goal of resource scheduling.

[0092] Current latency: The actual inference time (e.g., 30ms) measured by the model within the current scheduling period, reflecting the real computational pressure under the current load.

[0093] The electronic device first reads the preset "maximum acceptable reference latency for the user" (e.g., 50ms for real-time video super-resolution scenarios) from the user configuration or application scenario definition. This value serves as the absolute benchmark for judging whether the current inference speed meets the target. Simultaneously, the electronic device collects the actual time consumed by the target model to complete a single inference within the current scheduling cycle using a timer, recording it as the "current latency" (e.g., measured at 30ms). These two values ​​represent the "expected target" and "actual performance," respectively, and are the raw inputs for all subsequent latency-related decisions.

[0094] S402. The electronic device determines the delay margin based on the reference delay and the current delay.

[0095] Delay margin is used to characterize the time delay margin during the runtime of the target model.

[0096] Delay margin S t The difference between the reference delay and the current delay, divided by the reference delay, indicates that the system is running faster than required and has sufficient capacity. A negative value indicates that the system has timed out and must urgently reduce load. Where S... t Larger size → Sufficient delay margin → Allows for aggressive strategies; S t The smaller the value, the closer the latency will be to the upper limit, thus requiring a conservative approach.

[0097] Electronic devices calculate the "delay margin" by subtracting the current delay from the reference delay and then dividing by the reference delay.

[0098] S403. The electronic device determines the target batch size based on the delay margin, the delay margin correction function, the first degree, and the maximum batch size.

[0099] The delay margin correction function is used to correct the batch size based on the delay margin, the first degree being the aggressiveness of utilizing the hardware potential of the electronic device.

[0100] Delay margin correction function: A mathematical transformation rule (such as linear scaling, PID control, or step-piece function) that maps delay margin to batch size adjustment, used to determine "how much faster should be achieved, and how much larger the batch size should be." Delay margin correction function f(S) t ) is a nonlinear mapping function used to transform delay margin into an excitation or inhibition factor for batch size adjustment. The typical form of this function is f(S) t = tanh(k_learned · (S t - S0_learned)), where k_learned is the amplification factor, and S0_learned is the initial state value learned. The delay margin correction function has the following property: when the delay margin is large, f(S tWhen the delay margin approaches +1, it encourages the system to increase the batch size; when the delay margin is close to or less than 0, f(S) approaches +1, encouraging the system to increase the batch size. t When the value approaches -1, the system is forced to shrink the batch size. In this function, f(S) t ) represents the output value of the delay margin correction function.

[0101] First level (aggressiveness): A coefficient between 0 and 1, representing how aggressively electronic devices utilize hardware potential to catch up in performance. The higher the value, the more willing the electronic device is to significantly increase batch size to squeeze out computing power when there is a margin for latency; the lower the value, the more conservative the adjustment, and the more inclined to maintain stable latency.

[0102] The Aggressiveness Factor (η) is a parameter that indicates how aggressively the control system utilizes hardware potential, reflecting the scheduler's tendency to trade off between increasing throughput and ensuring latency. A larger value for Aggressiveness Factor, η indicates a more aggressive system adjustment, allowing for larger batch size increases within the same latency margin; a smaller value indicates a more conservative system, prioritizing latency stability. The Aggressiveness Factor can be dynamically configured based on application scenarios, user preferences, or system load. For example, a smaller Aggressiveness Factor, η can be set during peak hours to ensure service quality, while a larger value can be set during off-peak hours to maximize resource utilization.

[0103] The electronic device substitutes the calculated latency margin into a preset "latency margin correction function" (e.g., a linear function: adjustment amount = margin / single frame latency baseline), converting it into a recommended change in batch size (positive numbers represent an increase, negative numbers represent a decrease). Subsequently, the electronic device introduces a "first degree" (aggressiveness) to scale this change—if the aggressiveness is 1.0, the correction function's recommendation is followed completely; if it's 0.5, only half the recommended magnitude is executed, approximating the target batch size conservatively. Finally, the electronic device uses the maximum batch size supported by the hardware as an upper bound, confining the corrected recommended value within this boundary, and combines it with previously generated batch size candidate values ​​based on "hardware potential" (taking the intersection or weighted fusion) to ultimately determine the target batch size for the current scheduling cycle.

[0104] Target batch size B t This is the final batch size value, target batch size B. t From the second batch of size B t pred After further optimization using the potential perception safety correction algorithm, the formula is B. t= min(B_max, B t pred + η · f(S t The process of determining the target batch size introduces a critical delay margin S. t = (L_target - L_current) / L_target, making batch size adjustment not only dependent on hardware potential but also strictly controlled by the user-defined latency constraint L_target. When S t When the value is large (with sufficient delay margin), f(S) t The value approaches +1, allowing the target batch size B. t Aggressive growth; when S t When the value is close to 0 or negative (delay critical), f(S) t When the value approaches -1, the target batch size B is forcibly compressed. t To ensure service quality.

[0105] Furthermore, determining the target batch size may also involve other factors, such as expert load balancing imbalance and the current request queue length, forming a multi-layered decision-making loop. In the MoE architecture, the final value of the target batch size not only affects throughput but also indirectly determines the expert allocation strategy. t Because larger batch sizes typically mean higher parallelism, requiring more appropriate expert mapping. The first batch size is obtained by multiplying the first adjustment factor by the historical batch size. The second batch size is then predicted by combining the largest and smallest batch sizes. Finally, the target batch size is adjusted based on the second batch size and latency margin. This enables hardware-potential-driven dynamic batch processing, balancing throughput improvement and latency assurance, thereby maximizing the model's inference efficiency within the current scheduling cycle. The following section explains the process of model inference based on the target batch size in S104.

[0106] When the target model is a MoE (Hybrid Expert) architecture, the target model includes multiple expert networks. (Reference) Figure 5 The process may include, but is not limited to, S501 to S504 described below.

[0107] S501, The electronic device obtains the operating load rate of each expert network of the target model in the previous scheduling cycle.

[0108] An expert network is an independent sub-model in a MoE (Hybrid Expert) architecture. Each expert network has the same structure but independent parameters and is responsible for processing a portion of the input data. The number of expert networks is fixed during the model design phase, such as 8 or 16 expert networks. The runtime load rate refers to the degree of computational, memory, and communication resource usage of each expert network in the previous scheduling cycle, reflecting the workload of that expert network. The runtime load rate is collected in real-time by hardware performance analysis tools and normalized to the [0,1] interval to evaluate the working status of the expert networks. In actual implementation, electronic devices periodically collect metrics such as FLOPs (floating-point operations), parameter size, activation value size, and data transmission volume of each expert network, and calculate the resource usage ratio of that expert network to obtain the runtime load rate. For example, in a MoE model containing 8 expert networks, if the FLOPs of a certain expert network account for 15% of the total FLOPs, then the computational load rate of that expert network is 0.15. This dynamic collection mechanism allows the scheduler to monitor the operating status of each expert network in real time, providing a basis for subsequent load balancing and task allocation.

[0109] S502. The electronic equipment determines the load imbalance of the expert network based on the operating load rate of each expert network in the previous scheduling cycle.

[0110] Load imbalance is a quantitative indicator that measures the uniformity of load distribution among expert networks in the MoE model. It is used to identify whether some expert networks are overloaded while others are idle.

[0111] The calculation method is as follows: First, calculate the average load rate (Mean_Load) of each expert network. Then, calculate the variance (Var_expert) between the load rate of each expert network and the average. Finally, divide the variance by the average to obtain the load imbalance degree (Imbalance = Var_expert / Mean_Load). The load imbalance degree typically ranges from 0 to 1, with a smaller value indicating a more balanced load. For example, when the load rates of the expert networks are [0.7, 0.8, 0.75, 0.72, 0.68, 0.78, 0.71, 0.65], the average load rate is 0.725, and the variance is 0.048. Therefore, the load imbalance degree is 0.066, which is considered a well-balanced state. In practical applications, electronic devices set thresholds to determine the load imbalance status of each expert network: Imbalance < 0.2 indicates good load imbalance, 0.2 ≤ Imbalance < 0.4 indicates slight imbalance, and Imbalance ≥ 0.4 indicates severe imbalance. By using load imbalance data, the scheduler can promptly detect abnormal loads in expert networks and take corresponding adjustment measures to prevent overall inference performance degradation due to overload of individual expert networks.

[0112] S503. The electronic device determines a second adjustment factor for adjusting the expert network allocation strategy, based at least on the first instantaneous resource potential and load imbalance.

[0113] The second adjustment factor ΔΠ is a quantitative parameter used to guide the adjustment of expert network allocation strategies. The value range of the second adjustment factor ΔΠ is [-1, 1], where a positive number indicates that the number of expert networks on high-potential units needs to be increased, and a negative number indicates that the number needs to be decreased. The determination of the second adjustment factor ΔΠ comprehensively considers two key factors: the first instantaneous resource potential Pt and the load imbalance degree Imbalance.

[0114] The specific strategy is as follows: When the Instantaneous Resource Potential (IRP) Pt ≥ 0.7 and the Imbalance ≥ 0.3, the second adjustment factor ΔΠ = +0.5 + 0.3 × (Pt - 0.7), indicating that under conditions of high hardware potential and uneven load, the expert network should be actively migrated to high-potential units; when Pt ≥ 0.5 and Imbalance ≥ 0.4, ΔΠ = +0.3, indicating moderate adjustment under conditions of medium potential and severe uneven load; when Pt < 0.4, ΔΠ = -0.4 - 0.5 × (0.4 - Pt), indicating contraction to core units under conditions of low potential; when latency exceeds the limit, ΔΠ = -0.2, indicating conservative contraction; otherwise, ΔΠ = 0, maintaining the status quo. For example, with Pt=0.85 and Imbalance=0.066, since the load imbalance Imbalance is less than 0.2, ΔΠ=0, and no adjustment is needed. The design of the second adjustment factor ΔΠ realizes the dynamic coupling of hardware potential and load status, enabling the expert network allocation strategy to be flexibly adjusted according to the real-time status of electronic devices.

[0115] S504. The electronic device adjusts the assigned tasks of each expert network based on the second adjustment factor so as to achieve model inference through the adjusted expert networks.

[0116] This step performs specific expert network allocation and adjustment operations, driving expert network migration based on the second adjustment factor ΔΠ. First, it identifies the set of computing units with high instantaneous resource potential, U_HighPoten, which are GPU or CPU cores whose instantaneous resource potential is higher than the average. Then, it performs allocation based on the expert network type (compute-intensive or memory-intensive) and the difference in computing unit potential: compute-intensive expert networks are preferentially allocated to high-frequency potential units, and memory-intensive expert networks are preferentially allocated to high-bandwidth potential units. When ΔΠ > 0, N_move = ΔΠ × the total number of expert networks in the target model are migrated to high instantaneous resource potential units; when ΔΠ ≤ 0, conservative or distributed load allocation is performed. For example, when ΔΠ = +0.5 and the total number of expert networks in the target model = 8, 4 expert networks need to be migrated. The electronic device analyzes the resource requirement vector R of each expert network. j =[r_comp,r_mem,r_comm], migrates computationally intensive expert networks to high-computing-power units and memory-intensive expert networks to high-bandwidth units, thereby optimizing resource utilization efficiency. Through this potential-aware dynamic load balancing, the memory access / communication bottleneck in MoE model inference is effectively alleviated, and the overall token rate is improved.

[0117] In this embodiment, the load imbalance of each expert network is determined by obtaining the operating load rate of each expert network in the previous scheduling cycle. A second adjustment factor is then determined based on the first instantaneous resource potential, and finally, the task allocation of each expert network is adjusted based on this second adjustment factor. This allows for real-time perception and dynamic adjustment of the load status of each expert network in the MoE model, enabling more precise matching of the instantaneous resource potential of the hardware electronic devices. This maximizes the token rate of model inference while ensuring the stability of the electronic devices and user latency constraints.

[0118] The following describes the process by which electronic devices in S503 determine a second adjustment factor for adjusting the expert network allocation strategy, based at least on the first instantaneous resource potential and load imbalance.

[0119] In some embodiments, reference Figure 6 The process may include, but is not limited to, at least one of S601 to S603 described below.

[0120] S601. When the resource potential is within the first range at the first instant and the load imbalance is greater than or equal to the second range, the electronic device determines the second adjustment factor based on the first strategy.

[0121] The first range of instantaneous resource potential refers to the numerical interval of the instantaneous resource potential. This interval represents a high resource potential state for the hybrid expert model inference system hardware, typically corresponding to a region with a high instantaneous resource potential value, such as ≥0.7. In this state, the hybrid expert model inference system hardware possesses significant margin in computation, memory access, and communication capabilities, enabling more aggressive scheduling decisions to improve inference throughput. The setting of the first range of instantaneous resource potential is based on a quantitative analysis of the performance bottlenecks of the hybrid expert model inference system hardware, ensuring full utilization of its potential without triggering latency exceedance risks under high potential conditions.

[0122] The second range of load imbalance refers to the threshold range of load imbalance, which represents a significant difference in load distribution among expert networks, for example, a load imbalance ≥ 0.3. When the load imbalance reaches or exceeds this threshold range, it indicates that some expert networks are overloaded while others are idle, leading to a decrease in overall inference efficiency. In this case, it is necessary to optimize the load distribution by adjusting the expert network allocation strategy.

[0123] The first strategy used to determine the second adjustment factor is an aggressive adjustment strategy. Its core is to proactively increase the batch size and enhance the expert network task migration under the dual conditions of high resource potential and high load imbalance, thereby maximizing the system's inference throughput. Specifically, when the first instantaneous resource potential ≥ 0.7 and the load imbalance ≥ 0.3, the second adjustment factor = +0.5 + 0.3 × (first instantaneous resource potential - 0.7). That is, as the first instantaneous resource potential further increases, the expert network task migration amplitude increases linearly, thereby directing more tasks to high-potential units.

[0124] In real-world scenarios, such as during hybrid expert model inference, when the GPU core frequency is detected to be stable above 90% (GPU core frequency = 0.9), DDR bandwidth utilization is 60% (DDR bandwidth utilization = 0.6), and power supply margin is 40% (power supply margin = 0.4), the calculated first instantaneous resource potential is 0.85 (belonging to the first range of the first instantaneous resource potential). At the same time, the expert load variance is relatively large (expert load variance = 0.08), and the calculated load imbalance is 0.11 (if the calculated load imbalance value should be above 0.4 to trigger, it should be corrected to load imbalance ≥ 0.4). Then, the hybrid expert model inference system activates the first strategy to determine the second adjustment factor, setting the second adjustment factor to +0.55. This guides the hybrid expert model inference system to migrate some expert network tasks to high-frequency, high-bandwidth cores, thereby alleviating local bottlenecks and improving overall inference throughput.

[0125] Thus, the second adjustment factor is determined by employing a first strategy for determining the second adjustment factor under conditions of high initial instantaneous resource potential and high load imbalance. This allows for full utilization of the instantaneous performance margin of the hybrid expert model inference system hardware, enabling greater migration and batch scaling of expert network tasks, thereby significantly improving the inference throughput of the hybrid expert model.

[0126] S602. If the resource potential is in the third range and the load imbalance is in the fourth range at the first instant, the electronic device determines the second adjustment factor as the first reference value.

[0127] Among them, the first instantaneous resource potential of the first range is greater than that of the third range, and the load imbalance of the fourth range is greater than that of the second range.

[0128] The third range of the first instantaneous resource potential refers to the low to medium range of the first instantaneous resource potential, typically corresponding to a value between 0.5 and 0.7. This indicates that the hardware resource potential of the hybrid expert model inference system is at a moderate level, insufficient to support aggressive scheduling, but still has some room for optimization. If the third range of the first instantaneous resource potential is lower than the first range, it means that the available hardware resources of the hybrid expert model inference system are reduced, and the scheduling strategy needs to be more cautious.

[0129] The fourth range of load imbalance refers to the higher interval of load imbalance. For example, a load imbalance ≥ 0.4 indicates a severe imbalance in the expert network load, which may cause some expert networks to become system bottlenecks, seriously affecting the overall inference speed. If the fourth range of load imbalance is higher than the second range, it means the problem is more serious and requires immediate intervention.

[0130] The preset first reference value is a fixed value, such as a second adjustment factor of +0.3, used to perform appropriate adjustments to expert network tasks under the combination of moderate first instantaneous resource potential and severe load imbalance. The preset first reference value avoids latency exceeding limits due to excessive aggressiveness while ensuring a basic response to load imbalance issues.

[0131] For example, when the system detects that the first instantaneous resource potential is 0.6 (which falls within the third range of the first instantaneous resource potential), and the expert load variance increases significantly (expert load variance = 0.12), the calculated load imbalance is 0.16 (if corrected to load imbalance ≥ 0.4), then the system determines that it has entered the fourth range of load imbalance. At this time, the second adjustment factor is directly set to the preset first reference value + 0.3, and a medium-intensity expert network task migration operation is initiated to transfer some tasks from the high-load expert network to the low-load expert network in order to balance the overall load.

[0132] In this way, by setting the second adjustment factor to a preset first reference value under conditions of moderate first instantaneous resource potential and severe load imbalance, overly aggressive strategies due to insufficient resources can be avoided, thereby achieving safe and effective load balancing adjustments and maintaining the stability and inference efficiency of the hybrid expert model inference system.

[0133] S603, when the resource potential in the first instant is based on the fifth range, the electronic device determines the second adjustment factor based on the second strategy.

[0134] The first instantaneous resource potential of the fifth range is less than the first instantaneous resource potential of the third range, and the adjustment intensity of the second strategy is less than the adjustment intensity of the first strategy.

[0135] The fifth range of the first instantaneous resource potential refers to the lowest range of the first instantaneous resource potential, typically corresponding to a state where the first instantaneous resource potential value is less than 0.5. This indicates that the hardware resources of the hybrid expert model inference system are strained, and it may face problems such as insufficient power supply, frequency degradation, or bandwidth saturation. If the fifth range of the first instantaneous resource potential is lower than the third range, it means that the hardware of the hybrid expert model inference system is in a low-potential state, and a conservative strategy must be adopted.

[0136] The second strategy used to determine the second adjustment factor is a conservative adjustment strategy. The core of this strategy is to prioritize the stability of the hybrid expert model inference system under low first instantaneous resource potential, avoiding latency exceedances or system crashes due to improper scheduling. Specifically, when the first instantaneous resource potential is <0.4, the second adjustment factor = -0.4 - 0.5 × (0.4 - first instantaneous resource potential). That is, as the first instantaneous resource potential further decreases, the migration magnitude of expert network tasks negatively increases, prompting the hybrid expert model inference system to shrink tasks to core high-performance units, reducing its dependence on edge computing units.

[0137] Adjustment intensity refers to the degree of influence of the strategy on the second adjustment factor. The first strategy used to determine the second adjustment factor has a large adjustment intensity (e.g., the second adjustment factor can reach +0.55), while the second strategy used to determine the second adjustment factor has a small adjustment intensity (e.g., the second adjustment factor can be as small as -0.6). The contrast between the two strategies reflects the differentiated scheduling logic under different system states.

[0138] In real-world scenarios, for example, when the CPU core frequency drops to 70% (CPU core frequency = 0.7) due to heat dissipation limitations, memory bandwidth is used up to 90% (DDR bandwidth utilization = 0.1), and power supply margin is only 10% (power supply margin = 0.1), the calculated first instantaneous resource potential is 0.3 (which falls within the fifth range of the first instantaneous resource potential). At this point, the system initiates a second strategy to determine the second adjustment factor, setting the second adjustment factor to -0.5. This forces some expert network tasks to migrate from the edge core back to the main core, reducing the overall load and preventing system overload.

[0139] In this way, the second adjustment factor is determined based on the second strategy used to determine the second adjustment factor under the low first instantaneous resource potential state. This effectively avoids the overload risk of the hybrid expert model inference system, thereby ensuring that the user-set maximum latency constraint L_target is not exceeded, and thus enabling a safe and reliable hybrid expert model inference service.

[0140] Explanation of relationships: The first, third, and fifth ranges of the first instantaneous resource potential constitute a three-level division of the first instantaneous resource potential, decreasing sequentially from high to low, corresponding to high, medium, and low hardware potential states of the hybrid expert model inference system, respectively. The second and fourth ranges of load imbalance constitute a two-level division of load imbalance, increasing sequentially from low to high, corresponding to mild / moderate and severe load imbalance situations, respectively. The first and second strategies for determining the second adjustment factor provide differentiated adjustment schemes ranging from aggressive to conservative based on the combination of the hardware potential state and the load imbalance situation. These three-level divisions, two-level divisions, and differentiated adjustment schemes together constitute a complete dynamic scheduling decision framework, realizing a closed-loop control logic of hardware state perception → load status assessment → strategy strength matching for the hybrid expert model inference system.

[0141] In practical implementation, the hybrid expert model inference system collects real-time data on the first instantaneous resource potential and load imbalance indicators, determines the current state based on preset range boundaries, and calls the corresponding policy function to calculate the second adjustment factor, ultimately guiding the expert network's allocation strategy adjustment. This multi-level hierarchical and policy linkage mechanism enables the scheduling system to accurately adapt to the complex and ever-changing hardware operating environment of the hybrid expert model inference system, fully leveraging the performance potential of the hybrid expert model architecture.

[0142] In this embodiment, the second adjustment factor is determined by employing a first strategy, setting a preset first reference value, or a second strategy for determining the second adjustment factor under different combinations of first instantaneous resource potential and load imbalance. This allows for refined perception and response to the state of the hybrid expert model inference system, enabling dynamic adjustment of the expert network allocation strategy. This maximizes the inference throughput of the hybrid expert model while strictly ensuring user latency constraints, thereby improving overall inference efficiency and the stability of the hybrid expert model inference system.

[0143] In other embodiments, the second adjustment factor can also be determined by combining the delay margin.

[0144] In some embodiments, reference Figure 7 The process may include, but is not limited to, S701 and S702 described below.

[0145] S701, Electronic equipment determines the delay margin for the target model operation.

[0146] Delay margin is used to characterize the time delay margin during the runtime of the target model.

[0147] Delay margin refers to the ratio of the difference between the current actual inference delay and the target delay, expressed by the formula S. t =(L_target - L_current) / L_target, where L_target is the target delay (reference delay), and L_current is the actual measured end-to-end inference delay within the current scheduling period. Delay margin reflects how much safety margin the system has to improve throughput while meeting user latency constraints. For example, when L_target = 200ms and L_current = 60ms, S... t =(200-60) / 200=0.7, indicating that the system has sufficient delay margin and can aggressively increase the Batch Size to improve the TokenRate; while when L_current approaches or exceeds L_target, S t When the value is close to 0 or negative, the batch size must be conservatively reduced to avoid exceeding latency limits. Latency margin is a key bridge connecting the initial instantaneous resource potential perception with the user's quality of service assurance. Latency margin enables scheduling decisions to not only depend on hardware status but also respond in real time to changes in user latency requirements, achieving dynamic balance.

[0148] S702, the electronic device determines the second adjustment factor based on the first instantaneous resource potential, load imbalance, and delay margin.

[0149] The second adjustment factor is a quantitative parameter used to guide the dynamic adjustment of expert network allocation strategies. The value range of the second adjustment factor is typically [-1, 1]. A positive value indicates that expert allocation to high-potential units should be increased (diffusion strategy), a negative value indicates that allocation should be reduced (contraction strategy), and zero indicates maintaining the status quo. The determination of the second adjustment factor comprehensively considers three key dimensions: first instantaneous resource potential, load imbalance, and latency margin. Specifically, when the resource scheduling system is at high resource potential (first instantaneous resource potential ≥ 0.7) and the expert load is severely uneven (load imbalance ≥ 0.4), the resource scheduling system tends to migrate more experts to high-potential computing units. In this case, the second adjustment factor takes a positive value and increases with the increase of the first instantaneous resource potential. When the resource scheduling system is resource-constrained (first instantaneous resource potential < 0.4) or latency is critical (latency margin < 0.1), the resource scheduling system adopts a conservative strategy, reducing the second adjustment factor or even making it negative to alleviate the pressure on high-load units and ensure stability. In addition, the calculation process of the second adjustment factor also needs to combine the matching degree between expert type (such as compute-intensive or memory-intensive) and computing unit potential to achieve more granular resource optimization.

[0150] In practical applications, the calculation of the second adjustment factor employs a rule-based adaptive strategy, rapidly outputting results within millisecond-level scheduling cycles. For example, in a typical scenario, if the first instantaneous resource potential = 0.85, load imbalance = 0.066 (good load balancing), and latency margin = 0.7, then according to the rules, no adjustment to the expert network allocation strategy is needed, and the second adjustment factor = 0. In another scenario, if the first instantaneous resource potential = 0.6, load imbalance = 0.5, and latency margin = 0.3, then a moderate adjustment may be triggered, and the second adjustment factor = +0.3, guiding some experts to migrate to high-bandwidth or high-frequency computing units. This multi-factor fusion decision-making mechanism significantly improves the flexibility and robustness of the MoE inference system.

[0151] In this embodiment, by introducing the key indicator of latency margin, and using latency margin, along with the first instantaneous resource potential and load imbalance, as the decision basis for the second adjustment factor, a refined perception and utilization of the first instantaneous resource potential can be achieved. This allows for dynamic adjustment of the expert network allocation strategy without exceeding user latency constraints, thereby effectively alleviating memory access / communication bottlenecks in the MoE inference system, maximizing the token rate, and improving overall inference efficiency.

[0152] The following describes the process by which the electronic device in S504 adjusts the task allocation of each expert network based on the second adjustment factor.

[0153] In some embodiments, reference Figure 8 The process may include, but is not limited to, S801 to S804 described below.

[0154] S801, Electronic device: Determine the second instantaneous resource potential of each of the multiple computing units in the electronic device.

[0155] The second instantaneous resource potential is used to characterize the remaining hardware capacity of the computing unit for running expert networks.

[0156] The second instantaneous resource potential refers to the remaining hardware resources available to a computing unit (such as a GPU core or CPU core) for running expert networks at the current moment. This remaining capacity is dynamic and can be sensed and quantified in real time. This remaining capacity comprehensively reflects the instantaneous available potential of the computing unit in terms of frequency, memory bandwidth, communication bandwidth, and power supply margin, and is used to guide scheduling decisions. The second instantaneous resource potential is obtained by collecting low-level indicators such as the instantaneous frequency of the computing unit, DDR / PCIE bus bandwidth, and power supply margin, and then standardizing and weighting the sum. The formula is P. t =Σ n i=1 λ i ·Normalize(hi ), where h i Let λ represent the i-th hardware metric. i The weighting coefficients for this computing unit are normalized to ensure that all metrics are compared on the same scale. For example, when a GPU core frequency is at 90% of its peak, memory bandwidth utilization is 60%, and power supply margin is 40%, the second instantaneous resource potential value of this computing unit may reach 0.75, indicating that the computing unit has high resource reserve capacity. The higher the second instantaneous resource potential value, the more suitable the computing unit is for handling more or more complex expert network tasks.

[0157] S802. Among multiple computing units, the electronic device selects the computing unit whose second instantaneous resource potential is greater than the reference instantaneous resource potential as the target computing unit set.

[0158] The target computing unit set refers to the set of computing units selected from multiple computing units whose second instantaneous resource potential is higher than a preset reference value. The reference instantaneous resource potential is a threshold parameter used to distinguish between high-potential and low-potential computing units, typically set based on overall system performance and load balancing requirements, such as using an average value or a certain percentile (e.g., 0.5). The selection mechanism is based on evaluating the second instantaneous resource potential of each computing unit. If the second instantaneous resource potential of a computing unit is greater than the reference instantaneous resource potential, then that computing unit is included in the target computing unit set. For example, in a dual-card system, if the second instantaneous resource potential of GPU0 is 0.78, the second instantaneous resource potential of GPU1 is 0.45, and the reference instantaneous resource potential is set to 0.5, then only GPU0 will be selected into the target computing unit set. This mechanism of selecting computing units with a second instantaneous resource potential greater than the reference instantaneous resource potential as the target computing unit set ensures that tasks are migrated to computing units with higher resource potential, thereby improving overall inference efficiency and throughput.

[0159] S803, the electronic device determines the target expert network to be adjusted based on the second adjustment factor.

[0160] The role of the second adjustment factor is to provide a dynamic and adjustable control signal to guide the adaptive scheduler to perform fine-grained expert network mapping adjustments.

[0161] Based on the second adjustment factor, the electronic device first calculates the number of expert networks to be adjusted, and then selects the number of expert networks that most need adjustment as the target expert networks to be adjusted.

[0162] S804. The electronic device adjusts the target expert network to the target computing unit set so that the target expert network can be run through the target computing unit set.

[0163] This step enables the dynamic migration and reallocation of expert networks among computing units. The specific process includes: First, calculating the number of expert networks to be migrated, N_move = second adjustment factor × Total_Experts, based on the second adjustment factor; second, identifying the target computing unit set; then, based on the expert network's resource bottleneck type (compute-intensive or memory-intensive) and its corresponding computing unit potential characteristics (such as the frequency component and memory bandwidth component in the second instantaneous resource potential), prioritizing the allocation of target expert networks to computing units in the target computing unit set with high matching degree. For example, for compute-intensive expert networks, priority is given to computing units with high frequency components in the second instantaneous resource potential; for memory-intensive expert networks, priority is given to computing units with high memory bandwidth components in the second instantaneous resource potential. Finally, the migration of the expert network's model parameters and intermediate states is completed through a distributed communication protocol. This dynamic adjustment mechanism effectively alleviates the memory access / communication bottleneck in the MoE architecture and improves the overall Token-Rate.

[0164] By introducing the concept of a second instantaneous resource potential, the remaining hardware capacity of multiple computing units in electronic devices is quantified in real time. A target computing unit set is then constructed based on this concept, and a second adjustment factor is used to dynamically determine the expert network to be adjusted. Finally, the target expert network is migrated to the target computing unit set for operation. This approach fully utilizes the dynamically changing hardware potential of the system at the millisecond level, overcoming the limitations of traditional static or coarse-grained scheduling. This significantly improves the throughput (Token-Rate) of MoE model inference and achieves fine-grained resource utilization while ensuring user latency constraints.

[0165] The following describes the process by which the electronic device in S804 adjusts the target expert network to the target computing unit set.

[0166] In some embodiments, reference Figure 9 The process may include, but is not limited to, S901 to S903 described below.

[0167] S901, the electronic device determines the target processing type of each computing unit based on the second instantaneous resource potential of each computing unit in the target unit set.

[0168] The target processing types include: high computing power type, high bandwidth type, and high memory type.

[0169] The second instantaneous resource potential refers to the amount of hardware resources that a computing unit can dynamically utilize at any given moment. This potential is obtained by weighting and normalizing multiple underlying hardware metrics (such as core frequency, memory bandwidth, PCIe bus bandwidth, and power supply system margin). This potential reflects the maximum performance improvement that a computing unit can provide without exceeding a safe threshold. For example, this potential value is higher when the CPU core frequency is overclocked or the memory bandwidth is not fully utilized. This potential is collected and quantified in real time by the IRP-Sensor module to guide subsequent scheduling decisions.

[0170] Target processing type is a classification label for the main capability characteristics of a computing unit in a specific task scenario. High computing power type indicates that the computing unit has strong floating-point operation capabilities, suitable for performing computationally intensive tasks; high bandwidth type indicates that the computing unit has high memory or I / O throughput capabilities, suitable for performing memory-intensive or communication-intensive tasks; high memory type indicates that the computing unit has large capacity or high-speed cache capabilities, suitable for processing tasks with large amounts of data. These types are not fixed attributes, but are dynamically assessed based on the current instantaneous resource potential state. For example, a GPU may be classified as high computing power type when its frequency is high but its memory bandwidth is limited, while it may be classified as high bandwidth type when its memory bandwidth is sufficient but its frequency is low.

[0171] In practical applications, electronic devices traverse each computing unit in the target computing unit set, read the current second instantaneous resource potential value of that computing unit, and analyze it in conjunction with preset weight coefficients (such as computing weight λ_comp, memory weight λ_mem, etc.). For example, in a dual-GPU system, if GPU0 has an instantaneous frequency potential of 0.9 and a memory bandwidth potential of 0.5, then the target processing type of GPU0 is calculated to be high computing power; while if GPU1 has a frequency potential of 0.6 and a memory bandwidth potential of 0.8, it is calculated to be high bandwidth. This dynamic classification allows the scheduler to more accurately match tasks with hardware capabilities.

[0172] In this way, the target processing type of the computing unit is dynamically determined based on the resource potential of the second instant. This enables refined perception and classification of heterogeneous hardware resources, providing a scientific basis for subsequent expert network allocation, and thus improving the utilization rate and inference efficiency of heterogeneous hardware resources.

[0173] S902. For each target expert network, the electronic device determines the resource bottleneck type based on the computational weight, memory weight, and communication weight when each expert network performs the assigned task.

[0174] Resource bottleneck types include: computing type, memory access type, and communication type.

[0175] Computational weights, memory weights, and communication weights are quantitative indicators describing the degree to which an expert network depends on various resources during runtime. Computational weights reflect the amount of floating-point operations required by the expert network during execution, typically expressed as a normalized ratio of FLOPs to the maximum FLOPs. Memory weights reflect the expert network's requirements for parameter storage and intermediate activation value storage, typically expressed as the ratio of the sum of parameter and activation value sizes to the maximum memory usage. Communication weights reflect the amount of input and output data transmitted by the expert network in distributed inference, typically expressed as the ratio of data transmission volume to the maximum transmission volume. Computational weights, memory weights, and communication weights together constitute the resource requirement vector R. j =[r_comp,r_mem,r_comm], is used to characterize the resource consumption characteristics of expert networks.

[0176] Resource bottleneck type is determined by a weight vector composed of computation weight, memory weight, and communication weight, identifying the resource category most likely to be a performance limiting factor in the actual operation of the expert network. Computation type indicates that the expert network is primarily limited by computational capability, with its computation weight significantly higher than other weights. Memory access type indicates that the expert network is primarily limited by memory bandwidth or access latency, with its memory weight significantly higher than other weights. Communication type indicates that the expert network is primarily limited by data transmission bandwidth or latency, with its communication weight significantly higher than other weights. For example, an expert network with r_comp=0.9, r_mem=0.1, and r_comm=0.2 is classified as computation type; if r_comp=0.2, r_mem=0.9, and r_comm=0.3, it is classified as memory access type.

[0177] In practical implementation, the system collects performance data of each expert network using runtime profiling tools (such as NVIDIA Nsight and PyTorchProfiler), calculates the computational weight, memory weight, and communication weight of each expert network, and determines the type of resource bottleneck based on this. For example, in MoE model inference, differences in the distribution of input tokens among different experts may lead to higher activation sparsity for some experts, resulting in higher communication overhead, which is then identified as a communication bottleneck. This dynamic identification method avoids static assumptions and improves scheduling accuracy.

[0178] In this embodiment, the resource bottleneck type of the expert network is dynamically determined based on computational weight, memory weight, and communication weight. This allows for accurate identification of the actual performance constraints of the expert network, providing crucial input for subsequent matching scheduling and effectively mitigating the multi-dimensional resource bottleneck problem in the MoE architecture.

[0179] S903. Based on the resource bottleneck type of each target expert network, the electronic device selects the computing unit belonging to the processing type that matches the resource bottleneck type from the target computing unit set, and uses it as the adjusted computing unit for running the target expert network.

[0180] Among them, high computing power type matches computing type, memory access type matches high memory type, and high bandwidth type matches communication type.

[0181] This step implements potential-aware matching and scheduling between expert networks and computing units. Specifically, for expert networks identified as computation-oriented, computing units with high computing power as their target processing type are prioritized; for expert networks identified as memory-oriented, computing units with high memory usage as their target processing type are prioritized; and for expert networks identified as communication-oriented, computing units with high bandwidth as their target processing type are prioritized. This matching principle ensures that the resource bottlenecks of the expert network can be compensated for by the advantages of the corresponding hardware.

[0182] In practice, a mapping relationship is first established between experts and resource bottleneck types, and between computing units and target processing types. Then, the system uses ΔΠ (expert mapping adjustment factor) to drive a migration strategy, migrating the expert network to matching high-potential computing units. For example, when ΔΠ = +0.5, the system identifies a set of high-potential units U_HighPoten and migrates a certain number of experts (e.g., N_move = ΔΠ × Total_Experts) from low-potential units to high-potential units. Simultaneously, the system follows a matching principle: computational experts are migrated to high-computing-power units, memory-accessing experts are migrated to high-memory units, and communication-oriented experts are migrated to high-bandwidth units.

[0183] In this embodiment, a dynamic allocation mechanism for expert networks and computing units is implemented based on the principle of matching resource bottleneck types with target processing types. This allows for precise alignment between expert tasks and hardware potential, effectively reducing overall inference latency and maximizing the token rate throughput of the MoE model.

[0184] Below, taking the MoE model as an example, we will illustrate the model processing scheme through an embodiment.

[0185] To facilitate understanding, some terms are explained in contrast.

[0186] The initial instantaneous resource potential is P. t The first adjustment factor is κ_batch, and the target batch size is B. t The instantaneous operating frequency parameter of the central processing unit (CPU) is h_Freq, the instantaneous memory transfer rate parameter is h_MemBW, the instantaneous bus bandwidth parameter is h_PCIE, and the instantaneous remaining power parameter of the power supply is h_PowerHeadroom. The first batch size is B. t-1 × κ_batch, the maximum batch size is B_max, the minimum batch size is B_min, and the second batch size is B t pred Reference delay L_targe, current delay L_current, delay margin S t The delay margin correction function is f(S) t ), First degree η, Operating load rate Experts, Load imbalance Imbalance, Second adjustment factor ΔΠ.

[0187] The MoE architecture improves model capacity and efficiency through sparse activation. Its inference performance (token rate) is limited by computation, memory access, and communication. Existing optimization methods (such as vLLM's PagedAttention and underlying CUDA kernel optimization) mainly focus on operator-level efficiency or memory utilization, and typically employ continuous batching to improve throughput. However, their scheduling decisions mainly depend on the length of the request queue and the instantaneous load of the computation unit.

[0188] The following defects exist in the related technologies: Lack of awareness and static nature of hardware potential: In related technologies, schedulers treat computing units as entities with static performance. They cannot perceive and utilize the instantaneous resource potential of the hardware system, which can be dynamically adjusted at the millisecond level (such as the overclocking capability of the CPU core, the instantaneous bandwidth margin of the DDR / PCIe bus, and the power margin of the motherboard power supply system). This blindness leads to overly conservative scheduling decisions, failing to fully tap the system's maximum throughput potential under high-potential conditions.

[0189] A rigid trade-off between throughput and latency: Dynamic batch processing in related technologies lacks fine-grained awareness of the user's latency constraint Ltarget, making it impossible to safely increase the batch size using hardware potential without exceeding Ltarget. This makes it difficult for the system to find the optimal dynamic balance between maximizing throughput and ensuring latency.

[0190] Inefficient matching of expert tasks and I / O resources: In MoE model inference, different experts exhibit different resource bottlenecks (computation-intensive or memory-intensive) at runtime due to factors such as data access patterns and activation sparsity. The schedulers in related technologies lack fine-grained bandwidth awareness and cannot dynamically allocate I / O-intensive experts to cores with high instantaneous available bandwidth (high potential) based on runtime profiling (performance analysis), causing memory access bottlenecks to become the bottleneck for overall inference speed.

[0191] The purpose of this embodiment is to overcome the shortcomings of existing MoE inference systems in terms of hardware potential awareness and dynamic scheduling, and to provide a feedback scheduling system based on instantaneous resource potential (IRP). This system aims to: The dynamic potential of the underlying hardware (frequency, bandwidth, power supply) is quantified in real time and transformed into input for scheduling decisions.

[0192] While strictly adhering to the user-defined maximum acceptable latency constraint Ltarget, the MoE Token-Rate is maximized by dynamically adjusting the batch size and expert allocation. This maximizes the model processing rate. This embodiment proposes a "MoE Token-Rate maximization scheduling system based on dynamic resource potential perception", which constitutes a closed-loop feedback control system.

[0193] It includes: the Instantaneous Resource Potential Sensor (IRP-Sensor) module, the Token-Rate Policy Algorithm (TR-Policy Algorithm) module, and the Adaptive Scheduler module.

[0194] The Instantaneous Resource Potential Sensor (IRP-Sensor) module collects and integrates multi-dimensional underlying hardware data in real time, outputting a normalized IRP value, Pt. Data sources include: power supply system status, core instantaneous frequency, and memory / bus instantaneous bandwidth. Pt abstracts heterogeneous underlying hardware metrics (such as frequency and bandwidth) into a unified, quantifiable metric that can be used for upper-layer scheduling, avoiding straightforward engineering descriptions.

[0195] The Token-Rate (TR-Policy Algorithm) module employs a rule-based adaptive algorithm to quickly determine the predicted batch size B that maximizes the Token Rate based on the current system state. t pred And the expert mapping adjustment factor ΔΠ. B t predImplement hardware-state-driven dynamic batch size, allowing for larger batch sizes under high resource potential, thereby improving the token rate. ΔΠ implements hardware-state-driven dynamic expert tuning.

[0196] The Adaptive Scheduler module applies a potential-aware safety correction algorithm to adjust the predicted value to the final scheduling parameter B. t and Π t It also performs potential-aware expert mapping to ensure that the latency constraint L_target is strictly satisfied while maximizing the Token Rate. t It is a specific scheduling strategy that enables more granular, bandwidth-aware expert scheduling, solving the memory access / communication bottleneck of MoE.

[0197] These three modules constitute a closed-loop feedback control system (a cross-level architecture of perception → decision-making → execution → feedback).

[0198] IRP Sensor (Perception Layer): Hardware State → Quantification → Instantaneous Resource Potential P t .

[0199] TR Policy Algorithm (Decision-Making Level): Instantaneous Resource Potential P t →(Policy Algorithm)→(Batch Size B) t pred Adjustment factor ΔΠ) Adaptive Scheduler (Execution / Control Layer): (B t pred , ΔΠ)→(Safety Correction)→(B t ,Π t → (Execution) → L_current (delay) This embodiment has the following beneficial effects: Maximizing Token Rate: By sensing and utilizing the instantaneous overclocking and high bandwidth potential of the hardware system through the IRP-Sensor, the Batch Size is dynamically adjusted, and it is expected that a 20%-30% increase in Token Rate can be achieved under high potential conditions.

[0200] Refined utilization of hardware resources: For the first time, underlying engineering indicators such as dynamic power supply status, instantaneous core frequency, and PCIe / DDR bandwidth are transformed into decision factors for the upper-level scheduler, effectively tapping the limits of heterogeneous hardware performance.

[0201] Traditional expert load balancing methods often employ static round-robin, historical statistics-based (Deepseed MOE), utilization-based load balancing (vLLM), or expert weight-based allocation (Tutel). None of these consider the instantaneous resource potential of the hardware, the type of expert, or the magnitude of load migration triggered by Δπ. Furthermore, this solution is significantly faster. Traditional methods for adjusting batch size include fixing a fixed value, dynamically adjusting based on queue length (Nvidia Triton Inference Server), and adjusting based on latency feedback (VLLM, TensorRT). None of these methods take into account the instantaneous resource potential of the hardware, and there is no suitable strategy for adaptively adjusting the batch size.

[0202] The implementation of the three modules will be explained below.

[0203] Instantaneous Resource Potential Sensor (IRP-Sensor) module: Construction of the Instantaneous Resource Potential Index (IRP).

[0204] The IRP-Sensor collects N key hardware metrics H={h1,h2,…,h…} }, and then standardize and weighted sum them to obtain IRP-P t :P t =Σ n i=1 λ i ·Normalize(h i ).

[0205] Among them, the key hardware indicator h i This includes: h_Freq: The instantaneous average operating frequency of the CPU / accelerator cores in the running model (ratio to the maximum frequency). h_MemBW: The instantaneous available bandwidth of the system DDR (ratio to the peak bandwidth). h_PCIE: The instantaneous idle throughput potential of the PCIE bus (ratio to the Gen speed limit). h_PowerHeadroom: The instantaneous available headroom of the power supply system or thermal design power (TDP). Assuming the total power supply is 1200W, and the CPU, GPU, and other devices currently occupy 720W, then this parameter is (1200-720) / 1200=0.4, representing overclocking capability or sufficient headroom to increase the load.

[0206] λ iThese are weighting coefficients, which can be preset based on the model's memory access / computation bottlenecks.

[0207] λ i The method for setting weighting coefficients includes: Step 1: Benchmarking. Under a fixed batch size, artificially simulate fluctuations in hardware metrics (by limiting core frequency or memory bandwidth using system tools). Step 2: Sensitivity Calculation. Measure the Token Rate T for each hardware metric h. i Partial derivatives of: Sensitivity i =| h i / T|. Step 3: Weight normalization. Use sensitivity as λ. i The initial value of Σλ is then normalized to ensure that Σλ i =1:λ i =Sensitivity i / Σ n j= 1Sensitivity j .

[0208] This quantitative analysis, based on models and hardware bottlenecks, is used to set λ. i This can ensure IRPP t It can accurately represent the effective resource potential of the current system for reasoning about the model.

[0209] Token-Rate Policy Algorithm (TR-PolicyAlgorithm) module: To make decisions quickly within millisecond-level scheduling cycles, TR-PolicyAlgorithm employs a rule-based adaptive algorithm that uses four steps to rapidly determine the predicted batch size B that maximizes the Token Rate based on the current system state. t pred And the expert mapping adjustment factor ΔΠ.

[0210] The processing of the Token-Rate Policy Algorithm (TR-Policy Algorithm) module may include, but is not limited to, the following S1001 to S1004.

[0211] S1001, Calculate batch size and adjustment factor.

[0212] S1002, Predict Batch Size B t pred .

[0213] S1003, Calculate the expert load imbalance.

[0214] S1004, Calculate the expert mapping adjustment factor.

[0215] Input layer definition: The input layer receives three key indicators, which together form the input vector X. t :X t =[P t B t-1 ,Expert_Load t-1 ]. Among them, P t (Instantaneous Resource Potential): A core metric derived from IRP-Sensor. B t-1 (Final Batch Size of the previous scheduling cycle): Historical scheduling decisions. Expert_Load t-1 (Expert load distribution in the previous period): Expert load balance.

[0216] Algorithm output: Output vector Y t =[B t pred ,ΔΠ]. Among them, B t pred : Predicted batch size. ΔΠ: Expert mapping adjustment factor ∈ [-1,1].

[0217] Step 1: Calculate the batch size adjustment factor κ_batch.

[0218] Adjustment strategies based on hardware potential: ifP t ≥0.7: (Hardware potential ≥ 40%, secure) κ_batch = 1.3 + 0.5 × (P t - 0.7) / / Aggressive strategy: Leveraging hardware potential and latency margins to scale up batch sizes elif P t ≥ 0.5: (Hardware potential 10%~40%, proceed with caution) κ_batch = 1.1 / / Mild strategy: small increases elif P t ≥ 0.3: (Hardware potential 0~10%, dangerous) κ_batch = 0.9 / / Conservative strategy: appropriate contraction else: (S) t <0, delay exceeded (The delay is already exceeded) κ_batch = 0.7 / / Emergency strategy: significant contraction Step 2: Predict batch size.

[0219] B t pred=clip(B t-1 ×κ_batch,B_min,B_max) Where: B_min=1 (minimum batch); B_max=64 (maximum batch, depending on hardware configuration); clip(x,min,max)=max(min,min(max,x)).

[0220] B t pred B represents the total batch size for MoE inference. t pred This is the predicted value from TR-PolicyAlgorithm, not the final batch size. It will be corrected by a third module. The process is as follows: preprocessing of total batch requests → splitting into sub-batches according to expert allocation strategy → MoE sparse activation push → metric collection and feedback.

[0221] Step 3: Calculate the expert load imbalance degree (Imbalance).

[0222] Expert load variance: Var_expert = √(Σ(Expert_Load) i -Mean_Load)² / N_experts) Load imbalance: Imbalance = Var_expert / Mean_Load Judgment criteria: Imbalance < 0.2: Good load balancing; 0.2 ≤ Imbalance < 0.4: Slight imbalance; Imbalance ≥ 0.4: Severe imbalance.

[0223] Experts: Multiple independent sub-models (expert networks) in the MoE architecture are the core components of MoE sparse activation. Each expert network has the same structure but independent parameters. The number Nexperts is fixed during the model design phase (e.g., 8 / 16 / 32). The model data can be directly read.

[0224] Expert_Load i The load rate of the i-th expert in the previous scheduling cycle, normalized to [0,1], reflects the degree of computing power / memory / communication resource consumption and the busyness of task processing of the expert. It is obtained by real-time collection and normalization by hardware performance analysis tools during runtime.

[0225] Mean_Load: The average load rate of all experts in the previous scheduling period, which is the sum of all Expert_Load values. i The arithmetic mean of the values ​​reflects the overall expert load balancing status of the MoE, and is calculated using the formula of the two variables above plus the arithmetic mean.

[0226] Step 4: Calculate the expert mapping adjustment factor.

[0227] Adjustment strategies based on resource potential and load balancing: ifP t ≥0.7 and Imbalance ≥0.3:S t ≥0.1 / / High potential + Uneven load → Diffusion to high potential cells ΔΠ = +0.5 + 0.3 × (P t - 0.7); elif P t ≥ 0.5 and Imbalance ≥ 0.4: / / Moderate potential + Severe imbalance → Moderate adjustment ΔΠ = +0.3; elif P t <0.4: / / Low potential → Shrink to core unit ΔΠ = -0.4 - 0.5 × (0.4 - P t ) elif S t <0.1: / / Delay critical → Conservative contraction ΔΠ = -0.2 else: / / System stable → Maintain the status quo ΔΠ = 0 / / Limited to the valid range ΔΠ = clip(ΔΠ, -1, 1) Given: Ltarget = 200ms (user target latency), Lcurrent = 60ms (current actual latency), Btpred = 30 (predicted batch size), η = 5 (aggressive factor), Bmax = 64 (maximum batch size), f(St) ≈ +1 (S t (When it is relatively large).

[0228] Calculate the delay margin: St = (200 60) / 200=0.7 (belongs to S) t Larger); calculate the final batch size: Bt =min(64,30+5×1)=35; Scheduling result: Approved to increase the batch size from 30 to 35, fully utilize the hardware potential to improve throughput, and will not exceed L_target.

[0229] Example of token rate tuning: Assume input: P t=0.85 (high hardware potential), B t-1 =20, Expert_Load t-1 =[0.7,0.8,0.75,0.72,0.68,0.78,0.71,0.65], then the system processing procedure can be referred to the contents shown in Table 1.

[0230] Table 1. Example of system processing procedure

[0231] Conclusion: B t pred =22 (Increase batch size to improve throughput) ΔΠ=0 (Load balancing, no adjustment required).

[0232] Adaptive Scheduler module: The adaptive scheduler mainly consists of two core functions: the Potential-Aware Safety Correction (IRP) algorithm, used to determine the final batch size B. t IRP-AwareExpert Mapping: Used to determine the final expert assignment strategy. t .

[0233] Potential-aware safety correction algorithm: The goal is to improve the ideal batch size B predicted by the TR-Policy algorithm. t pred Revised to a safe and efficient final batch size B t This prevents the actual delay L_current from exceeding the target delay L_target. Input: B t pred , ΔΠ, L_current, L_target.

[0234] Core correction logic: B t pred Based solely on the current resource potential P t Compared with the predicted value of historical delay L_hist, the final B t It must be adjusted in conjunction with real-time latency margin. Latency margin: S t =(L_target-L_current) / L_target. Where, S... t Larger size → Sufficient delay margin → Allows for aggressive strategies; S t The smaller the value, the closer the latency will be to the upper limit, thus requiring a conservative approach.

[0235] L_current is the actual end-to-end inference latency of the MoE model in processing a batch of requests within the current scheduling period, measured in milliseconds (ms). It is collected in real time through system-level timing and hardware performance tool calibration.

[0236] Batch Size Modified Formula B t = min(B_max, B t pred + η · f(S t Where: B_max: The maximum batch size allowed by the system. η (Aggressiveness Factor): The degree of aggressiveness (empirical value) by which the control system utilizes the hardware potential. f(S) t ): Correction function based on delay margin f(S t = tanh(k_learned · (S t - S0_learned)). k_learned, the magnification factor, and the initial state value of S0_learned are obtained through training.

[0237] The delay margin threshold (learning-based) is a personalized hyperparameter of the model obtained through offline training and online fine-tuning in this approach. The correction function f(S) t ): When S t Larger (L_current) L_target, high safety margin): f(S t → 1; Allow B t Close to or slightly higher than B t pred (Aggressive growth). When S t Approaching 0 or less than 0 (delay approaching or exceeding the limit): f(S) t → 1; B t It is rapidly compressed (conservative contraction).

[0238] Example 1: Ample delay margin, aggressive strategy.

[0239] Input: L_current=80ms, L_target=150ms, -B t pred =24, η=5, B_max=64.

[0240] Calculate: S t=(150-80) / 50=0.47 (good margin); f(S) t )≈+0.8 (close to +1, indicating potential for growth); B t =28.

[0241] Result: Approval to increase Batch Size (24 B) t pred →28 B t ).

[0242] The purpose of IRP-Aware Expert Mapping is to determine how to perform expert transfer within each GPU based on the obtained ΔΠ.

[0243] During the MoE model inference process, although the network structures of each expert are the same, due to differences in input data distribution, activation mode, and cache hit rate, different experts exhibit different resource bottleneck characteristics (computation-intensive, memory-intensive, and I / O-intensive) at runtime.

[0244] This feature leverages IRP's fine-grained resource awareness and runtime profiling to enable expert allocation of bandwidth and computing potential.

[0245] Expert Task Classification and Resource Requirements Modeling: For each expert E j Determine its resource bottleneck type: Type(E) j )∈{Compute-Bound, Memory-Bound}. Resource requirement vector definition: R j =[r_comp,r_mem,r_comm] represent the weights of computation, memory, and communication resource requirements, respectively.

[0246] The type of resource bottleneck can be obtained by collecting and calculating data using performance analysis tools such as NVIDIA Nsight and PyTorch Profiler.

[0247] Runtime resource bottleneck analysis: Although the network structure of each expert in MoE is the same, dynamic differences will occur in actual inference due to the following factors: differences in the data distribution of input tokens lead to different computation paths; different activation sparsity of different experts; cache hit rate varies depending on the access frequency of experts; and differences in communication overhead in distributed scenarios.

[0248] The calculation methods for each component may include: Calculate the demand weights: r_comp = FLOPs / FLOPs_max, normalized to [0, 1].

[0249] FLOPs: The floating-point operations of this expert. FLOPs_max: The maximum floating-point operations among all experts. Memory requirement weight: r_mem = (Params_size + Activation_size) / Memory_max. Params_size: Memory occupied by parameters. Activation_size: Memory occupied by intermediate activation values. Memory_max: The maximum memory usage among all experts.

[0250] Communication demand weight: r_comm = Data_transfer / Transfer_max. Data_transfer: Input / output data transfer volume. Transfer_max: Maximum data transfer volume.

[0251] r_comp, r_mem, and r_comm are normalized to [0, 1].

[0252] The three parameters reflect the degree to which experts rely on computing, memory, and communication resources during reasoning—the higher the value, the higher the demand for these resources, and the more likely it is that insufficient resources will create a reasoning bottleneck.

[0253] r_comm is the key innovation that distinguishes this solution from existing technologies. Its core contribution is to fill the gap in the MoE scheduling that "ignores communication bottlenecks" and perfectly adapt to the distributed inference characteristics of MoE.

[0254] Example: Expert 1: Compute-Bound R j =[0.9,0.1,0.2]→Requires high computing power and is sensitive to CPU / GPU frequency.

[0255] Expert 2: Memory-Bound (Memory-Intensive) R j =[0.2,0.9,0.3]→Requires high bandwidth and is sensitive to memory speed. 2. Evaluation of computing unit potential (hardware processing capability of the model).

[0256] The scheduler focuses on the instantaneous resource potential of computing units, rather than static load.

[0257] P_unit,k = λ_comp · P_Freq,k + λ_mem · P_MemBW,k.

[0258] Where: P_Freq,k: Instantaneous core frequency potential of computing unit k, obtained by reading the system API. P_MemBW,k: Instantaneous memory bandwidth potential of computing unit k, obtained by reading the system API. λ_comp, λ_mem: Weight coefficients, obtained through adaptive weights (why only two weights here?). The formula λ_comp + λ_mem = 1 (normalization constraint) is satisfied. Initial values ​​are obtained offline, and performance testing tools monitor online in real time, dynamically updating the weights of the system's core bottlenecks.

[0259] 3. Dynamic expert allocation strategy (Π) t ).

[0260] The scheduler performs expert mapping based on ΔΠ and the difference in computational unit potential: ① High Potential Unit Identification: Select the set of units U_HighPoten where P_unit,k is higher than the average value.

[0261] ② The principle of matching potential with ability: Memory-Bound intensive memory access expert: Prioritize allocation to the highest unit of P_MemBW,k.

[0262] Compute-Bound computationally intensive experts: preferentially allocate to the highest P_Freq,k cell.

[0263] ③ΔΠ-driven migration strategy.

[0264] Based on the existing expert allocation, first determine the matching degree, and then perform the "migration + local reallocation" operation.

[0265] If ΔΠ>0: This indicates that high-potential units need to be utilized more fully, and N_move=ΔΠ×Total_Experts experts should be migrated to U_HighPoten. If ΔΠ<=0: This indicates that resources are becoming scarce, and conservative or distributed load balancing should be implemented.

[0266] This mechanism enables potential-aware dynamic load balancing, effectively alleviating the limitation of memory access bottlenecks in MoE on overall inference speed.

[0267] For example: Step 1: Expert task classification.

[0268] First, calculate the [computation, memory, communication] vector for each expert.

[0269] Expert 1: Compute-Bound R j= [0.9, 0.1, 0.2] → Requires high computing power and is sensitive to CPU / GPU frequency. Expert 2: Memory-Bound (Memory-intensive) R j = [0.2, 0.9, 0.3] → Requires high bandwidth and is sensitive to memory speed.

[0270] Step 2: Calculate the potential assessment of the unit.

[0271] Evaluate the "capabilities" of each GPU core.

[0272] Example: GPU0, GPU2: P_Freq,0=0.9 (high frequency potential), P_MemBW,0=0.5 (medium bandwidth potential), P_unit,0=0.6×0.9+0.4×0.5=0.74, high overall potential.

[0273] Identify high-potential units: U_HighPoten={GPU0,GPU2} ← Potential is above average.

[0274] Step 3: ΔΠ-driven migration strategy.

[0275] For example, ΔΠ=+0.5→ requires more full utilization of high-potential units.

[0276] Calculate the number of migrations: N_move = ΔΠ × Total_Experts = 0.5 × 8 = 4 experts.

[0277] Strategy: 1. Identify U_HighPoten={GPU0, GPU1, GPU2} (high-potential units). 2. Transfer 4 experts to these units.

[0278] Before migration: GPU0 (P_unit=0.74, high computing power): Expert 1 (computational); GPU1 (P_unit=0.60, high bandwidth): Expert 2 (memory access); GPU2 (P_unit=0.74, high computing power): Expert 3 (computational); GPU3 (P_unit=0.55, low performance): Expert 4, 5, 6, 7, 8.

[0279] After migration (ΔΠ = +0.5): GPU0 (high computing power): Expert 1, Expert 4 (computational), Expert 6 (computational) ← Increase load; GPU1 (high bandwidth): Expert 2, Expert 5 (memory access) ← Increase memory access; GPU2 (high computing power): Expert 3, Expert 7 (computational) ← Increase load; GPU3 (low performance): Expert 8 (lightweight) ← Reduce load.

[0280] Because GPU3 was identified as low-performance, and there are 8 experts on that GPU, let's assume that expert number 8 is a lightweight expert. So we move 4, 5, 6, and 7. Since 4 and 6 are computationally intensive, they are moved to GPU0 with high computing power. 5 is memory-intensive, so it is moved to GPU1 with high bandwidth. To balance computing power, experts number 7 are moved to GPU2 (high computing power).

[0281] Result: Fully utilize high-performance GPUs to improve throughput.

[0282] Test data: TCX hardware platform: 2×NVIDIA A6000 48GB GPU, CPU: Ultra9 285, memory: 256GB DDR4.

[0283] Model configuration: Total parameters: 56BMixtral-8x7B MOE, Number of experts: 8 experts, Each expert: 7B parameters, Number of activated experts: Top-2.

[0284] Secondly, embodiments of this application provide a model processing apparatus, such as... Figure 11 As shown, the model processing device 110 includes: The first determining unit 1101 is used to determine a first instantaneous resource potential based on multiple hardware parameters of an electronic device running a target model; the first instantaneous resource potential is used to characterize the instantaneous remaining hardware capacity of the electronic device for running the target model. The second determining unit 1102 is used to determine a first adjustment factor for adjusting the batch size of the target model based on the first instantaneous resource potential; the first instantaneous resource potential is positively correlated with the first adjustment factor; The third determining unit 1103 is used to determine the target batch size of the target model in the current scheduling cycle based on the first adjustment factor; The processing unit 1104 is used to perform model inference based on the target batch size in the current scheduling cycle to match the remaining hardware capacity of the electronic device.

[0285] In some embodiments, the first determining unit 1101 is further configured to: collect multiple hardware parameters of the electronic device; the multiple hardware parameters include at least one or more of the following: instantaneous operating frequency parameter of the central processing unit (CPU) of the electronic device, instantaneous transfer rate parameter of the memory of the electronic device, instantaneous bandwidth parameter of the bus of the electronic device, and instantaneous remaining power parameter of the power supply of the electronic device; and perform weighted and normalized processing on the multiple hardware parameters to obtain a first instantaneous resource potential.

[0286] In some embodiments, the third determining unit 1103 is further configured to: determine the first batch size by multiplying the first adjustment factor by the historical batch size of the target model; obtain the maximum batch size and minimum batch size that the electronic device can support; predict the second batch size that the target model can process in the current scheduling cycle based on the first batch size, the maximum batch size and the minimum batch size; and determine the target batch size based at least on the second batch size.

[0287] In some embodiments, the third determining unit 1103 is further configured to: obtain a set maximum acceptable reference delay for the user and the current delay of the target model operation; determine a delay margin based on the reference delay and the current delay; the delay margin is used to characterize the time delay margin during the operation of the target model; determine a target batch size based on the delay margin, the delay margin correction function, a first degree, and the maximum batch size; the delay margin correction function is used to correct the batch size based on the delay margin, and the first degree is the aggressiveness of utilizing the hardware potential of the electronic device.

[0288] In some embodiments, when the target model includes multiple expert networks, the processing unit 1104 is further configured to: obtain the operating load rate of each expert network of the target model in the previous scheduling cycle; determine the load imbalance of the expert networks based on the operating load rate of each expert network in the previous scheduling cycle; determine a second adjustment factor for adjusting the expert network allocation strategy based at least on the first instantaneous resource potential and the load imbalance; and adjust the allocation tasks of each expert network based on the second adjustment factor so as to achieve model inference through the adjusted expert networks.

[0289] In some embodiments, the processing unit 1104 is further configured to: determine a second adjustment factor based on a first strategy when the first instantaneous resource potential belongs to a first range and the load imbalance is greater than or equal to a second range; determine the second adjustment factor as a first reference value when the first instantaneous resource potential belongs to a third range and the load imbalance belongs to a fourth range; wherein the first instantaneous resource potential of the first range is greater than the first instantaneous resource potential of the third range, and the load imbalance of the fourth range is greater than the load imbalance of the second range; determine the second adjustment factor based on a second strategy when the first instantaneous resource potential is based on a fifth range; the first instantaneous resource potential of the fifth range is less than the first instantaneous resource potential of the third range, and the adjustment strength of the second strategy is less than the adjustment strength of the first strategy.

[0290] In some embodiments, the processing unit 1104 is further configured to: determine the delay margin of the target model operation; the delay margin is used to characterize the time delay margin of the target model operation; and determine a second adjustment factor based on the first instantaneous resource potential, the load imbalance degree and the delay margin.

[0291] In some embodiments, the processing unit 1104 is further configured to: determine a second instantaneous resource potential for each of a plurality of computing units in an electronic device; the second instantaneous resource potential is used to characterize the remaining hardware capacity of the computing unit for running an expert network; select, among the plurality of computing units, computing units whose second instantaneous resource potential is greater than a reference instantaneous resource potential as a target computing unit set; determine a target expert network to be adjusted based on a second adjustment factor; and adjust the target expert network to the target computing unit set so as to run the target expert network through the target computing unit set.

[0292] In some embodiments, the processing unit 1104 is further configured to: determine the target processing type of each computing unit based on the second instantaneous resource potential of each computing unit in the target unit set; the target processing type includes: high computing power type, high bandwidth type, and high memory type; for each target expert network, determine the resource bottleneck type of each target expert network based on the computing weight, memory weight, and communication weight when each expert network performs the assigned task; the resource bottleneck type includes: computing type, memory access type, and communication type; based on the resource bottleneck type of each target expert network, select the computing unit to which the processing type that matches the resource bottleneck type belongs in the target computing unit set as the adjusted computing unit for running the target expert network; wherein, the high computing power type matches the computing type, the memory access type matches the high memory type, and the high bandwidth type matches the communication type.

[0293] It should be noted that the processing device of the model provided in this application embodiment includes all the units included, which can be implemented by a processor in an electronic device; of course, it can also be implemented by specific logic circuits; in the implementation process, the processor can be a central processing unit (CPU), a microprocessor (MPU), a digital signal processor (DSP), or a field-programmable gate array (FPGA), etc.

[0294] The descriptions of the above device embodiments are similar to those of the above method embodiments, and have similar beneficial effects. For technical details not disclosed in the device embodiments of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0295] It should be noted that, in the embodiments of this application, if the above methods are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, or the parts that contribute to related technologies, can be embodied in the form of software products. These computer software products are stored in a storage medium and include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), magnetic disks, or optical disks. Thus, the embodiments of this application are not limited to any specific hardware and software combination.

[0296] Thirdly, embodiments of this application provide an electronic device that can implement the processing method of the model provided in the first aspect.

[0297] In some embodiments, the electronic device includes multiple hardware components and a processor, on which a target model runs; the processor is configured to: determine a first instantaneous resource potential based on multiple hardware parameters; the first instantaneous resource potential characterizes the instantaneous total remaining hardware capacity of the electronic device for running the target model; determine a first adjustment factor for adjusting the batch size of the target model based on the first instantaneous resource potential; the first instantaneous resource potential is positively correlated with the first adjustment factor; determine a target batch size of the target model in the current scheduling period based on the first adjustment factor; and in the current scheduling period, the target model performs model inference based on the target batch size to match the processing capacity of the electronic device.

[0298] Fourthly, embodiments of this application provide a storage medium, namely a computer-readable storage medium, on which a computer program or instructions are stored, which, when executed by a processor, implement the steps in the processing method of any of the models provided in the first aspect of the above embodiments.

[0299] Fifthly, embodiments of this application provide a computer program product, which includes a computer program or instructions. When the computer program or instructions are executed by a processor, they implement the steps in the processing method of any of the models provided in the first aspect of the above embodiments.

[0300] It should be noted that the descriptions of the above embodiments of storage media, devices, apparatuses, and program products are similar to the descriptions of the above method embodiments and have similar beneficial effects. For technical details not disclosed in the embodiments of storage media, devices, apparatuses, and program products of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0301] It should be understood that the phrase "one embodiment" or "an embodiment" throughout the specification means that a specific feature, structure, or characteristic related to the embodiment is included in at least one embodiment of this application. Therefore, "in one embodiment" or "in some embodiments" appearing throughout the specification do not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. It should be understood that in the various embodiments of this application, the sequence numbers of the above-described processes do not imply a sequential order of execution; the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application. The sequence numbers of the above-described embodiments are merely descriptive and do not represent the superiority or inferiority of the embodiments.

[0302] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0303] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components can be combined, or integrated into another electronic device, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection between devices or units can be electrical, mechanical, or other forms.

[0304] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units. They may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.

[0305] In addition, each functional unit in the various embodiments of this application can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0306] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media that can store program code, such as mobile storage devices, read-only memory (ROM), magnetic disks, or optical disks.

[0307] Alternatively, if the integrated units described above are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, or the parts that contribute to related technologies, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0308] The above are merely embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A method for processing a model, the method comprising: Based on multiple hardware parameters of the electronic device running the target model, the resource potential in the first instant is determined; The first instantaneous resource potential is used to characterize the instantaneous remaining hardware capacity of the electronic device for running the target model; Based on the first instantaneous resource potential, a first adjustment factor is determined for adjusting the batch size of the target model; The first instantaneous resource potential is positively correlated with the first adjustment factor; The target batch size of the target model in the current scheduling cycle is determined based on the first adjustment factor; In the current scheduling cycle, the target model performs model inference based on the target batch size to match the remaining hardware capabilities of the electronic device.

2. The method according to claim 1, wherein the first instantaneous resource potential determined based on multiple hardware parameters of the electronic device running the target model includes: Collect multiple hardware parameters of the electronic device; The plurality of hardware parameters include at least one or more of the following: the instantaneous operating frequency parameter of the central processing unit (CPU) of the electronic device, the instantaneous transfer rate parameter of the memory of the electronic device, the instantaneous bandwidth parameter of the bus of the electronic device, and the instantaneous remaining power parameter of the power supply of the electronic device. The first instantaneous resource potential is obtained by weighting and normalizing the multiple hardware parameters.

3. The method according to claim 1, wherein determining the target batch size of the target model in the current scheduling period based on the first adjustment factor includes: The first batch size is determined by multiplying the first adjustment factor by the historical batch size of the target model. Obtain the maximum and minimum batch sizes that the electronic device can support; Based on the first batch size, the largest batch size, and the smallest batch size, predict the second batch size that the target model can process in the current scheduling cycle; The target batch size is determined based at least on the second batch size.

4. The method of claim 3, wherein determining the target batch size based at least on the second batch size comprises: Obtain the maximum acceptable reference latency set by the user and the current latency of the target model running; Based on the reference delay and the current delay, a delay margin is determined; the delay margin is used to characterize the time delay margin during the execution of the target model. The target batch size is determined based on the delay margin, the delay margin correction function, the first degree, and the maximum batch size; the delay margin correction function is used to correct the batch size based on the delay margin, and the first degree is the aggressiveness of utilizing the hardware potential of the electronic device.

5. The method according to any one of claims 1-4, wherein when the target model comprises multiple expert networks, the target model performs model inference based on the target batch size, comprising: Obtain the operating load rate of each expert network of the target model in the previous scheduling cycle; Based on the operating load rate of each expert network in the previous scheduling cycle, the load imbalance of the expert network is determined. A second adjustment factor for adjusting the expert network allocation strategy is determined based at least on the first instantaneous resource potential and the load imbalance. The task assignments of each expert network are adjusted based on the second adjustment factor so that model inference can be achieved through the adjusted expert networks.

6. The method of claim 5, wherein determining the second adjustment factor for adjusting the expert network allocation strategy based at least on the first instantaneous resource potential and the load imbalance includes: If the first instantaneous resource potential falls within a first range and the load imbalance is greater than or equal to a second range, the second adjustment factor is determined based on the first strategy. When the first instantaneous resource potential belongs to the third range and the load imbalance belongs to the fourth range, the second adjustment factor is determined as the first reference value; wherein, the first instantaneous resource potential of the first range is greater than the first instantaneous resource potential of the third range, and the load imbalance of the fourth range is greater than the load imbalance of the second range. When the first instantaneous resource potential is based on the fifth range, the second adjustment factor is determined based on the second strategy; the first instantaneous resource potential of the fifth range is less than the first instantaneous resource potential of the third range, and the adjustment intensity of the second strategy is less than the adjustment intensity of the first strategy.

7. The method according to claim 5, wherein determining a second adjustment factor for adjusting the expert network allocation strategy based at least on the first instantaneous resource potential and the load imbalance includes: Determine the latency margin for the target model's execution; The delay margin is used to characterize the time delay margin during the runtime of the target model; The second adjustment factor is determined based on the first instantaneous resource potential, the load imbalance, and the delay margin.

8. The method according to claim 5, wherein adjusting the task allocation of each expert network based on the second adjustment factor comprises: Determine the second instantaneous resource potential of each of the plurality of computing units in the electronic device; The second instantaneous resource potential is used to characterize the remaining hardware capacity of the computing unit for running the expert network; Among the plurality of computing units, the computing units whose second instantaneous resource potential is greater than the reference instantaneous resource potential are selected as the target computing unit set; Based on the second adjustment factor, the target expert network to be adjusted is determined; The target expert network is adjusted to the target computing unit set so that the target expert network can be run through the target computing unit set.

9. The method according to claim 8, wherein adjusting the target expert network to the target computing unit set comprises: Based on the second instantaneous resource potential of each computing unit in the target unit set, the target processing type of each computing unit is determined; The target processing types include: high computing power type, high bandwidth type, and high memory type; For each of the target expert networks, the resource bottleneck type of each target expert network is determined based on the computation weight, memory weight, and communication weight when each expert network performs the assigned task; the resource bottleneck type includes: computation type, memory access type, and communication type. Based on the resource bottleneck type of each target expert network, in the target computing unit set, computing units belonging to the processing type that matches the resource bottleneck type are selected as adjusted computing units for running the target expert network; wherein, the high computing power type matches the computing type, the memory access type matches the high memory type, and the high bandwidth type matches the communication type.

10. An electronic device comprising a plurality of hardware components and a processor, wherein a target model is running on the electronic device; The processor is configured to: determine a first instantaneous resource potential based on the plurality of hardware parameters; the first instantaneous resource potential is used to characterize the instantaneous comprehensive hardware remaining capacity of the electronic device for running the target model; Based on the first instantaneous resource potential, a first adjustment factor is determined for adjusting the batch size of the target model; The first instantaneous resource potential is positively correlated with the first adjustment factor; The target batch size of the target model in the current scheduling cycle is determined based on the first adjustment factor; In the current scheduling cycle, the target model performs model inference based on the target batch size to match the processing capability of the electronic device.

Images