Simulation method and device for artificial intelligence inference system performance and computer readable storage medium

Abstract

The invention relates to the technical field of artificial intelligence, for example, relates to a simulation method and device for artificial intelligence reasoning system performance and a computer readable storage medium. The simulation method comprises the following steps: receiving an inference request, wherein the inference request comprises batch size and priority attributes; generating pre-filling nodes and decoding nodes corresponding to the requests according to attributes of the reasoning requests and predefined node mapping rules, and constructing an execution graph containing dependency relationships among the nodes; pre-filling nodes and decoding nodes in the execution graph are simulated respectively, memory occupation changes in the execution process are inferred, and a system memory consumption curve is generated; and performing memory conflict detection and scheduling on the simulation execution process of the execution graph according to the system memory consumption curve and the scheduling strategy, and outputting a performance index based on the scheduled execution result. By introducing a memory dynamic calculation and conflict resolution mechanism in a simulation process, the accuracy of dynamic request reasoning performance prediction in a limited memory environment is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and in particular to a simulation method and device for performance of an artificial intelligence inference system, and a computer readable storage medium. BACKGROUND

[0002] In actual operation scenarios of the artificial intelligence inference system, the inference request has significant dynamics, which mainly manifests in the following aspects: request batch dynamics, a large number of requests need to be processed for high-throughput inference at peak times, and high-quality low-delay inference can be performed at request trough times; request priority changes with scenarios, for example, in high-speed driving of a vehicle-mounted AI, lane deviation detection has the highest priority, and entertainment content recommendation has the lowest priority, while the priorities are reversed in low-speed parking.

[0003] However, the existing performance simulation method has the problem that the performance simulation is not sensitive to memory constraints: the dynamics of the request brings significant memory dynamics, and under limited memory constraints, the memory dynamics directly affect the performance level, for example, when the edge device has insufficient display memory, multiple requests will be blocked due to memory conflicts, resulting in a large deviation between the simulation results and the real running environment, and the simulation results cannot effectively guide system optimization.

[0004] It should be noted that the information disclosed in the above background section is only used to strengthen the understanding of the background of the present application, and therefore can include information that does not constitute prior art known to those of ordinary skill in the art. SUMMARY

[0005] To provide a basic understanding of some aspects of the disclosed embodiments, a brief summary is given below. The summary is not an overall description of the application, nor is it intended to identify key / important elements or delineate the scope of the embodiments, but to serve as a prelude to the detailed description below.

[0006] The embodiments of the present disclosure provide a simulation method and device for performance of an artificial intelligence inference system, and a computer readable storage medium, which introduce memory dynamic calculation and conflict resolution mechanisms in the simulation process, and improve the accuracy of dynamic request inference performance prediction under limited memory environment.

[0007] In some embodiments, a method for simulating performance of an artificial intelligence inference system is provided, comprising: receiving an inference request, the inference request comprising a batch size and a priority attribute; generating pre-filling nodes and decoding nodes corresponding to each request according to the attributes of the inference request and a predefined node mapping rule, and constructing an execution graph comprising inter-node dependencies; simulating the pre-filling nodes and the decoding nodes in the execution graph respectively, the memory occupancy changes in the inference execution process, and generating a system memory consumption curve; performing memory conflict detection and scheduling on the simulated execution process of the execution graph according to the system memory consumption curve and a scheduling strategy, and outputting performance indicators based on the execution results after scheduling, with the device memory upper limit as a constraint.

[0008] Optionally, the attributes of the inference request comprise a pre-filling batch size and a decoding batch size, and the node mapping rule comprises determining the computational attribute and the basic memory occupancy attribute of the corresponding node according to the batch size and the structural parameters of the large language model.

[0009] Optionally, generating the pre-filling nodes and the decoding nodes corresponding to each request according to the attributes of the inference request and the predefined node mapping rule, and constructing an execution graph comprising inter-node dependencies, comprises: parsing the inference request to extract the pre-filling batch size, the decoding batch size, and the priority in the inference request; calculating and generating the pre-filling nodes according to the pre-filling batch size, the decoding batch size, and the model parameters of the large language model through the node mapping rule, wherein the computational attribute of the pre-filling node is positively correlated with the pre-filling batch size; generating one or more decoding node sequences according to the decoding batch size and the model parameters through the node mapping rule, wherein each decoding node corresponds to the generation of a single token, and the computational attribute of the decoding node is positively correlated with the decoding batch size; establishing the dependency relationship between the pre-filling node and all the decoding nodes corresponding thereto to generate the execution graph.

[0010] Optionally, simulating the pre-filling nodes and the decoding nodes in the execution graph respectively, the memory occupancy changes in the inference execution process, and generating a system memory consumption curve, comprises: for each pre-filling node, accumulating the occupancy of the model parameter memory, the initial key-value cache memory, and the intermediate feature memory when the node starts executing; releasing the occupancy of the intermediate feature memory when the node execution ends; for each decoding node, simulating its memory occupancy according to the token generation time sequence: at the beginning of each iteration, accumulating the occupancy of the newly added key-value cache memory and the intermediate feature memory corresponding to the current token; at the end of each iteration, releasing the occupancy of the intermediate feature memory and retaining the accumulated key-value cache memory; based on the time sequence changes of the memory occupancy of all nodes, generating a system memory consumption curve by time axis aggregation; wherein the system memory consumption curve takes time as the horizontal axis, the total system memory occupancy as the vertical axis, and marks the start and end execution time points of each node.

[0011] Optionally, the memory conflict detection and scheduling of the simulated execution process of the execution graph is performed according to the system memory consumption curve and the scheduling strategy, with the device memory upper limit as a constraint, including: determining a set of candidate nodes that can be executed currently based on the dependency relationship of the nodes in the execution graph; selecting a to-be-executed node from the set of candidate nodes according to the scheduling strategy; obtaining the memory peak demand of the to-be-executed node during execution according to the system memory consumption curve; judging whether the sum of the currently occupied system memory and the memory peak demand is greater than the device memory upper limit value; if less than the device memory upper limit value, simulating the execution of the node, updating the currently occupied system memory and the current simulation time, and marking the node as completed; if greater than the device memory upper limit value, adjusting the node execution order or execution mode according to the conflict resolution strategy, and re-performing scheduling decision.

[0012] Optionally, the conflict resolution strategy includes at least one of the following: splitting the batch size of the to-be-executed node to generate multiple sequentially executed sub-nodes; delaying the execution of the to-be-executed node and preferentially executing other nodes in the set of candidate nodes with smaller memory demand; preferentially executing a pre-padding node corresponding to the to-be-executed node when the to-be-executed node is a decoding node and the pre-padding node has not been executed; pausing the execution of a node corresponding to a low-priority request according to the request priority, and releasing memory for a node of a high-priority request.

[0013] Optionally, performance indicators are output based on the scheduled execution results, including: recording the output time of the first word piece of each inference request to calculate the first word piece delay; recording the output time of the last word piece of each inference request to calculate the end-to-end delay; counting the number of inference requests completed per unit time to calculate the system throughput; and generating a memory utilization curve by counting the memory usage rate over time.

[0014] Optionally, the scheduling strategy includes: a first priority and a second priority, the first priority having a higher level than the second priority, and the first priority being a priority scheduling pre-padding node; and the second priority being a scheduling decoding node according to the request priority when the pre-padding node is not executable.

[0015] In some embodiments, a device for simulating the performance of an artificial intelligence inference system is provided, including a processor and a memory storing program instructions, the processor being configured to execute the program instructions when running, to perform the method for simulating the performance of an artificial intelligence inference system as described in any of the above embodiments.

[0016] In some embodiments, a computer-readable storage medium is provided, storing program instructions, the program instructions being used to cause a computer to execute the method for simulating the performance of an artificial intelligence inference system as described in any of the above embodiments when running.

[0017] The simulation method and device for artificial intelligence inference system performance provided by the embodiments of the present disclosure, the computer readable storage medium can realize the following technical effects: (1) The present application introduces the upper limit of the device memory as a physical constraint in the simulation process, and performs real-time memory conflict detection and resolution during execution, thereby improving the consistency of the simulated execution process and the behavior on the actual hardware. The latency and throughput calculated in this way include the queuing and fragmentation overhead caused by insufficient memory, thereby significantly reducing the simulation bias and making the predicted value closer to the actual measured value.

[0018] (2) Through dynamic execution graph generation, the real-time changes of the request rate and batch size can be responded to. Through node-level differentiated memory simulation, especially the accurate modeling of the cumulative effect of the key value cache memory in the decoding stage, the slow growth and pressure of the memory in the long text generation or high concurrency scenario can be accurately reflected. Combined with the two, the influence of complex dynamic scenarios such as burst traffic and mixed requests of different priorities on the stability and performance of the system can be evaluated.

[0019] (3) The simulation method of the present application not only outputs performance indicators, but also outputs intermediate results such as memory occupation curves and node scheduling time sequence diagrams. Developers can clearly see whether the performance bottleneck is due to the excessively high pre-filling node memory peak value or the excessively fast accumulation of the key value cache memory of the decoding node. Therefore, targeted optimization can be performed, such as adjusting the fragmentation strategy and optimizing the request batch size. In this way, the present application can not only realize performance testing, but also realize optimization of the simulation method.

[0020] The foregoing general description and the following description are only exemplary and explanatory, and are not intended to limit the present application. BRIEF DESCRIPTION OF DRAWINGS

[0021] One or more embodiments are exemplarily illustrated by corresponding drawings, which do not constitute limitations on the embodiments, elements with the same reference numerals in the drawings are shown as similar elements, the drawings do not constitute proportional limits, and wherein: Figure 1 is a schematic diagram of a simulation method for artificial intelligence inference system performance provided by an embodiment of the present disclosure; Figure 2 is a schematic diagram of a method for constructing an execution graph containing inter-node dependencies provided by an embodiment of the present disclosure; Figure 3 is a schematic diagram of a method for generating a system memory consumption curve provided by an embodiment of the present disclosure; Figure 4 is a schematic diagram of a method for memory conflict detection and scheduling of the simulation execution process of the execution graph according to the system memory consumption curve and the scheduling strategy provided by an embodiment of the present disclosure; Figure 5 is a schematic diagram of a method for simulating performance of an artificial intelligence inference system according to another embodiment of the present disclosure; Figure 6 is a schematic diagram of a device for simulating performance of an artificial intelligence inference system according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0022] In order to enable a more detailed understanding of the features and technical content of the embodiments of the present disclosure, the implementation of the embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings, which are only used for reference and do not limit the embodiments of the present disclosure. In the following technical description, in order to facilitate explanation, a plurality of details are provided to provide a full understanding of the disclosed embodiments. However, one or more embodiments can still be implemented without these details. In other cases, well-known structures and devices can be simplified to facilitate the drawings.

[0023] The terms "first", "second", and the like in the specification and claims of the embodiments of the present disclosure and the above-described drawings are used to distinguish similar objects, and do not necessarily describe a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present disclosure described herein can be implemented. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion.

[0024] Unless otherwise specified, the term "a plurality of" means two or more.

[0025] In the embodiments of the present disclosure, the character " / " represents an "or" relationship between the objects before and after it. For example, A / B represents: A or B.

[0026] The term "and / or" is a description of the association relationship between objects, which means that there can be three relationships. For example, A and / or B means: A or B, or, A and B, the three relationships.

[0027] The term "corresponding" can refer to an association relationship or a binding relationship. A and B correspond to each other means that there is an association relationship or a binding relationship between A and B.

[0028] In some embodiments, in conjunction with Figure 1 As shown, a method for simulating performance of an artificial intelligence inference system is provided, comprising: S101, receiving an inference request, the inference request including a batch size and a priority attribute.

[0029] Optionally, the attributes of the inference request include a pre-filled batch size and a decoded batch size.

[0030] The pre-filling batch size refers to the number of parallel requests for processing input prompts. The decoding batch size refers to the number of parallel requests for generating output tokens.

[0031] The node is caused to generate a node matching the request size and the model structure, to provide an accurate node attribute basis for subsequent memory simulation and scheduling.

[0032] S102, according to the attribute of the inference request and the pre-defined node mapping rule, a pre-filling node and a decoding node corresponding to each request are generated, and an execution graph containing the dependency relationship between nodes is constructed.

[0033] Optionally, the node mapping rule includes determining the computing amount attribute and the basic memory occupation attribute of the corresponding node according to the batch size and the structure parameter of the large language model.

[0034] Through the node mapping rule, the batch size of each request is associated with the structure parameter of the model for calculation. This enables the computing amount and the basic memory occupation of each generated node to accurately reflect the actual load of the request.

[0035] S103, respectively simulating the pre-filling node and the decoding node in the execution graph, the change of memory occupation in the inference execution process, and generating a system memory consumption curve.

[0036] In view of the completely different memory behavior characteristics of the pre-filling stage and the decoding stage, the memory occupation is monitored respectively, and the dynamic change on the time axis is simulated to generate a memory consumption curve. By timing simulation and aggregation to generate a system memory consumption curve, the instantaneous peak value of memory occupation and the trend of growth over time in the scene of long text generation, high concurrency request, etc. can be captured. In this way, it can be identified whether the performance decline is due to the peak memory of a single pre-filling node being too high or due to the accumulation of key-value cache memory caused by multiple decoding nodes being concurrent to reach the memory upper limit, so as to perform targeted optimization.

[0037] S104, taking the device memory upper limit as a constraint, according to the system memory consumption curve and the scheduling strategy, performing memory conflict detection and scheduling on the simulated execution process of the execution graph, and outputting performance indicators based on the execution result after scheduling.

[0038] The simulation method provided by the present disclosure receives an inference request containing a batch size and a priority. Based on the attributes of the inference request and the predefined node mapping rules, an execution graph is generated, which can reflect the real-time, concurrency and diversity of the request arrival. Further, by distinguishing the memory occupation changes of the simulation pre-filled nodes and the decoding nodes, and performing conflict detection and scheduling with the upper limit of the device memory as a hard constraint, the queuing, blocking, fragmentation and other phenomena caused by memory resource contention in the real hardware environment can be reproduced. Therefore, the performance indicators of the simulation output are no longer ideal values, but are highly close to the performance data actually measured on a device with a specific memory capacity, overcoming the defects of existing simulation methods that are detached from actual memory constraints, and the prediction deviation is greatly reduced.

[0039] Further, the scheduling strategy and the conflict resolution mechanism are taken as configurable inputs and are simulated in a closed loop with the memory constraints. The effectiveness of different scheduling strategies under memory pressure is tested. The trade-off effect of different batch sizes on throughput and delay is evaluated. With the upper limit of the device physical memory as a constraint, memory conflicts are dynamically detected during simulation execution, and the preset scheduling strategy is applied for conflict resolution, thereby simulating the real execution timing and performance indicators under actual resource competition.

[0040] In this way, various system design options can be quantitatively evaluated and verified before actual deployment, significantly reducing the trial-and-error cost and accelerating the optimization iteration cycle of the inference system.

[0041] Optionally, in combination with Figure 2 As shown, according to the attributes of the inference request and the predefined node mapping rules, pre-filled nodes and decoding nodes corresponding to each request are generated, and an execution graph containing inter-node dependencies is constructed, including: S201, parsing the inference request, extracting the pre-filled batch size, decoding batch size and priority in the inference request.

[0042] S202, according to the pre-filled batch size, decoding batch size and model parameters of the large language model, the pre-filled nodes are calculated and generated through the node mapping rules.

[0043] The calculation amount attribute of the pre-filled node is positively correlated with the pre-filled batch size; S203, according to the decoding batch size and the model parameters, one or more decoding node sequences are generated through the node mapping rules.

[0044] Each decoding node corresponds to the generation of a single token, and the calculation amount attribute of the decoding node is positively correlated with the decoding batch size. The decoding node sequence refers to the generation step of each decoding node corresponding to an output token, and the sequence length is equal to the output length.

[0045] S204: Establish the dependency relationships between the pre-filled nodes and all their corresponding decoded nodes to generate the execution graph.

[0046] In this context, the dependency relationship is that the pre-filled node is the predecessor node of all its decoded nodes.

[0047] In this embodiment, the pre-filled batch size, decoding batch size, and priority, which affect performance, are extracted from the inference requests. Using node mapping rules, a single pre-filled node corresponding to the pre-filled batch size and a series of decoding nodes corresponding to the decoding batch size are generated. Predecessor-successor dependencies are established from the pre-filled nodes to all their decoding nodes. The generated new nodes and dependencies can be incrementally added to the global execution graph. The execution graph's data structure supports efficient node addition and dependency maintenance. This enables the simulation system to handle continuous request flows with low overhead and reflect dynamic changes in system state in real time. Whether simulating a load test lasting several hours or a sudden request spike, this solution improves the efficiency and scalability of the simulation process, meeting the needs of large-scale, long-term simulations in practical applications.

[0048] Furthermore, in LLM (Large Language Model) inference, the pre-filling and decoding stages differ fundamentally in computational density, memory access patterns, and execution time. Related simulation methods treat the pre-filling and decoding stages as the same task, leading to distorted simulation results. This solution sets independent parameter drivers for the pre-filling and decoding stages: the pre-filling node is driven by the pre-filling batch size, and the decoding node is driven by the decoding batch size. This allows for independent variation of both, enabling a differentiated strategy and improving simulation accuracy.

[0049] Furthermore, the decoding stage is modeled as a sequence of nodes, rather than a single, unified node. This decomposition allows the simulation to track the gradual accumulation of memory and the iterative computation at the word level, enabling high-precision dynamic memory simulation and fine-grained scheduling. In this way, the simulation can not only distinguish between the two types of tasks but also perform micro-simulations during the decoding stage, thus accurately predicting complex phenomena such as memory exhaustion caused by long text generation or the impact of mixing different decoding batch sizes on cross-scheduling.

[0050] In some examples, the steps involved in generating a dynamic execution graph include: Receive a dynamic request stream, including inference request attributes, LLM model parameters, and device hardware parameters. Inference request attributes include the pre-filled batch size, decoding batch size, and priority for each request. LLM model parameters include the number of layers, the number of attention heads, and the hidden layer dimension.

[0051] The request is converted into executable computing nodes by applying a node mapping rule. The rule calculates the theoretical computation amount, base memory occupation, and other attributes of the computing nodes based on the model parameters and batch size. It includes: generating a prefill node for each request, representing the initial computation stage of processing its input prompt. Generating one or more decoding nodes for each request, decoding is usually related to the number of generated tokens, which can be modeled as a node sequence, representing the stage of generating output per token. The prefill node is the predecessor node of all decoding nodes it belongs to.

[0052] Further, the newly generated nodes are added to the global dynamic execution graph, which is a directed acyclic graph with data dependency between nodes, such as predecessor-successor relationship.

[0053] The dynamic execution graph evolves over time, containing all prefill nodes and decoding nodes to be executed and their dependencies.

[0054] Optionally, in combination with Figure 3 As shown, respectively simulate the memory occupation changes of prefill nodes and decoding nodes in the execution graph during inference execution, generate system memory consumption curves, including: S301, for each prefill node, when the node starts execution, accumulate the memory occupation of model parameters, initial key-value cache memory, and intermediate feature memory. At the end of node execution, release the occupation of the intermediate feature memory.

[0055] S302, for each decoding node, simulate its memory occupation according to the token generation time sequence iteration: S303, at the beginning of each iteration, accumulate the occupation of the newly added key-value cache memory and intermediate feature memory corresponding to the current token.

[0056] The initial key-value cache memory refers to the key-value cache allocated for the input sequence in the prefill stage, which is proportional to the batch size and input length.

[0057] The newly added key-value cache memory refers to the key-value cache that needs to be added for each token generated in the decoding stage.

[0058] The intermediate feature memory refers to the temporary activated tensor memory during the calculation process, which is released after each step of calculation is completed.

[0059] S304, at the end of each iteration, release the occupation of the intermediate feature memory, and retain the accumulated key-value cache memory.

[0060] S305, based on the time sequence changes of the memory occupation of all nodes, generate system memory consumption curves according to the time axis.

[0061] The system memory consumption curve has time as the horizontal axis and the total system memory occupation as the vertical axis, and marks the start and end execution time points of each node.

[0062] In this embodiment, based on the nodes in the dynamic execution graph, LLM model parameters, and device memory parameters, the detailed memory occupation change of each node from start to end is simulated.

[0063] Specifically, the pre-filled node memory simulation step includes: at startup, loading model parameters, allocating initial key-value cache memory, and allocating intermediate activation memory. During execution, the intermediate activation memory is continuously occupied. After execution is complete, the intermediate activation memory is released, and the model parameters and initial key-value cache memory are retained. The memory peak and release points during this process are recorded.

[0064] The decoding node memory simulation, in the steps of iterating by word units, includes: at the beginning of each iteration, on the basis of the key-value cache memory of the last iteration, additional allocation of new key-value cache memory and intermediate feature memory is performed. After each iteration calculation is complete, the intermediate activation memory of this iteration is released, and the cumulatively increased key-value cache memory is retained.

[0065] Therefore, the total memory occupation of the decoding node is a stepwise rising curve over time, and the main increment comes from the continuously accumulated key-value cache memory.

[0066] By integrating the memory simulation results of all nodes, a global, time-point-specific system total memory consumption curve is generated.

[0067] The memory occupation details of each node and the system total memory consumption curve are output.

[0068] The simulation method in the related art regards memory occupation as a static or average constant, which cannot depict the memory fluctuations caused by the calculation phase and iteration process in LLM inference, resulting in simulation distortion.

[0069] The simulation method provided by the present disclosure distinguishes between node types, respectively simulates the memory occupation of pre-filled nodes and decoding nodes during their respective running processes. The start and end of node execution are used as key event points to trigger memory loading, accumulation and release operations. Persistent memory and transient memory are clearly distinguished. The key-value cache memory is simulated as allocated and retained, while the intermediate feature memory is simulated as transiently allocated and then released. For the decoding stage, the decoding nodes are iteratively simulated by word elements to capture the dynamic step-by-step growth of memory occupation over time. The time axis is aggregated to generate a system memory consumption curve based on the time sequence changes of all node memory occupations. The memory consumption curve can reproduce the memory spikes, memory sawtooth waves and monotonic growth trends observed on real hardware, achieving high-precision and fine-grained dynamic simulation of memory occupation, and breaking through the limitations of static simulation. Moreover, the time-stamped system memory consumption curve contains the theoretical memory requirements of each node at any time. By querying this curve and combining the memory already occupied by other nodes at the current time point, it can be predicted whether executing a new node will cause an overflow, and errors caused by inaccurate memory prediction can be avoided in advance, thereby improving the correctness and stability of the entire simulation process and the credibility of the output performance results.

[0070] Optionally, in combination with Figure 4 As shown in the figure, with the device memory upper limit as a constraint, the simulation execution process of the execution graph is detected for memory conflicts and scheduled according to the system memory consumption curve and the scheduling strategy, including: S401, determining a set of candidate nodes that can be executed based on the dependency relationship of the nodes in the execution graph; S402, selecting a node to be executed from the set of candidate nodes according to the scheduling strategy; Wherein, the set of candidate nodes refers to all nodes whose predecessor nodes have been executed.

[0071] S403, obtaining the memory peak demand of the node to be executed during execution according to the system memory consumption curve; Wherein, the memory peak demand refers to the maximum memory value of the node during execution extracted from the system memory consumption curve.

[0072] S404, judging whether the sum of the currently occupied system memory and the memory peak demand is greater than the device memory upper limit: S405, if less than the device memory upper limit value, simulate the execution of the node, update the system occupied memory and the current simulation time, and mark the node as completed; S406, if greater than the device memory upper limit value, adjust the node execution order or execution mode according to the conflict resolution strategy, and make a new scheduling decision.

[0073] In this embodiment, candidate nodes are determined based on dependency relationships, nodes to be executed are selected according to a scheduling strategy, it is detected whether the memory is over the limit, and conflict resolution is executed or triggered. A simulation scheduling closed loop under limited memory is realized, so that the simulation process can reflect the memory competition and queuing phenomenon in the real system.

[0074] In the related art simulation method, the scheduling process is open-loop and ideal during performance estimation, and cannot reflect the real system behavior caused by insufficient resources, such as task queuing, execution failure, performance degradation, etc.

[0075] The present disclosure constructs a closed-loop, event-driven discrete simulation loop by maintaining two state variables of the current system occupied memory and the current simulation time. At each scheduling time, the node is selected based on the dependency relationship and the strategy, and the upcoming operation is detected in advance by using the system memory consumption curve. According to the detection result, a feedback is formed. That is, it is determined whether the device memory upper limit value is exceeded. If the result is not over the limit, the state is updated and the forward is pushed. If the result is over the limit, the conflict resolution strategy is triggered, and the execution plan is dynamically adjusted, rather than simply assumed to fail or ignored. In this way, through the simulation method provided by the present disclosure, the dynamic memory management in the real reasoning system is reproduced, such as resource application, competition, queuing, fragmentation, priority processing, etc. In this way, the performance indicators of the simulation output contain the queuing waiting time and other non-ideal overheads caused by resource contention. The simulation result is changed from ideal laboratory data to real deployment environment pre-rehearsal, and the prediction confidence is greatly improved.

[0076] Further, the present application takes the scheduling strategy and the conflict resolution strategy as explicit and configurable input modules, and embeds them into the core decision point of the above-mentioned closed-loop simulation. From the candidate node set, the node to be executed is selected according to the scheduling strategy, and the node execution order or execution mode is adjusted according to the conflict resolution strategy, and the scheduling decision is re-made. In this way, under the same request load and memory constraint, different scheduling and conflict resolution combinations are quickly tested, and the influence of different scheduling and conflict resolution combinations on key indicators such as throughput and tail delay is quantitatively compared. The optimal parameters of the selected strategy are found. And verify the effectiveness of a specific strategy, such as high-priority preemption, in guaranteeing the quality of service. Since the simulation process is gradually pushed forward, the whole process of each scheduling decision, conflict occurrence and resolution can be completely observed. In this way, any change in performance results can be traced back to specific scheduling decision events and conflict resolution events. For example, it can be explicitly pointed out that the increase of a certain request delay is due to the delay of its decoding node for several milliseconds due to insufficient memory at a certain time point, and the reason for the delay is to give way to a higher priority pre-filling node.

[0077] That is, by using the simulation method of the present disclosure, high-fidelity dynamic reproduction of the memory resource competition and management system behavior is achieved, making the simulation results have practical guiding significance. Moreover, a scheduling strategy evaluation and optimization platform is provided, greatly reducing the cost and risk of system tuning.

[0078] Optionally, the conflict resolution strategy includes at least one of the following: splitting the batch size of the to-be-executed node to generate multiple sequentially executed sub-nodes; delaying execution of the to-be-executed node and preferentially executing other nodes in the candidate node set with smaller memory requirements; preferentially executing a pre-filling node corresponding to the to-be-executed node when the to-be-executed node is a decoding node and the pre-filling node has not been executed; and pausing execution of a node corresponding to a low-priority request and releasing memory for a node of a high-priority request according to request priorities.

[0079] In this embodiment, when it is judged that the sum of the currently occupied memory and the peak memory requirement of the system is greater than the upper limit of the device memory, a strategy or a combination of strategies is selected according to context information.

[0080] For example, if the main cause is that the node batch is too large, splitting is triggered.

[0081] If there are other small tasks in the system that can be run immediately, delay execution and execute small tasks are triggered.

[0082] If the blocked node is a decoding node and its pre-filling node can be run, dependency priority is triggered.

[0083] If the overrun is caused by a high-priority request that cannot be met, priority preemption is triggered.

[0084] Further, the conflict resolution strategy involves the combination and cooperation of multiple strategies. For example, the system may first attempt to split for a high-priority request; if it still cannot be done, it attempts to delay a node of medium priority; if it still cannot be done, it finally suspends a low-priority node. After the conflict is resolved, scheduling decisions are made again, forming a closed loop of detection, decision, execution, and re-detection until the conflict is resolved.

[0085] Optionally, the batch size of the to-be-executed node is split to generate multiple sequentially executed sub-nodes. Splitting refers to splitting a large batch node into multiple small batch nodes that are sequentially executed.

[0086] In this embodiment, when the peak memory requirement of a single node exceeds the available memory, this strategy logically splits the original node into multiple smaller batch sub-nodes through an algorithm. These sub-nodes inherit the dependency relationship of the original node, but are executed in sequence, thereby adjusting the one-time high memory peak to multiple lower memory occupancies. By adjusting the task granularity to actively adapt to resource constraints, overflow caused by a task size that is too large is avoided.

[0087] Optionally, the execution of the pending node is delayed, and other nodes in the set of candidate nodes with smaller memory requirements are executed in priority. This strategy is an optimization for execution timing. When multiple candidate nodes compete for limited memory, the selection of temporarily shelving large memory nodes that may cause conflicts gives priority to the execution of small memory nodes. The execution and completion of these small nodes will generally release the intermediate feature memory they occupy, thus possibly freeing up space for the subsequent execution of the originally delayed nodes.

[0088] Optionally, when the pending node is a decoding node and its corresponding pre-filling node has not been executed, the pre-filling node is executed in priority; this strategy is also an optimization for execution timing, ensuring that the pre-filling node corresponding to the decoding node is executed in priority to release intermediate memory. This strategy takes advantage of the specific dependency semantics of LLM inference tasks. A decoding node is often waiting because its predecessor pre-filling node has not been executed. Executing the pre-filling node not only advances the overall progress of the request, but more importantly, after the pre-filling node is completed, a large amount of intermediate feature memory will be released, and its memory occupancy will decrease significantly, leaving only model parameter and initial key-value cache memory, which in turn releases more memory for subsequent tasks.

[0089] Optionally, according to the request priority, the execution of nodes corresponding to low-priority requests is suspended to release memory for nodes of high-priority requests. This strategy allows the interruption of low-priority nodes that are being executed in the simulation process, and the released memory is allocated to high-priority nodes. This simulates preemptive scheduling based on service level agreements in real systems, so that the performance goals of critical requests can still be met under extreme resource competition.

[0090] The present disclosure provides a variety of configurable conflict resolution mechanisms that can flexibly cope with memory pressure in different scenarios, improving the practicality and guidance value of the simulation.

[0091] Optionally, performance indicators are output based on the execution results after scheduling, including: The output time of the first word piece of each inference request is recorded, and the first word piece delay is calculated; The output time of the last word piece of each inference request is recorded, and the end-to-end delay is calculated; The number of inference requests completed per unit of time is counted, and the system throughput is calculated; The usage rate of device memory over time is counted, and a memory utilization curve is generated.

[0092] In this embodiment, the first word piece delay reflects the initial response speed of the system to user requests, which is mainly affected by the calculation speed in the pre-filling stage, the scheduling and queuing delay, and the initial memory allocation speed. Optimizing the first word piece delay is crucial to improving the smoothness of interaction.

[0093] End-to-end latency reflects the total time the system takes to complete the entire request. It is a combination of the first token latency and the cumulative time of all decoding steps, which is affected by the single iteration speed of the decoding phase, the decoding length, and the decoding interruption / delay caused by memory contention. Optimizing end-to-end latency is crucial for the completion efficiency of task-based requests.

[0094] By splitting and independently measuring these two latencies, it is possible to determine which phase the performance bottleneck is located in. For example, excessively high first token latency may indicate that pre-filling scheduling or computation is the bottleneck; while normal first token latency but excessively high end-to-end latency points to efficiency problems or resource contention in the decoding phase.

[0095] Throughput is a core indicator of measuring the overall efficiency and economy of the system. In this simulation scheme, the calculation of throughput is based on the actual execution results after scheduling. This means that in the simulation process, the completion time of requests queued, fragmented, or preempted due to memory conflicts will affect the calculation results of throughput.

[0096] Memory utilization curve is a function of time. It intuitively shows the utilization of device memory resources throughout the entire timeline of the simulation. Through the memory utilization curve, it is possible to identify the cause of performance problems with high latency or low throughput. For example: sustained near 100% utilization may cause frequent queuing delays; large amplitude sawtooth waves mean frequent allocation and release overhead of intermediate feature memory; a curve that steadily rises to the top then verifies the memory exhaustion problem caused by the accumulation of key-value cache memory.

[0097] The first token latency refers to the time from the start of the request to the production of the first output token.

[0098] End-to-end latency refers to the time from the start of the request to the production of the last output token.

[0099] System throughput refers to the number of requests or tokens completed per unit of time.

[0100] In some examples, the dynamic execution graph performance simulation steps under memory constraints include: Inputs: dynamic execution graph, node memory details, system total memory consumption curve, device physical memory upper limit, scheduling strategy rules.

[0101] The event-based discrete simulation process includes: Initialization: set the current time t=0, initialize the memory usage to 0, and prepare the ready node queue.

[0102] Event loop: a. Scheduling decision: select the next node to be executed from the ready node according to the scheduling strategy, such as: high priority request node priority, pre-filling node priority to release intermediate active memory as soon as possible.

[0103] b. Memory conflict detection: Check if the sum of the memory peak required to execute this node and the current system occupied memory is greater than the device memory upper limit.

[0104] c. Conflict resolution: If not over limit: Simulate the execution of this node, update the current time t to increase the node calculation delay, and update the memory occupation.

[0105] If over limit: Trigger the resolution mechanism in the scheduling strategy. For example: Node fragmentation: Split a large batch of pre-filled nodes into several small batch nodes for continuous execution to reduce the single memory peak.

[0106] Execution delay: Suspend the current node, and preferentially execute other nodes with small memory requirements or capable of releasing memory.

[0107] Priority preemption: Suspend nodes with low priority requests to give memory to high priority nodes.

[0108] d. State update: After the node is executed, its successor node is added to the ready queue. Record the completion time of the request.

[0109] After the simulation ends, based on the arrival time and completion time of all requests, the performance indicators are obtained. The performance indicators include but are not limited to: first word delay, end-to-end delay, system throughput, and memory utilization curve.

[0110] Output: Report the quantitative performance indicators that the system can achieve under the given memory constraints and scheduling strategy.

[0111] Optionally, the scheduling strategy includes a first priority and a second priority, the first priority is higher than the second priority, and the first priority is preferentially scheduling pre-filled nodes; the second priority is scheduling decoding nodes according to request priority when pre-filled nodes are not executable.

[0112] In this embodiment, the first priority is preferentially scheduling pre-filled nodes, considering that in the life cycle of a reasoning request, the pre-filled stage is a necessary and non-parallel first process. A decoding node must wait for its corresponding pre-filled node to complete before it can start. Therefore, accelerating the completion of pre-filled nodes is the key to shortening the end-to-end delay of the request. Preferentially scheduling pre-filled nodes essentially actively shortens the waiting time of all subsequent tasks, optimizing the overall critical path of the system.

[0113] Also, the pre-filling node occupies a large amount of intermediate feature memory during execution, but after the execution is completed, this part of the memory can be completely released. By executing the pre-filling node preferentially, the high memory occupation can be released as soon as possible, and the high memory occupation is converted into a persistent key-value cache with less occupation, so that the memory is released quickly, and memory space is released for subsequent execution of more tasks, and the turnover rate and overall utilization of the memory are improved. By completing the pre-filling node preferentially, the memory is released, the blocking of subsequent tasks is reduced, the overall throughput potential is improved, and the average waiting time of the task queue is reduced.

[0114] The second priority is to schedule the decoding node according to the request priority when the pre-filling node is not executable. When there is no pre-filling node available for scheduling, for example, the pre-filling nodes are all waiting for memory or are dependent, at this time, a selection needs to be made among multiple decoding nodes. In this way, the request priority is introduced to select the decoding node. The decoding node is scheduled according to the request priority, so that the token generation of the high-priority request can be processed preferentially, thereby meeting the delay service level agreement thereof. Also, in the period when no pre-filling node is executable, if no arbitration is made, it may be occupied by a low-priority decoding task. Through the second priority, a differentiated service queue is determined, so that the decoding node is scheduled according to the request priority, and a fair allocation of resources among different business value requests is realized.

[0115] The second priority makes the decoding stage of the high-priority request not be preempted by the low-priority request. This makes it possible to quantitatively evaluate and guarantee the tail delay of the high-priority request in simulation, so as to meet the predictability requirement of the service level in the production environment.

[0116] In some embodiments, in combination with Figure 5 As shown, a simulation method for performance of an artificial intelligence inference system is provided, comprising: S501, receiving a dynamic inference request.

[0117] In this step, one or more inference requests are received. Each request contains at least the following attributes: pre-filling batch size, decoding batch size, priority. The pre-filling batch size refers to the number of parallel requests for processing input prompts. The decoding batch size refers to the number of parallel requests for generating output tokens. The priority represents the level of importance of the request execution.

[0118] S502, dynamically constructing an execution graph.

[0119] In this step, each arriving inference request is parsed, and the pre-filling batch size, decoding batch size, and priority in the inference request are extracted.

[0120] According to the extracted attributes and model parameters of the large language model, the model parameters include but are not limited to: number of layers, hidden dimensions, and pre-filled nodes and decoding node sequences are generated for each request through a predefined node mapping rule. The calculation amount of the pre-filled node is positively correlated with the pre-filled batch size. Each node in the decoding node sequence corresponds to the generation of an output token, and the calculation amount is positively correlated with the decoding batch size.

[0121] A predecessor-successor dependency relationship is established between the pre-filled node and all its decoding nodes.

[0122] All newly generated nodes and their dependencies are incrementally added to a global directed acyclic graph to form a dynamic execution graph that evolves over time.

[0123] S503, node-level differentiated memory dynamic simulation.

[0124] In this step, for each pre-filled node in the execution graph, simulate its memory occupation in time sequence: When the node starts to execute, accumulate the occupation of the model parameter memory, the initial key-value cache memory and the intermediate feature memory.

[0125] When the node execution ends, release the occupation of the intermediate feature memory.

[0126] For each decoding node in the execution graph, simulate its memory occupation in token generation time sequence iteration: At the beginning of each iteration, accumulate the occupation of the newly added key-value cache memory and the intermediate feature memory corresponding to the current token.

[0127] At the end of each iteration, release the occupation of the intermediate feature memory and retain the accumulated key-value cache memory.

[0128] Based on the time sequence changes of the memory occupation of all nodes, aggregate along the time axis to generate a system memory consumption curve. The curve takes time as the horizontal axis, system total memory occupation as the vertical axis, and marks the start and end time points of each node.

[0129] S504, conflict detection and scheduling simulation under memory constraints.

[0130] Specifically, first, the initialization simulation state is performed, including setting the current simulation time, the system currently occupied memory, initially 0, and preparing the ready node queue.

[0131] Enter the discrete event simulation loop driven by events, including: Based on the dependency relationship of the execution graph, add all nodes whose predecessor nodes have been completed to the current executable candidate node set.

[0132] Apply a two-level hybrid scheduling strategy: First priority: preferentially select a pre-filled node from the candidate set for execution.

[0133] Second priority: if no pre-filled node is available, select a decoding node from the candidate set for execution according to the request priority.

[0134] According to the system memory consumption curve, the memory peak required by the selected node to be executed during its execution is queried. It is judged whether the sum of the current occupied memory and the node memory peak requirement is greater than the upper limit value of the device memory.

[0135] If the sum of the current occupied memory and the node memory peak requirement is less than the upper limit value of the device memory, the node is simulated for execution. The current simulation time is advanced, the system occupied memory is updated according to the memory curve, and the node is marked as completed.

[0136] If the sum of the current occupied memory and the node memory peak requirement is greater than the upper limit value of the device memory, a conflict resolution strategy is triggered, and at least one of the following means is selected for execution: The to-be-executed node is batched and fragmented to generate multiple sequentially executed sub-nodes.

[0137] The node is delayed for execution, and other nodes with smaller memory requirements are selected from the candidate set.

[0138] If the to-be-executed node is a decoding node and its pre-filled node has not been executed, the pre-filled node is preferentially executed.

[0139] According to the request priority, the execution of the node of the low-priority request is suspended to release the memory.

[0140] After applying the conflict resolution strategy, return to step: apply the two-level hybrid scheduling strategy and make scheduling decisions again.

[0141] After each node is completed, its successor node is added to the ready queue. The output time of the first word element and the tail word element of the request to which the node belongs is recorded.

[0142] Repeat the above steps until all nodes in the execution graph are simulated for execution.

[0143] S505, performance index statistics and output.

[0144] After the simulation is completed, based on the arrival and completion time of all requests and the system resource usage record, the following performance indicators are calculated and output: First word element delay: the time from the start of each request to the generation of the first output word element.

[0145] End-to-end delay: the time from the start of each request to the generation of the last output word element.

[0146] System throughput: the number of inference requests completed or the total number of generated tokens per unit of time.

[0147] Memory utilization curve: a curve of device memory occupancy over time.

[0148] In combination Figure 6 As shown in the figure, the embodiment of the disclosure provides a simulation device 60 for the performance of an artificial intelligence inference system, which includes a processor 600 and a memory 601. Optionally, the device 60 can also include a communication interface 602 and a bus 603. Wherein, the processor 600, the communication interface 602, and the memory 601 can complete mutual communication through the bus 603. The communication interface 602 can be used for information transmission. The processor 600 can call the logical instructions in the memory 601 to execute the simulation method for the performance of the artificial intelligence inference system in the above-mentioned embodiment.

[0149] In addition, the logical instructions in the memory 601 described above can be implemented in the form of a software function unit and sold or used as an independent product, which can be stored in a computer readable storage medium.

[0150] The memory 601 as a computer readable storage medium can be used to store software programs, computer executable programs, such as program instructions / modules corresponding to the method in the embodiment of the disclosure. The processor 100 executes the program instructions / modules stored in the memory 601, thereby performing function applications and data processing, that is, implementing the simulation method for the performance of the artificial intelligence inference system in the above-mentioned embodiment.

[0151] The memory 601 can include a program storage area and a data storage area, wherein the program storage area can store an operating system and at least one application required by a function; the data storage area can store data created according to the use of the terminal device, etc. In addition, the memory 601 can include a high-speed random access memory, and can also include a non-volatile memory.

[0152] The embodiment of the disclosure provides an electronic device, which includes a device body and the simulation device for the performance of the artificial intelligence inference system described above. The simulation device for the performance of the artificial intelligence inference system is installed on the device body. The installation relationship described herein is not limited to being placed in the interior of the device body, but also includes installation connection with other components of the electronic device, including but not limited to physical connection, electrical connection or signal transmission connection, etc. Those skilled in the art can understand that the simulation device for the performance of the artificial intelligence inference system can be adapted to a feasible device body, and thus other feasible embodiments can be realized.

[0153] In some embodiments, a computer readable storage medium storing program instructions is provided, wherein the program instructions, when executed, cause a computer to perform the method for simulating performance of an artificial intelligence inference system according to any one of the above embodiments.

[0154] The technical solutions of the embodiments of the present disclosure can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes one or more instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method disclosed in the embodiments of the present disclosure. The aforementioned storage medium can be a non-transitory storage medium, such as a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0155] The above description and drawings sufficiently illustrate the embodiments of the present disclosure to enable one skilled in the art to practice them. Other embodiments can include structural, logical, electrical, process, and other changes. The embodiments represent only a few of the possible variations. Individual components and functions are optional unless explicitly required, and the order of operations can be changed. Parts and features of some embodiments can be included or replaced by parts and features of other embodiments. Also, the words used in this application are used only to describe the embodiments and not to limit the claims. As used in the description of the embodiments and the claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. Similarly, the term "and / or" as used in this application refers to any and all possible combinations of one or more associated listed items. In addition, when used in this application, the term "comprise" and its variants "comprises" and / or comprises" and the like mean the presence of stated features, integers, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. Without more limitations, the element defined by the phrase "comprising a" does not exclude the presence of additional identical elements in the process, method, or device that includes the stated element. In this document, each embodiment focuses on the differences from other embodiments, and the same or similar parts between embodiments can be referred to each other. For the method, product, etc. disclosed in the embodiments, if it corresponds to the method part disclosed in the embodiments, the relevant part can be referred to the description of the method part.

[0156] Those skilled in the art can understand that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized by electronic hardware or a combination of computer software and electronic hardware. Whether the functions are realized in hardware or software depends on the specific application and design constraints of the technical solution. The skilled person can use different methods for each specific application to realize the described functions, but such implementation should not be considered beyond the scope of the embodiments of the present disclosure. The skilled person can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described system, device and unit can refer to the corresponding processes in the foregoing method embodiments, which will not be repeated here.

[0157] In the embodiments disclosed herein, the disclosed methods, products (including but not limited to devices, equipment, etc.) can be implemented in other ways. For example, the above-described device embodiments are only schematic, for example, the division of the units can only be a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interface, device or unit, and can be electrical, mechanical or other forms. The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, that is, they can be located in one place, or can be distributed on a plurality of network units. Part or all of the units can be selected according to actual needs to implement the embodiments. In addition, each functional unit in the embodiments of the present disclosure can be integrated in one processing unit, or each unit can be a physically independent unit, or two or more units can be integrated in one unit.

[0158] The computer program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other processing device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other processing device to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

Claims

1. A method for simulating the performance of an artificial intelligence inference system, characterized in that, include: Receive inference requests, which include batch size and priority attributes; Based on the attributes of the inference request and the predefined node mapping rules, pre-filled nodes and decoding nodes corresponding to each request are generated, and an execution graph containing the dependencies between nodes is constructed. Simulate the memory usage changes of pre-filled nodes and decoding nodes in the execution graph during the inference execution process to generate system memory consumption curves; Constrained by the device's memory limit, and based on the system memory consumption curve and scheduling strategy, the simulated execution process of the execution graph is subjected to memory conflict detection and scheduling, and performance indicators are output based on the execution results after scheduling.

2. The method according to claim 1, characterized in that, The attributes of the inference request include the pre-filled batch size and the decoding batch size. The node mapping rules include determining the computational cost and basic memory usage attributes of the corresponding node based on the batch size and the structural parameters of the large language model.

3. The method according to claim 2, characterized in that, Based on the attributes of the inference request and predefined node mapping rules, pre-filled nodes and decoding nodes corresponding to each request are generated, and an execution graph containing the dependencies between nodes is constructed, including: Parse the inference request and extract the pre-filled batch size, decoding batch size, and priority from the inference request; Based on the pre-filled batch size, the decoding batch size, and the model parameters of the large language model, pre-filled nodes are calculated and generated through node mapping rules. The computational cost of pre-filled nodes is positively correlated with the pre-filled batch size. Based on the decoding batch size and model parameters, one or more decoding node sequences are generated through node mapping rules, wherein each decoding node corresponds to the generation of a single word, and the computational complexity of the decoding node is positively correlated with the decoding batch size; Establish the dependencies between pre-filled nodes and all their corresponding decoded nodes to generate an execution graph.

4. The method according to any one of claims 1 to 3, characterized in that, Simulate the memory usage changes of pre-filled nodes and decoding nodes in the execution graph during inference execution to generate system memory consumption curves, including: For each pre-filled node, when the node starts execution, the memory occupied by model parameters, initial key-value cache, and intermediate features is accumulated; when the node execution ends, the memory occupied by the intermediate features is released. For each decoding node, its memory usage is simulated iteratively according to the word generation sequence: At the beginning of each iteration, the memory usage of the newly added key-value cache and intermediate feature corresponding to the current word element is accumulated; At the end of each iteration, release the memory occupied by intermediate features and retain the accumulated key-value cache memory; Based on the time-series changes in memory usage across all nodes, a system memory consumption curve is generated by aggregating the data along the time axis. The system memory consumption curve is plotted with time on the horizontal axis and total system memory usage on the vertical axis, marking the start and end times of each node.

5. The method according to any one of claims 1 to 3, characterized in that, Constrained by the device's memory limit, and based on the system memory consumption curve and scheduling policy, memory conflict detection and scheduling are performed on the simulated execution process of the execution graph, including: Based on the dependencies between nodes in the execution graph, determine the set of candidate nodes that can be executed at present; Select the node to be executed from the candidate node set according to the scheduling strategy; Based on the system memory consumption curve, obtain the peak memory requirement of the node to be executed during the execution period; Determine if the sum of the current system memory usage and the peak memory demand exceeds the device's memory limit: If the value is less than the device's memory limit, then the node will be simulated and executed, the system's occupied memory and the current simulation time will be updated, and the node will be marked as completed. If the value exceeds the device's memory limit, the node execution order or execution method will be adjusted according to the conflict resolution strategy, and a new scheduling decision will be made.

6. The method according to claim 5, characterized in that, Conflict resolution strategies include at least one of the following: The batch size of the nodes to be executed is divided into pieces, generating multiple child nodes to be executed sequentially; Delay the execution of nodes to be executed, and prioritize the execution of other nodes with lower memory requirements in the candidate node set; When the node to be executed is a decoding node and its corresponding pre-filled node has not yet been executed, the pre-filled node will be executed first. Based on request priority, pause the execution of nodes corresponding to low-priority requests to free up memory for nodes with high-priority requests.

7. The method according to claim 5, characterized in that, The performance metrics output based on the execution results after scheduling include: Record the output time of the first word in each inference request and calculate the first word delay; Record the output time of the last word in each inference request and calculate the end-to-end latency; Count the number of inference requests completed per unit of time and calculate the system throughput; The system calculates the usage rate of device memory over time and generates a memory utilization curve.

8. The method according to any one of claims 1 to 3, characterized in that, The scheduling strategy includes a first priority and a second priority. The first priority is higher than the second priority. The first priority prioritizes scheduling pre-filled nodes. The second priority schedules decoding nodes according to request priority when pre-filled nodes are not executable.

9. A simulation device for the performance of an artificial intelligence inference system, comprising a processor and a memory storing program instructions, characterized in that, The processor is configured to, when executing the program instructions, perform the simulation method for the performance of an artificial intelligence inference system as described in any one of claims 1 to 8.

10. A computer-readable storage medium storing program instructions, characterized in that, When the program instructions are executed, they cause the computer to perform the simulation method for the performance of an artificial intelligence inference system as described in any one of claims 1 to 8.

Images