Large model inference optimization method, system, medium, terminal and program product for ascension processor

Abstract

The application provides a large model inference optimization method, system, medium, terminal and program product for Ascend processors. The method comprises: comparing the automatically detected version with a built-in compatibility table; if the versions do not match, automatically loading a runtime patch or calling a backup library; if the versions match, processing a user request based on a preset hybrid scheduling mechanism; dynamically selecting a MindIE inference framework or a vLLM inference framework based on a runtime adaptation layer according to the user request parameters; converting large model weights into low-bit weights and compensating based on an error compensation matrix; identifying subgraphs and fusing operators for a model graph according to the computing characteristics of the Ascend processor, and rearranging intermediate tensors according to the 512B memory access alignment rule of the Ascend processor; performing pipelined overlap processing on the matrix multiplication calculation of the current token and the full aggregation operation based on an asynchronous execution flow; and performing inference and generating an inference result according to the user request. The application can improve the efficiency and reliability of large model inference.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a method, system, medium, terminal and program product for large model inference optimization for Ascend processors. Background Technology

[0002] As the parameter scale of Large Language Models (LLMs) continues to grow, higher demands are placed on underlying computing power, memory bandwidth, and system scheduling capabilities. The Ascend 910B NPU (Neural Network Processing Unit) boasts significant advantages due to its high-density Cube matrix computation units, HBM's high bandwidth, and the HCCL / HDC high-speed interconnect mechanism. However, limited by its dedicated compilation link (CANN / Ascend GE), memory management, and runtime interface, which differ from the GPU (Graphics Processing Unit) architecture, existing LLM inference solutions still face the following architectural bottlenecks and engineering challenges when fully leveraging the advantages of the Ascend 910B architecture:

[0003] (1) The performance of dedicated operators is difficult to be stable and efficient: The tensor layout, memory access alignment, and fusion strategy of core operators of large models (such as Attention and RoPE) must strictly follow Ascend's 512B memory access alignment rules and Cube cell scheduling characteristics. However, existing frameworks are unable to eliminate memory access conflicts and pipeline interruptions at the architecture level.

[0004] (2) Architectural conflict in memory resource allocation: The memory management mechanism of Ascend 910B has strict requirements for continuity, and fine-grained memory fragmentation is easy to occur. The dynamic expansion feature of KV Cache in LLM inference conflicts with the strict memory management mechanism of NPU, resulting in inefficient allocation of KV Cache blocks and page table management.

[0005] (3) Asynchronousness of HCCL cross-card communication and LLM incremental inference: LLM's token-level incremental inference causes cross-card communication mode to be characterized by small batches, high frequency and periodic fluctuations. However, existing frameworks are difficult to achieve fine-grained, low-latency pipelined scheduling in the computation-communication cycle.

[0006] (4) Containerized deployment lacks an automated compatibility mechanism for the Ascend environment: The Ascend ecosystem is strongly coupled with driver version, firmware version and CANN acceleration library version. However, existing container images lack automated, non-intrusive detection and dynamic compatibility mechanisms for NPU devices and dependent versions, which seriously affects the robustness and maintainability of the production environment.

[0007] (5) Lack of a unified high-performance service and adaptive scheduling system for Ascend architecture: Existing frameworks are unable to cope with both long text reasoning (requiring MindIE graph optimization) and high-concurrency incremental generation (requiring vLLM PagedAttention) scenarios at the same time, and lack unified abstraction and adaptive scheduling capabilities for dual runtimes (MindIE / vLLM).

[0008] Therefore, it is necessary to provide a large-scale inference optimization method, system, medium, terminal, and program product for Ascend processors to solve the above-mentioned problems in the prior art. Summary of the Invention

[0009] In view of the shortcomings of the prior art described above, the purpose of this application is to provide a method, system, medium, terminal and program product for large model inference optimization for Ascend processors, to solve the technical problems of operator adaptation difficulties, low memory utilization, insufficient cross-card communication efficiency, poor container environment compatibility and weak service capabilities in large model inference scenarios.

[0010] To achieve the above and other related objectives, the first aspect of this application provides a large-model inference optimization method for Ascend processors, comprising: upon container startup, comparing the automatically detected Ascend processor driver version, firmware version, and CANN version with a built-in compatibility table; if the versions do not match, automatically loading a runtime patch or calling a backup library; if the versions match, processing user requests based on a preset hybrid scheduling mechanism; and dynamically selecting the MindIE inference framework or the vLLM inference framework based on the obtained user request parameters and loading the packaged incline. The system implements a large model within a container; it converts the large model weights into low-bit weights based on a low-bit quantization mechanism and compensates for the converted low-bit weights using an error compensation matrix; it performs subgraph identification and operator fusion on the model graph according to the computing characteristics of the Ascend processor to generate an instruction sequence adapted to the Ascend processor, and rearranges intermediate tensors according to the 512B memory access alignment rules of the Ascend processor; it performs pipelined overlapping processing of the matrix multiplication calculation of the current token and the full aggregation operation based on the asynchronous execution flow of the Ascend processor for cross-card communication; it executes inference and generates inference results based on user requests.

[0011] In some embodiments of the first aspect of this application, if the runtime adaptation layer selects the vLLM inference framework, it pre-allocates a physically contiguous key-value cache block pool for the generated key-value cache, and compares the current user request sequence length with a set threshold of the key-value cache block pool. If the current sequence length exceeds the set threshold, it determines whether the key-value block is a hot key-value block or a cold key-value block. If it is a hot key-value block, it is migrated to a high-bandwidth key-value cache block pool; if it is a cold key-value block, it is migrated to a low-bandwidth key-value cache block pool.

[0012] In some embodiments of the first aspect of this application, if the runtime adaptation layer selects the MindIE inference framework, the specific process of subgraph recognition and operator fusion is as follows: if the subgraph pattern is identified as linear projection, reshaping, or transposition, the linear projection operator, reshaping operator, and transposition operator are fused to obtain a projection and recombination fusion operator; if the subgraph pattern is identified as rotation position encoding, attention mechanism, or matrix multiplication, the rotation position encoding operator, attention operator, and matrix multiplication operator are fused to obtain a rotation position encoding and attention fusion operator; if the subgraph pattern is identified as layer normalization, linear layer, GELU function, or linear layer, the layer normalization operator, linear operator, GELU function operator, and linear operator are fused to obtain a feedforward network fusion operator.

[0013] In some embodiments of the first aspect of this application, the preset hybrid scheduling mechanism is as follows: user requests are processed based on a short text priority strategy and a user request queue, and similar user requests are dynamically merged into batches for processing.

[0014] In some embodiments of the first aspect of this application, the method further includes: reading the interconnection topology of the Ascend processor nodes based on the collection communication library, and selecting the communication mechanism with the lowest latency according to the topology.

[0015] In some embodiments of the first aspect of this application, the user request parameters include service type, request concurrency, sequence length, and large model size information.

[0016] To achieve the above and other related objectives, a second aspect of this application provides a large model inference optimization system for Ascend processors, comprising: a containerization detection module, used to compare automatically detected Ascend processor driver version, firmware version, and CANN version with a built-in compatibility table before container startup; if the versions do not match, automatically loading runtime patches or calling backup libraries; if the versions match, processing user requests based on a preset hybrid scheduling mechanism; an adaptive selection module, used to dynamically select the MindIE inference framework or vLLM inference framework based on the obtained user request parameters and a runtime adaptation layer, and load the large model packaged in the container; a weight quantization module, used to convert the large model weights into low-bit weights based on a low-bit quantization mechanism, and compensate the converted low-bit weights based on an error compensation matrix; and an operator fusion module, used to perform subgraph recognition and operator fusion on the model graph according to the computational characteristics of the Ascend processor to generate an instruction sequence adapted to the Ascend processor, and based on the 512B of the Ascend processor. The memory access alignment rules rearrange intermediate tensors; the cross-card communication module is used to perform pipelined overlapping processing of the matrix multiplication calculation of the current token and the full aggregation operation based on the asynchronous execution flow of the Ascend processor for cross-card communication; the inference generation module is used to perform inference and generate inference results according to user requests.

[0017] To achieve the above and other related objectives, a third aspect of this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method.

[0018] To achieve the above and other related objectives, a fourth aspect of this application provides a computer program product comprising computer program code that, when executed on a computer, causes the computer to implement the method.

[0019] To achieve the above and other related objectives, a fifth aspect of this application provides an electronic terminal, including a memory, a processor, and a computer program stored in the memory; the processor executes the computer program to implement the method.

[0020] As described above, the large-model inference optimization method, system, medium, terminal, and program product for Ascend processors in this application have the following beneficial effects:

[0021] After the container starts, it automatically detects the Ascend processor driver version, firmware version, and CANN version, and compares the detected versions with the built-in compatibility table. If the versions do not match, it automatically loads runtime patches or calls backup libraries. If the versions match, it processes user requests based on a preset hybrid scheduling mechanism. Then, based on the acquired user request parameters, it dynamically selects either the MindIE inference framework or the vLLM inference framework based on the runtime adaptation layer and loads the large model packaged in the container. Next, it converts the large model weights to low-bit weights based on a low-bit quantization mechanism and compensates for the converted low-bit weights based on an error compensation matrix. Furthermore, it performs subgraph recognition and operator fusion on the model graph according to the computational characteristics of the Ascend processor to generate an instruction sequence adapted to the Ascend processor, and based on the 512B of the Ascend processor... The memory access alignment rules rearrange the intermediate tensors; at the same time, the asynchronous execution flow based on the Ascend processor pipelines and overlaps the matrix multiplication calculation of the current token with the full aggregation operation; perform inference and generate inference results, and complete the full-stack technology system from underlying operator optimization, inference framework adaptation, cluster communication scheduling to containerized deployment and service scheduling governance, which improves the efficiency and reliability of large model inference. Attached Figure Description

[0022] Figure 1 The diagram shown is a flowchart illustrating a large-model inference optimization method for Ascend processors according to an embodiment of this application.

[0023] Figure 2 The diagram shown illustrates the working principle of a large-model inference optimization method for Ascend processors in one embodiment of this application.

[0024] Figure 3 The diagram shown is a block diagram of a large-model inference optimization system for Ascend processors according to an embodiment of this application.

[0025] Figure 4 The diagram shown is a structural schematic of an electronic terminal according to an embodiment of this application. Detailed Implementation

[0026] The following specific examples illustrate the implementation of this application. Those skilled in the art can easily understand other advantages and effects of this application from the content disclosed in this specification. This application can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this application. It should be noted that, unless otherwise specified, the following embodiments and features in the embodiments can be combined with each other.

[0027] In the embodiments of this application, terms such as "first" and "second" are used to distinguish identical or similar items with essentially the same function and effect. For example, "first XX" and "second XX" are merely used to distinguish different XXs and do not limit their order. Those skilled in the art will understand that terms such as "first" and "second" do not limit the quantity or execution order, and that "first" and "second" do not necessarily imply that they are different.

[0028] It should be noted that, in the embodiments of this application, the words "exemplary" or "for example" indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in this application should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0029] In this application embodiment, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0030] Before providing a further detailed description of the present invention, the nouns and terms used in the embodiments of the present invention are explained, and the nouns and terms used in the embodiments of the present invention are subject to the following interpretations:

[0031] <1> Ascend processor: Huawei's high-performance AI processor is based on its self-developed Da Vinci architecture. It is the flagship product of the Ascend-Max series and is designed for artificial intelligence training scenarios. It adopts the 7nm+ EUV advanced process technology, with a half-precision (FP16) computing power of 256 Tera-FLOPS and an integer precision (INT8) computing power of up to 512 Tera-OPS, while the power consumption is controlled at 310W, with excellent energy efficiency.

[0032] <2> MindIE (Mind Inference Engine) inference framework: Huawei Ascend's inference acceleration suite for AI full-scenario business. By opening up AI capabilities in layers, it supports users' diverse AI business needs, enables hundreds of models and thousands of states, and releases the computing power of Ascend hardware devices.

[0033] <3> vLLM (Virtual Large Language Model) inference framework: It is a high-performance open-source library focused on optimizing Large Language Model (LLM) inference and services. Its core goal is to significantly improve the efficiency and throughput of LLM services in large-scale computing environments.

[0034] <4> Paged Attention is a core technology proposed by the vLLM inference framework. It aims to solve the bottleneck problem of GPU memory management in the Large Language Model (LLM) inference process. The core idea is to divide the KV Cache into fixed-size blocks and achieve non-contiguous storage through dynamic mapping between logical blocks and physical blocks.

[0035] <5> The AllGather operation collects and synchronizes data from all nodes to each node. In model parallelism, when forward computation requires complete parameters, AllGather is used to aggregate parameter fragments distributed across different GPUs onto each GPU.

[0036] <6> AllReduce operation: This operation performs the same reduction operation (such as SUM, MAX) on all nodes and distributes the results to all nodes. A typical scenario is data-parallel gradient synchronization: After each GPU calculates its local gradient, it sums and broadcasts the results through AllReduce, so that all GPUs have the same global gradient.

[0037] <7> CANN (Compute Architecture for Neural Networks): This architecture connects the upper-layer AI training framework with the lower-layer Ascend chip, enabling developers to access the underlying computing power.

[0038] <8> RESTful (Representational State Transfer) is a software architectural style primarily used for designing APIs (Application Programming Interfaces) for web applications. It is based on the HTTP protocol and emphasizes the representation of resources and state transfer, making it suitable for distributed systems and web service development.

[0039] <9> gRPC (Google Remote Procedure Call) is a modern, open-source, high-performance remote procedure call framework that can run in any environment. It can efficiently connect services within and across data centers, and supports load balancing, tracing, health checks, and authentication.

[0040] <10> QPS (Queries Per Second) is a common metric that measures the number of queries received by an information retrieval system (such as a search engine or database) in one second. It is one of the core metrics in high-concurrency system design, performance testing, and operation and maintenance monitoring. It directly reflects the system's processing capacity and throughput. High QPS usually means high user traffic or good system performance optimization.

[0041] <11> Pattern matching is a powerful programming technique that allows developers to identify and process data based on specific patterns.

[0042] <12> Huawei Collective Communication Library (HCCL) is a high-performance collective communication library based on the Ascend AI processor, providing data parallelism and model parallelism collective communication solutions between single machines with multiple GPUs and between multiple machines with multiple GPUs.

[0043] To facilitate understanding of the embodiments of this application, in conjunction with Figure 1 and Figure 2 Detailed explanation. Figure 1 A flowchart illustrating a large-model inference optimization method for Ascend processors is shown in an embodiment of the present invention. Figure 2 This illustration shows a schematic diagram illustrating the working principle of a large model inference optimization method for Ascend processors according to an embodiment of the present invention. The large model inference optimization method for Ascend processors in this embodiment includes the following steps:

[0044] Step S11: After the container starts, the automatically detected Ascend processor driver version, firmware version, and CANN version are compared with the built-in compatibility table; if the versions do not match, the runtime patch is automatically loaded or the backup library is called; if the versions match, the user request is processed based on the preset hybrid scheduling mechanism.

[0045] Specifically, after the container starts, it reads the Ascend processor driver version, firmware version, and CANN version. It then queries the built-in compatibility table of the container image. If a version mismatch is detected, it triggers runtime patches or dynamically configured dependency backup library paths to achieve dynamic alignment between the container environment and the host machine's NPU dependencies. This avoids NPU initialization failures caused by container deployment errors. If a version match is detected, user requests are processed based on a preset hybrid scheduling mechanism. This constitutes containerized build and deployment.

[0046] By designing automatic version detection and dynamic matching capabilities, it supports automatic identification of NPU devices, versions, and quantities, and implements on-demand injection of dependency libraries inside the container, enabling the container to have cross-environment consistency. This can reduce the deployment error rate by more than 90%, significantly improve operational efficiency and system stability, simplify containerized deployment, and enhance compatibility.

[0047] In some embodiments of this application, the preset hybrid scheduling mechanism is as follows: user requests are processed based on a short text priority strategy and a user request queue, and similar user requests are dynamically merged into batches for processing.

[0048] User requests are received through a unified API gateway, supporting RESTful / gRPC dual-protocol adaptation, multi-model routing, multi-tenant isolation, rate control, and token-level timeout control. Based on a short-text-first strategy with estimated latency and user request queues (e.g., regular users and VIP users), it can also improve throughput through batch processing and concurrent scheduling, ensuring low latency and high throughput for the first token in high-concurrency environments. This achieves model service-oriented architecture and scheduling.

[0049] By building a unified API gateway, multi-level priority scheduling, and batch processing integration, the availability and elasticity of inference services are enhanced, unified model service capabilities are achieved, and system stability and scalability are improved.

[0050] Step S12: Based on the obtained user request parameters, dynamically select the MindIE inference framework or the vLLM inference framework based on the runtime adaptation layer, and load the large model packaged in the container.

[0051] During the inference startup phase, a unified runtime adapter layer dynamically selects either the MindIE or vLLM inference framework as the inference execution engine based on user request parameters, and loads the model weights, graph structure, and execution operators of the large model. The runtime adapter layer centrally manages the loading of the large model, weight mapping, key-value cache allocation, and tensor communication, ensuring seamless switching between the two inference frameworks on the Ascend processor. Furthermore, it loads the MindIE library, ACL graph execution library, vLLM-Ascend extension library, and HCCL communication plugin as needed, depending on the actual running framework. Simultaneously, it automatically configures key environment variables in the container to ensure stable cross-machine startup, such as configuring the three key environment variables LD_LIBRARY_PATH, ASCEND_DEVICE_ID, and RUNTIME_CONFIG_PATH.

[0052] Through a unified runtime adaptation layer, the MindIE inference framework (suitable for end-to-end optimization) and the vLLM inference framework (suitable for high-concurrency incremental inference) can be flexibly replaced and work together on the Ascend processor. This enables coverage of various large-model business scenarios such as search, dialogue, text-to-graph, enterprise question answering, and code generation, enhancing versatility and adaptability. It supports both the MindIE and vLLM inference frameworks, with strong compatibility and wide applicability.

[0053] Furthermore, a four-layer image structure is adopted, with model weights supporting chunk-level distribution and incremental updates, thereby achieving lightweight images and layered construction. Specifically, the four-layer image structure consists of a base layer (e.g., Ascend Runtime), a framework layer (MindIE inference framework or vLLM inference framework), a model layer (model weights), and a service layer (e.g., API and business logic). It supports separate updates to the model layer without rebuilding the base and runtime layers, reducing image size by 30% to 50% and achieving lightweight design.

[0054] By adopting layered construction, separation of weights and business logic, and incremental update technology of the model layer, the speed of image construction and distribution is increased by 2-5 times, effectively reducing cluster deployment costs and network transmission overhead, achieving image lightweighting, and improving delivery efficiency.

[0055] In some embodiments of this application, the user request parameters include service type, request concurrency, sequence length, and large model size information.

[0056] In some embodiments of this application, if the runtime adaptation layer selects the vLLM inference framework, it pre-allocates a physically contiguous key-value cache block pool for the generated key-value cache, and compares the current user request sequence length with a set threshold of the key-value cache block pool. If the current sequence length exceeds the set threshold, it determines whether the key-value block is a hot key-value block or a cold key-value block. If it is a hot key-value block, it is migrated to a high-bandwidth key-value cache block pool; if it is a cold key-value block, it is migrated to a low-bandwidth key-value cache block pool.

[0057] If the runtime adaptation layer selects the vLLM inference framework as the inference engine, a dynamic expansion mechanism is designed for it. Specifically, the paging attention mechanism of the vLLM inference framework is deeply integrated into the memory management of the Ascend processor. First, contiguous physical memory blocks are pre-allocated for the key-value cache based on the maximum sequence length for page table initialization. When the actual sequence exceeds the estimate, new physical blocks are automatically requested and background migration is performed, thereby achieving incremental expansion based on the number of tokens. Furthermore, based on the access frequency, hot key-value blocks are migrated to high-bandwidth areas, and cold key-value blocks are placed in secondary high-speed areas, performing non-blocking migration to separate hot and cold key-value blocks. The key-value blocks for the next token are pre-loaded to improve the hit rate and achieve multi-threaded pre-fetching. This dynamic expansion mechanism reduces the key-value access latency of the vLLM inference framework by 20% to 40%. It should be understood that hot key-value blocks refer to blocks that have been frequently accessed recently; cold key-value blocks refer to blocks that have not been accessed for a long time.

[0058] Meanwhile, dynamic batch processing and token parallel execution are performed for multiple user requests, thereby improving throughput by 30% to 80%.

[0059] By designing a key-value cache sequential allocation mechanism and a hot / cold separation mechanism adapted to the Ascend processor, the inference memory usage can be reduced by 40% to 60% without significantly reducing model accuracy, thereby supporting the loading of larger-scale LLMs on a single card or improving stability under multiple concurrent requests.

[0060] Step S13: Convert the large model weights into low-bit weights based on the low-bit quantization mechanism, and compensate the converted low-bit weights based on the error compensation matrix.

[0061] For example, taking into account the characteristics of the Ascend processor, the large model weights are converted to W4 / W8 (weight bits), while the activation values ​​are kept at A16 (activation value bits). That is, the converted low-bit weights are W4A16 / W8A16, and an error compensation matrix is ​​introduced to compensate for the low-bit weights to ensure that the accuracy does not decrease.

[0062] Step S14: Perform subgraph recognition and operator fusion on the model graph according to the computing characteristics of the Ascend processor to generate an instruction sequence adapted to the Ascend processor, and rearrange the intermediate tensors according to the 512B memory access alignment rules of the Ascend processor.

[0063] Specifically, pattern matching and rewriting are performed on core subgraphs such as Q (query) / K (key) / V (value) projection, RoPE (rotation position encoding), and Softmax function to generate instruction sequences for Cube matrix calculation units that are strictly adapted to the Ascend processor. All critical intermediate tensors are forced to follow the 512-byte memory access alignment rule of the Ascend processor, eliminating memory access conflicts and fragmentation at the source.

[0064] In some embodiments of this application, if the runtime adaptation layer selects the MindIE inference framework, the specific process of subgraph recognition and operator fusion is as follows: if the subgraph pattern is identified as linear projection, reshaping, or transpose, the linear projection operator, reshaping operator, and transpose operator are fused to obtain a projection and recombination fusion operator; if the subgraph pattern is identified as rotation position encoding, attention mechanism, or matrix multiplication, the rotation position encoding operator, attention operator, and matrix multiplication operator are fused to obtain a rotation position encoding and attention fusion operator; if the subgraph pattern is identified as layer normalization, linear layer, GELU function, or linear layer, the layer normalization operator, linear operator, GELU function operator, and linear operator are fused to obtain a feedforward network fusion operator.

[0065] If the runtime adaptation layer selects the MindIE inference framework as the inference engine, the pattern matching technology of the MindIE inference framework fuses the operator sequences in the following subgraph patterns. Specifically, for Q / K / V linear projection → Reshape → Transpose, the linear projection operator, reshape operator, and transpose operator are fused to obtain the projection and recombination fusion operator; for Rotation Position Encoding (RoPE) → Attention Softmax → Matrix Multiplication (MatMul), the rotation position encoding operator, attention operator, and matrix multiplication operator are fused to obtain the rotation position encoding and attention fusion operator; for Layer Normalization (LayerNorm) → Linear Layer → GELU Function → Linear Layer, the layer normalization operator, linear operator, GELU function operator, and linear operator are fused to obtain the feedforward network fusion operator. In this way, the resulting projection and recombination fusion operator, rotation position encoding and attention fusion operator, and feedforward network fusion operator are instruction sequences adapted to the Cube matrix computing unit of the Ascend processor, reducing boundary synchronization and memory access. Only one calculation is required, eliminating the need for multiple calculations. At the same time, it reduces memory access between operators by 20% and reduces subgraph execution latency by 15% to 35%.

[0066] Meanwhile, intermediate tensors are laid out with 512-bit alignment to reduce memory access conflicts and maintain the parallel stability of multiple execution streams. Furthermore, contiguous memory areas are pre-allocated before inference to store the KV cache for each layer, reducing fragmentation. After each sequence inference, small fragments are merged to improve the availability of large contiguous areas.

[0067] By designing operator fusion and memory access alignment mechanisms, key operators such as matrix multiplication and layer normalization can fully utilize the hardware advantages of the Ascend processor, significantly improving the large model inference performance of the Ascend processor and further enhancing hardware resource utilization.

[0068] Step S15: The asynchronous execution flow based on the Ascend processor pipelines and overlaps the matrix multiplication calculation of the current token with the full aggregation operation to enable cross-card communication.

[0069] By designing a computation-communication pipelined mechanism, cross-card communication is achieved. Specifically, the Ascend processor's asynchronous execution stream technology is used to pipeline the computation of the attention matrix multiplication of the current token with the full aggregation communication operation or the full reduction operation. The computation of the current token and the communication of the previous token are executed in parallel. The multi-execution stream technology is used to isolate communication and computation instructions, thereby reducing the total communication latency by 20% to 35%.

[0070] In some embodiments of this application, the method further includes: reading the interconnection topology of the Ascend processor nodes based on the collection communication library, and selecting the communication mechanism with the lowest latency according to the topology.

[0071] Specifically, during the startup phase, the HCCL API is automatically executed to read the interconnect topology between Ascend processor nodes. Based on the topology type (e.g., Full-Mesh, single-machine 8-card, dual-machine 16-card, etc.), a communication strategy is automatically selected. For example, a full interconnect uses global Ring operations, while an asymmetric topology uses Hybrid aggregation mode. Different communication weights are applied to different model layers to reduce low-return communication. It should be understood that Ring operations are a detailed implementation of full-mesh operations, referring to communication with only two adjacent nodes at a time; Hybrid aggregation mode is a flexible configuration that supports adding multiple VLAN tags to data packets or stripping data packets to a single tag, enabling more efficient transmission of data packets between different network segments.

[0072] By streamlining communication and computation and selecting topology-aware communication modes, multi-card inference maintains stable low latency in high-concurrency scenarios, significantly improves cross-card HCCL communication efficiency, reduces inference latency, and greatly increases token throughput for multi-card inference, making it particularly suitable for high-load enterprise services.

[0073] Step S16: Perform inference and generate inference results based on user requests.

[0074] Based on the input user request, perform inference operations and generate inference results to complete high-throughput inference, and return the generated token result to the API gateway.

[0075] Furthermore, after the new version model is loaded into the background runtime and warmed up, a zero-disruption upgrade strategy is used to smoothly redirect traffic to the new model, ensuring business continuity. Specifically, weight mapping and static compilation of the MindIE graph are completed first; then, key-value cache pre-allocation is completed. With both the new and old versions of the model coexisting, gateway traffic is smoothly switched until the old version model is unloaded, and the upgrade process does not affect ongoing requests. Simultaneously, the first token latency (FTL), token throughput, HCCL error rate, memory usage, and fragmentation rate are monitored. When anomalies occur, the smallest scheduling unit (Pod) is automatically rebuilt, the runtime is reinitialized, or traffic is migrated to improve availability.

[0076] The large model inference optimization method for Ascend processors provided in this application integrates operator, communication, memory, container, and service optimization to form a closed-loop acceleration of large model inference, which has higher QPS, lower first token latency, better stability and fault self-healing capabilities, providing enterprises with a more efficient and reliable large model inference service infrastructure.

[0077] Figure 3 This is a schematic block diagram of a large-model inference optimization system for Ascend processors provided in an embodiment of this application. Figure 3 As shown, the large model inference optimization system 300 for Ascend processors includes:

[0078] The containerization detection module 301 is used to compare the automatically detected Ascend processor driver version, firmware version, and CANN version with the built-in compatibility table before the container starts. If the versions do not match, the runtime patch is automatically loaded or the backup library is called. If the versions match, the user request is processed based on the preset hybrid scheduling mechanism.

[0079] The adaptive selection module 302 is used to dynamically select the MindIE inference framework or the vLLM inference framework based on the obtained user request parameters and the runtime adaptation layer, and load the large model packaged in the container.

[0080] The weight quantization module 303 is used to convert the weights of a large model into low-bit weights based on a low-bit quantization mechanism, and to compensate the converted low-bit weights based on an error compensation matrix.

[0081] The operator fusion module 304 is used to perform subgraph recognition and operator fusion on the model graph according to the computing characteristics of the Ascend processor, so as to generate an instruction sequence adapted to the Ascend processor, and rearrange the intermediate tensors according to the 512B memory access alignment rules of the Ascend processor.

[0082] The cross-card communication module 305 is used to perform pipelined overlapping processing of the matrix multiplication calculation of the current token and the full aggregation operation based on the asynchronous execution flow of the Ascend processor for cross-card communication.

[0083] The reasoning generation module 306 is used to perform reasoning and generate reasoning results based on user requests.

[0084] The large model inference optimization system for Ascend processors in this application is designed to address the computational architecture characteristics of Ascend processors. Based on the MindIE and vLLM inference frameworks, it performs in-depth optimizations on operator execution, memory management, communication scheduling, container construction, and service management, thereby enabling high throughput, low latency, and scalable deployment of large models on the Ascend platform.

[0085] It should be understood that the specific process of each module performing the above-mentioned steps has been described in detail in the above method embodiments, and will not be repeated here for the sake of brevity.

[0086] It should also be understood that the module division in the embodiments of this application is illustrative and only represents a logical functional division; in actual implementation, there may be other division methods. Furthermore, the functional modules in the various embodiments of this application can be integrated into a single processor, exist as separate physical entities, or be integrated into a single module. The integrated modules described above can be implemented in hardware or as software functional modules.

[0087] Figure 4 This is a schematic block diagram of the electronic terminal provided in an embodiment of this application. Figure 4 As shown, the electronic terminal 400 includes at least one processor 401, a memory 402, at least one network interface 403, and a user interface 405. The various components in the electronic terminal 400 are coupled together via a bus system 404. It is understood that the bus system 404 is used to implement communication between these components. In addition to a data bus, the bus system 404 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in… Figure 4 The general will label all buses as bus systems.

[0088] The user interface 405 may include a monitor, keyboard, mouse, trackball, clicker, button, touchpad, or touch screen.

[0089] It is understood that memory 402 can be volatile memory or non-volatile memory, or both. Non-volatile memory can be read-only memory (ROM) or programmable read-only memory (PROM), used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM) and synchronous static random access memory (SSRAM). The memories described in the embodiments of this invention are intended to include, but are not limited to, these and any other suitable categories of memory.

[0090] In this embodiment of the invention, the memory 402 is used to store various types of data to support the operation of the electronic terminal 400. Examples of this data include: any executable program for operation on the electronic terminal 400, such as the operating system 4021 and application programs 4022; the operating system 4021 contains various system programs, such as the framework layer, core library layer, driver layer, etc., for implementing various basic services and handling hardware-based tasks. The application program 4022 may contain various applications, such as a media player, browser, etc., for implementing various application services. The methods provided in this embodiment of the invention can be included in the application program 4022.

[0091] The methods disclosed in the above embodiments of the present invention can be applied to processor 401, or implemented by processor 401. Processor 401 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the integrated logic circuit of the hardware in processor 401 or by instructions in the form of software. The processor 401 may be a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Processor 401 can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of the present invention. General-purpose processor 401 may be a microprocessor or any conventional processor, etc. The steps of the accessory optimization method provided in the embodiments of the present invention can be directly reflected as being executed by a hardware decoding processor, or being executed by a combination of hardware and software modules in the decoding processor. The software module may be located in a storage medium, which is located in a memory. The processor reads the information in the memory and combines it with its hardware to complete the steps of the aforementioned method.

[0092] In an exemplary embodiment, the electronic terminal 400 may be used by one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), or complex programmable logic devices (CPLDs) to execute the aforementioned method.

[0093] According to the method provided in the embodiments of this application, this application also provides a computer program product, which includes: computer program code, which, when run on a computer, causes the computer to execute... Figures 1 to 2 The method of any of the embodiments shown.

[0094] According to the method provided in the embodiments of this application, this application also provides a computer-readable storage medium storing program code, which, when executed on a computer, causes the computer to perform... Figures 1 to 2 The method of any of the embodiments shown.

[0095] As used in this specification, the terms "component," "module," "system," etc., are used to refer to computer-related entities, hardware, firmware, combinations of hardware and software, software, or software in execution. For example, a component can be, but is not limited to, a process running on a processor, a processor, an object, an executable file, an execution thread, a program, and / or a computer. As illustrated, applications running on computing devices and computing devices can both be components. One or more components may reside in a process and / or an execution thread, and components may be located on a single computer and / or distributed among two or more computers. Furthermore, these components can be executed from various computer-readable media on which various data structures are stored. Components can communicate, for example, via local and / or remote processes based on signals having one or more data packets (e.g., data from two components interacting with another component between a local system, a distributed system, and / or a network, such as the Internet interacting with other systems via signals).

[0096] Those skilled in the art will recognize that the various illustrative logical blocks and steps described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this application.

[0097] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0098] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0099] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0100] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0101] In the above embodiments, the functions of each functional unit can be implemented entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. A computer program product includes one or more computer instructions (programs). When the computer program instructions (programs) are loaded and executed on a computer, all or part of the flow or function according to the embodiments of this application is generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. Computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., high-density digital video discs, DVDs), or semiconductor media (e.g., solid-state disks, SSDs, etc.).

[0102] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0103] In summary, addressing the technical challenges of existing technologies in large-model inference scenarios, such as operator adaptation difficulties, low memory utilization, insufficient cross-card communication efficiency, poor container environment compatibility, and weak service capabilities, this application provides an optimization method, system, medium, terminal, and program product for large-model inference using the Ascend processor. After container startup, it automatically detects the Ascend processor driver version, firmware version, and CANN version, comparing the detected versions with a built-in compatibility table. If the versions do not match, it automatically loads runtime patches or calls backup libraries; if the versions match... The system first processes user requests based on a pre-defined hybrid scheduling mechanism. Then, based on the acquired user request parameters, it dynamically selects either the MindIE inference framework or the vLLM inference framework based on the runtime adaptation layer and loads the large model packaged in a container. Next, it converts the large model weights into low-bit weights based on a low-bit quantization mechanism and compensates for the converted low-bit weights based on an error compensation matrix. Furthermore, it performs subgraph recognition and operator fusion on the model graph according to the computational characteristics of the Ascend processor to generate an instruction sequence adapted to the Ascend processor, and rearranges intermediate tensors according to the Ascend processor's 512B memory access alignment rules. Simultaneously, based on the Ascend processor's asynchronous execution flow, it pipelines and overlaps the matrix multiplication calculation of the current token with the full aggregation operation. Finally, it executes inference and generates inference results, achieving full-stack acceleration from operator-level tuning and low-level communication optimization to containerized delivery and service layer scheduling, meeting the enterprise-level requirements for high-performance inference, scalability, stability, and maintainability. Therefore, this application effectively overcomes the various shortcomings of existing technologies and has high industrial application value.

[0104] The above embodiments are merely illustrative of the principles and effects of this application and are not intended to limit this application. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of this application. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in this application should still be covered by the claims of this application.

Claims

1. A large-model inference optimization method for Ascend processors, characterized in that, include: Once the container starts, the automatically detected Ascend processor driver version, firmware version, and CANN version will be compared with the built-in compatibility table. If the versions do not match, the runtime patch will be loaded automatically or a backup library will be called; if the versions match, the user request will be processed based on a preset hybrid scheduling mechanism. Based on the obtained user request parameters, the MindIE inference framework or vLLM inference framework is dynamically selected based on the runtime adaptation layer, and the large model packaged in the container is loaded. The large model weights are converted into low-bit weights based on the low-bit quantization mechanism, and the converted low-bit weights are compensated based on the error compensation matrix. Based on the computing characteristics of the Ascend processor, subgraph identification and operator fusion are performed on the model graph to generate an instruction sequence adapted to the Ascend processor, and intermediate tensors are rearranged according to the 512B memory access alignment rules of the Ascend processor. The asynchronous execution flow based on the Ascend processor pipelines and overlaps the matrix multiplication calculation of the current token with the full aggregation operation to enable cross-card communication; Perform inference and generate inference results based on user requests.

2. The large-model inference optimization method for Ascend processors according to claim 1, characterized in that, If the runtime adaptation layer selects the vLLM inference framework, it pre-allocates a physically contiguous key-value cache block pool for the generated key-value cache, and compares the current user request sequence length with a set threshold of the key-value cache block pool. If the current sequence length exceeds the set threshold, it determines whether the key-value block is a hot key-value block or a cold key-value block. If it is a hot key-value block, it migrates it to a high-bandwidth key-value cache block pool. If it is a cold key-value block, it will be migrated to a low-bandwidth key-value cache block pool.

3. The large-model inference optimization method for Ascend processors according to claim 1, characterized in that, If the runtime adaptation layer selects the MindIE inference framework, the specific process of subgraph recognition and operator fusion is as follows: If the subgraph pattern is identified as linear projection, reshaping, or transpose, then the linear projection operator, reshaping operator, and transpose operator are fused to obtain the projection and recombination fusion operator. If the subgraph pattern is identified as a rotation position encoding, attention mechanism, or matrix multiplication, then the rotation position encoding operator, attention operator, and matrix multiplication operator are fused to obtain a rotation position encoding and attention fusion operator. If the subgraph pattern is identified as layer normalization, linear layer, GELU function, or linear layer, then the layer normalization operator, linear operator, GELU function operator, and linear operator are fused to obtain the feedforward network fusion operator.

4. The large-model inference optimization method for Ascend processors according to claim 1, characterized in that, The preset hybrid scheduling mechanism is as follows: user requests are processed based on a short text priority strategy and a user request queue, and similar user requests are dynamically merged into batches for processing.

5. The large-model inference optimization method for Ascend processors according to claim 1, characterized in that, Also includes: The system reads the interconnection topology of the Ascend processor nodes based on the collection communication library and selects the communication mechanism with the lowest latency according to the topology.

6. The large-model inference optimization method for Ascend processors according to claim 1, characterized in that, The user request parameters include business type, request concurrency, sequence length, and large model scale information.

7. A large-model inference optimization system for Ascend processors, characterized in that, include: The containerization detection module is used to compare the automatically detected Ascend processor driver version, firmware version, and CANN version with the built-in compatibility table before the container is started. If the versions do not match, the runtime patch will be loaded automatically or a backup library will be called; if the versions match, the user request will be processed based on a preset hybrid scheduling mechanism. The adaptive selection module is used to dynamically select the MindIE inference framework or the vLLM inference framework based on the obtained user request parameters and the runtime adaptation layer, and load the large model packaged in the container. The weight quantization module is used to convert the weights of a large model into low-bit weights based on a low-bit quantization mechanism, and to compensate the converted low-bit weights based on an error compensation matrix. The operator fusion module is used to perform subgraph recognition and operator fusion on the model graph according to the computing characteristics of the Ascend processor, so as to generate an instruction sequence adapted to the Ascend processor and rearrange the intermediate tensors according to the 512B memory access alignment rules of the Ascend processor. The cross-card communication module is used to perform pipelined overlapping processing of the matrix multiplication calculation of the current token and the full aggregation operation based on the asynchronous execution flow of the Ascend processor, so as to enable cross-card communication; The reasoning generation module is used to perform reasoning and generate reasoning results based on user requests.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 6.

9. A computer program product, characterized in that, The computer program product includes computer program code that, when run on a computer, causes the computer to implement the method as described in any one of claims 1 to 6.

10. An electronic terminal, comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the method as described in any one of claims 1 to 6.

Images