Large language model deployment method and device based on mercuric chloride 910B and medium
By conducting performance testing and dynamic monitoring on the Ascend 910B chip and intelligently selecting deployment modes, the problem of uneven resource utilization in the deployment of large language models was solved, achieving efficient matching of computing resources and system stability, and meeting the real-time requirements of the production environment.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- SHANDONG SYNTHESIS ELECTRONICS TECH
- Filing Date
- 2025-12-30
- Publication Date
- 2026-04-21
AI Technical Summary
The existing multi-GPU deployment method for large language models cannot be dynamically adjusted according to real-time load, resulting in a mismatch between resource utilization and business needs. This leads to resource waste at low concurrency and memory overflow or communication bottlenecks at high concurrency. Furthermore, dynamic deployment solutions are complex, costly, and difficult to meet the real-time requirements of production environments.
By conducting benchmark performance tests on the Ascend 910B chip, the set of performance parameters corresponding to the concurrency is determined, load data and hardware status are dynamically monitored, and multi-card single instance or multi-card multi-instance deployment mode is intelligently selected. Combined with the forced switching threshold of hardware status such as video memory and bandwidth, dynamic deployment mode switching is achieved.
It achieves an efficient match between computing resources and business needs, avoids memory overflow and communication bottlenecks, improves the robustness and stability of the system, and meets the high availability and real-time requirements of the production environment.
Smart Images
Figure CN121900965A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of artificial intelligence technology, and in particular to a method, device and medium for deploying a large language model based on Ascend 910B. Background Technology
[0002] As the scale of large language model parameters continues to break through from tens of billions to hundreds of billions (such as Qwen3-72B), the inference process places stringent demands on computing resources, memory bandwidth, and inter-card communication efficiency, making efficient deployment a core industry requirement. Ascend 910B, as a domestically produced flagship AI acceleration chip, adopts a customized computing unit based on the Da Vinci architecture, supports 8-card NVLink interconnection on a single machine (with bandwidth up to 200GB / s), and provides full-stack software support through the Ascend CANN heterogeneous computing architecture, becoming the core hardware foundation for deploying large domestic language models.
[0003] The existing mainstream multi-GPU deployment methods for large models fall into two categories: one is multi-GPU single instance, which uses tensor parallelism, pipeline parallelism and other technologies to divide model parameters to multiple chips and uses the HCCL collection communication library to achieve data synchronization, suitable for low-concurrency and latency-sensitive tasks; the other is multi-GPU multi-instance, where each or each group of cards independently loads the complete model and achieves parallel processing through request distribution, which is suitable for high-concurrency and throughput-priority tasks.
[0004] However, existing deployment methods are static and fixed, and cannot be dynamically adjusted according to real-time load. This leads to a mismatch between resource utilization and business needs. Multiple instances cause resource waste during low concurrency, while single instances are prone to memory overflow or communication bottlenecks during high concurrency. Furthermore, dynamic deployment solutions often rely on complex online learning algorithms, with cumbersome decision-making logic and high deployment costs, making it difficult to meet the real-time requirements of production environments. Summary of the Invention
[0005] To address the aforementioned issues, this application provides a method, device, and medium for deploying a large language model based on the Ascend 910B chip. The method includes: performing benchmark performance tests on a target large language model within a preset concurrency range on a single Ascend 910B chip to obtain a set of performance parameters corresponding to different concurrency levels; the target large language model is a multi-GPU deployed large language model; the performance parameter set includes P99 latency, token throughput, video memory utilization, and computing core utilization; determining low-concurrency and high-concurrency thresholds for large language model deployment based on the performance parameter set; when the target large language model is running in any deployment mode, periodically collecting load data from the request queue through a preset sliding window and monitoring the hardware status data of each Ascend 910B chip; the load data includes at least the average number of concurrent requests within the collection period; any deployment mode includes a multi-GPU single-instance mode and a multi-GPU multi-instance mode; comparing the average number of concurrent requests with the low-concurrency and high-concurrency thresholds respectively to determine the load assessment result; and determining the deployment mode of the target large language model based on the load assessment result and the hardware status data.
[0006] In one example, the method further includes: when the deployment mode needs to be switched, determining the topology and resource allocation of the target large language model on each Ascend 910B chip according to the deployment mode to be switched; determining the difference between the deployment mode to be switched and the current deployment mode according to the topology and resource allocation; and calling the Ascend computing architecture interface to switch the difference to complete the deployment mode switch.
[0007] In one example, based on the performance parameter set, low-concurrency and high-concurrency thresholds for large language model deployment are determined, specifically including: fitting memory utilization-concurrency curves and latency-throughput models according to the performance parameter set; determining the low-concurrency threshold as the concurrency level corresponding to the memory utilization reaching a preset danger threshold based on the memory utilization-concurrency curve; and determining the high-concurrency threshold as the concurrency level corresponding to the communication bandwidth utilization between each Ascend 910B chip reaching saturation in multi-card single-instance mode based on the latency-throughput model and the communication bandwidth data between each Ascend 910B chip.
[0008] In one example, the average number of concurrent requests is compared with a low concurrency threshold and a high concurrency threshold to determine the load assessment result. Specifically, this includes: when the average number of concurrent requests is less than the low concurrency threshold for a first preset duration, the load assessment result is determined to be a low load state; when the average number of concurrent requests is greater than the high concurrency threshold for a second preset duration, the load assessment result is determined to be a high load state; when the average number of concurrent requests is greater than or equal to the low concurrency threshold and less than or equal to the high concurrency threshold, the load assessment result is determined to be an intermediate load state.
[0009] In one example, the deployment mode of the target large language model is determined based on the load assessment result and the hardware status data. Specifically, this includes: when the assessment result is a low load state and the hardware status data does not exceed a preset normal data threshold, the deployment mode is determined to be a multi-card single instance mode; when the assessment result is a high load state and the hardware status data does not exceed a preset normal data threshold, the deployment mode is determined to be a multi-card multi-instance mode; when the assessment result is an intermediate load state and the hardware status data does not exceed a preset normal data threshold, the current deployment mode of the target large language model is maintained; when the hardware status data exceeds a preset normal data threshold, the current deployment mode of the target large language model is directly switched. The preset normal data threshold includes at least one of the following: a preset threshold for the memory utilization rate of a single Ascend 910B chip, a preset threshold for the communication bandwidth utilization rate between each Ascend 910B chip, and a preset threshold for the utilization rate of computing cores.
[0010] In one example, before the target large language model runs in any deployment mode, the method further includes: when the deployment mode is multi-card single instance mode, according to a preset parallel strategy, calling the graph engine toolchain in the Ascend architecture to split and deploy the target large language model to multiple Ascend 910B chips; the number of multiple Ascend 910B chips is a preset number greater than 1; when the deployment mode is multi-card multi-instance mode, according to a preset subgrouping rule, calling the Ascend MindIE toolchain to load target large language model instances on multiple subgroups respectively, and deploying each target large language model instance to the Ascend 910B chip of the corresponding subgroup.
[0011] In one example, after the target large language model is running in any deployment mode, the method further includes: when the deployment mode is multi-GPU single instance mode, distributing inference requests to idle pipeline stages using a pipeline stage-aware scheduling algorithm; and when the deployment mode is multi-GPU multi-instance mode, distributing inference requests to the target large language model instance with the lowest GPU memory usage using a round-robin scheduling algorithm.
[0012] In one example, after calling the Ascend computing architecture interface and switching the difference part to complete the deployment mode switch, the method further includes: collecting the load index of the target large language model and the hardware status index of the Ascend 910B chip according to a preset monitoring period; if the load index and / or hardware status index are not within the preset range, fine-tuning at least one of the low concurrency threshold and high concurrency threshold according to a preset fine-tuning ratio.
[0013] On the other hand, embodiments of this application provide a large language model deployment device based on Ascend 910B, including: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to execute any of the above-mentioned large language model deployment methods based on Ascend 910B.
[0014] On the other hand, embodiments of this application provide a non-volatile computer storage medium for deploying a large language model based on Ascend 910B, which stores computer-executable instructions that can execute any of the above-mentioned methods for deploying a large language model based on Ascend 910B.
[0015] The above-described technical solutions adopted in the embodiments of this application can achieve the following beneficial effects: This application employs a dynamic decision-making mechanism driven by both concurrency and hardware status to intelligently select the optimal deployment mode (multi-GPU single instance or multi-GPU multi-instance) based on real-time load. It ensures low latency under low concurrency and maximizes throughput under high concurrency, overcoming the resource imbalance problem of traditional static deployment modes and achieving efficient matching of computing resources and business needs. It also introduces a forced switching threshold based on hardware status such as GPU memory and bandwidth, which can proactively trigger mode switching when there is a risk of GPU memory overflow or communication saturation, effectively avoiding service interruptions or performance drops and improving system robustness and stability. Furthermore, it adopts a three-step switching method of "preprocessing-hot loading-seamless scheduling," combined with hot loading of model weights and a request caching queue, which can complete deployment mode reconstruction in less than 2 seconds, transparently and without the user's awareness, meeting the stringent requirements of high availability and real-time performance in production environments. Attached Figure Description
[0016] To more clearly illustrate the technical solution of this application, some embodiments of this application will be described in detail below with reference to the accompanying drawings, in which: Figure 1 A flowchart illustrating a method for deploying a large language model based on Ascend 910B, provided as an embodiment of this application; Figure 2 A system architecture diagram of a large language model deployment method based on Ascend 910B provided for embodiments of this application; Figure 3 A flowchart illustrating the deployment mode switching logic of a large language model deployment method based on Ascend 910B, provided for embodiments of this application; Figure 4 A schematic diagram of the single-instance pipeline deployment topology of Qwen3-8B on 4 cards for a large language model deployment method based on Ascend 910B provided in this application embodiment; Figure 5 A schematic diagram of the dual-instance tensor parallel pipeline deployment topology of Qwen3-8B on 4 cards for a large language model deployment method based on Ascend 910B provided in this application embodiment; Figure 6 This is a schematic diagram of the structure of a large language model deployment device based on Ascend 910B, provided as an embodiment of this application. Detailed Implementation
[0017] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0018] Some embodiments of this application will now be described in detail with reference to the accompanying drawings.
[0019] Figure 1 This is a flowchart illustrating a method for deploying a large language model based on Ascend 910B, as provided in this application embodiment. This method can be applied to different business domains. Certain input parameters or intermediate results in this process allow for manual adjustment to help improve accuracy.
[0020] The analysis method involved in the embodiments of this application can be implemented by a terminal device or a server, and this application does not impose any special limitations on it. For ease of understanding and description, the following embodiments are all described in detail using a server as an example.
[0021] Based on this Figure 1 The process may include the following steps: S101: On a single Ascend 910B chip, benchmark performance tests are performed on the target large language model within a preset concurrency range to obtain a set of performance parameters corresponding to different concurrency levels; the target large language model is a large language model deployed on multiple GPUs; the set of performance parameters includes P99 latency, token throughput, video memory utilization, and computing core utilization.
[0022] In some embodiments of this application, benchmark performance tests are performed on a target large language model within a preset concurrency range on a single Ascend 910B chip to obtain a set of performance parameters corresponding to different concurrency levels. This step forms the data foundation for constructing an adaptive decision-making model.
[0023] Specifically, in a clean testing environment, the target large language model (e.g., Qwen3-8B) needs to be loaded onto a single Ascend 910B chip with FP16 precision, and the corresponding software stack, including the MindSpore framework and the AscendCANN computing architecture, needs to be configured.
[0024] Furthermore, during testing, a simulated client continuously sends inference requests to the model in a gradient-increasing manner, such as preset concurrency levels of 1, 10, 20, 30, 40, 60, and 128. The requested content is a text sequence from the standard test dataset. At each concurrency level, the system needs to run stably for a sufficient period of time to collect steady-state performance data.
[0025] The monitoring tool records and calculates several key performance parameters in real time, including P99 latency, which reflects service response speed; token throughput, which measures processing efficiency; video memory utilization, which assesses memory pressure; and computing core utilization, which characterizes chip computing activity.
[0026] Meanwhile, the system will closely monitor for any abnormal phenomena such as memory overflow or communication bandwidth saturation. After all tests are completed, the data collected under various concurrency levels, such as P99 latency, token throughput, video memory utilization, and computing core utilization, will be summarized and organized to form a structured set of performance parameters. This set of parameters will serve as the core basis for subsequent threshold analysis and decision-making.
[0027] S102: Based on the set of performance parameters, determine the low-concurrency threshold and high-concurrency threshold for deploying large language models.
[0028] In some embodiments of this application, after obtaining a detailed set of performance parameters, this step aims to extract quantitative boundaries that guide deployment mode switching through data analysis.
[0029] In practical implementation, the first step is to plot key performance curves based on the parameter set, the most important of which are the memory utilization-concurrency curve and the latency-throughput relationship model. By analyzing the memory utilization-concurrency curve, a trend can be observed where memory utilization increases sharply with the increase of concurrent requests. The number of concurrent requests corresponding to the first time the memory utilization reaches or exceeds a preset danger threshold (e.g., 98%, at which point memory overflow is highly likely to be triggered) is formally defined as the low concurrency threshold. This threshold marks the safe boundary of memory pressure for a single card (or a single card in single-instance mode). When the real-time concurrency exceeds the low concurrency threshold, it means that continuing to use the multi-card single-instance mode will face a high risk of memory overflow.
[0030] On the other hand, analysis was conducted using a latency-throughput model supplemented by monitoring data on inter-card communication bandwidth. The focus was on the phenomenon that, in a multi-card single-instance deployment mode, the utilization rate of inter-card communication bandwidth gradually increases with increasing concurrency. The high-concurrency threshold was defined as the point at which communication bandwidth utilization reaches saturation (e.g., ≥180GB / s, near the theoretical peak bandwidth of the Ascend 910B NVLink interconnect), or the inflection point where latency increases sharply and throughput growth stagnates even though bandwidth is not fully saturated. This threshold identifies the communication or computational bottleneck in the single-instance mode.
[0031] In addition, to avoid frequent and unnecessary switching of deployment modes due to small fluctuations in concurrency around the threshold, which could cause service performance jitter, this method also sets a hysteresis interval between the low concurrency threshold and the high concurrency threshold (i.e., low concurrency threshold ≤ concurrency ≤ high concurrency threshold). When the real-time concurrency falls within this interval, the system will maintain the current deployment mode unchanged.
[0032] S103: When the target large language model is running in any deployment mode, the load data of the request queue is periodically collected through a preset sliding window, and the hardware status data of each Ascend 910B chip is monitored; the load data includes at least the average number of concurrent requests within the collection period; the deployment mode includes multi-card single instance mode and multi-card multi-instance mode.
[0033] In some embodiments of this application, firstly, the target large language model needs to be deployed in any mode. When the deployment mode is multi-card single instance mode, according to the preset parallel strategy, the graph engine toolchain in the Ascend architecture is called to divide the target large language model and deploy it to multiple Ascend 910B chips; wherein, the number of multiple Ascend 910B chips is a preset number greater than 1.
[0034] When the deployment mode is multi-card multi-instance mode, the Ascend MindIE toolchain is called according to the preset subgroup division rules to load the target large language model instance on multiple subgroups respectively, and deploy each target large language model instance to the Ascend 910B chip of the corresponding subgroup.
[0035] During service operation, the system activates a load monitoring module. This module employs a sliding window sampling method to smoothly calculate request load: a fixed time window of 1 second is set, and the window slides forward periodically in 100-millisecond increments. Within each window, the number of newly arriving inference requests is accumulated, and the average concurrent requests for the current period are obtained by calculating "total requests within the window / window duration (1 second)". This method effectively filters out instantaneous spikes or brief troughs in request arrivals, providing stable and reliable load assessment metrics.
[0036] Meanwhile, a hardware monitoring agent collects key hardware metrics for each Ascend 910B chip at a high frequency of 100 milliseconds, including real-time memory usage, utilization of each computing core, and actual bandwidth for data exchange between cards via the HCCL communication library. The collected raw load and hardware data undergoes a simple noise reduction filter to eliminate outliers that may be caused by monitoring jitter or extremely short-lived requests, thereby ensuring that the data used by the subsequent decision engine is highly accurate and representative.
[0037] It should be noted that when the deployment mode is multi-GPU single instance mode, the pipeline stage-aware scheduling algorithm distributes inference requests to idle pipeline stages; when the deployment mode is multi-GPU multi-instance mode, the round-robin scheduling algorithm distributes inference requests to the target large language model instance with the lowest GPU memory usage.
[0038] S104: Compare the average number of concurrent requests with the low concurrency threshold and the high concurrency threshold respectively to determine the load assessment result.
[0039] In some embodiments of this application, this step is the load judgment layer in the decision logic. The decision engine compares the average number of concurrent requests with predetermined low-concurrency and high-concurrency thresholds in real time. The comparison is not based on a single instantaneous sample, but introduces a continuous condition to enhance the stability of the decision: if the average number of concurrent requests is below the low-concurrency threshold for one consecutive second (i.e., for 10 consecutive monitoring periods), the system is determined to be in a low-load state; if the average number of concurrent requests is above the high-concurrency threshold for one consecutive second, it is determined to be in a high-load state; and if the average number of concurrent requests is between the low-concurrency and high-concurrency thresholds (i.e., within the hysteresis interval), it is determined to be in an intermediate-load state. This judgment mechanism, which introduces a duration, avoids frequent mode switching caused by the load briefly hovering around the threshold, ensuring the smooth operation of the system.
[0040] S105: Determine the deployment mode of the target large language model based on the load assessment results and the hardware status data.
[0041] In some embodiments of this application, this is a core step in adaptive decision-making, taking into account both load and hardware dimensions.
[0042] The decision engine first examines the load assessment results: if the load is low, it initially determines that a multi-GPU single-instance mode is more suitable to achieve the lowest possible latency per request; if the load is high, it initially determines that a multi-GPU multi-instance mode is more suitable to maximize system throughput to handle massive concurrent requests; if the load is intermediate, it initially decides to maintain the current mode.
[0043] However, the initial decision still needs to be finalized by the hardware health status. The decision engine synchronously checks the monitored real-time hardware status data and compares it with a set of preset hardware forced switching thresholds. These thresholds are the red lines to ensure the safe operation of the system. For example, any card's memory utilization rate >90%, or inter-card communication bandwidth continuously saturated (≥180GB / s), or computing core utilization abnormally low for a period of time. Once any hardware indicator exceeds its corresponding forced switching threshold, regardless of the current load assessment result, the decision engine will immediately issue a forced mode switch command, and hardware triggering has higher priority than load-based triggering. For example, even under low load, if a card's memory utilization rate abnormally increases and exceeds 90% due to memory leaks or other reasons, the system will force a switch from single-instance mode to multi-instance mode to distribute memory pressure. Only when the load assessment result points to a mode switch and all hardware indicators are within the normal threshold range will a regular load-based mode switch be finally triggered.
[0044] This dual-drive mechanism, driven by load and backed by hardware, ensures that the chosen deployment mode meets both business efficiency requirements and firmly safeguards system stability. Once the target deployment mode is determined, the system enters the mode switching execution phase, seamlessly switching between dynamic model reconstruction and request scheduling within the cluster by calling underlying interfaces such as Ascend CANN.
[0045] It should be noted that when the deployment mode needs to be switched, the topology and resource allocation of the target large language model on each Ascend 910B chip are determined according to the deployment mode to be switched; then, based on the topology and resource allocation, the differences between the deployment mode to be switched and the current deployment mode are determined; finally, the Ascend computing architecture interface is called to switch the differences to complete the deployment mode switch.
[0046] It should also be noted that after the deployment mode is switched, the load index of the target large language model and the hardware status index of the Ascend 910B chip will be collected according to the preset monitoring cycle. If the load index and / or hardware status index are not within the preset range, at least one of the low concurrency threshold and high concurrency threshold will be fine-tuned according to the preset fine-tuning ratio.
[0047] It should be noted that, although the embodiments in this application are based on... Figure 1 Steps S101 to S105 will be described sequentially, but this does not mean that steps S101 and S105 must be performed in a strict order. The reason this embodiment follows this order is... Figure 1The order in which steps S101 to S105 are described is provided to facilitate understanding of the technical solutions of the embodiments of this application by those skilled in the art. In other words, in the embodiments of this application, the order of steps S101 to S105 can be appropriately adjusted according to actual needs.
[0048] pass Figure 1 This application employs a dynamic decision-making mechanism driven by both concurrency and hardware status to intelligently select the optimal deployment mode (multi-GPU single instance or multi-GPU multi-instance) based on real-time load. It ensures low latency under low concurrency and maximizes throughput under high concurrency, overcoming the resource imbalance problem of traditional static deployment modes and achieving efficient matching of computing resources and business needs. Furthermore, it introduces a forced switching threshold based on hardware status such as GPU memory and bandwidth, which can proactively trigger mode switching when there is a risk of GPU memory overflow or communication saturation, effectively avoiding service interruptions or performance drops and improving system robustness and stability. It also adopts a three-step switching method of "preprocessing-hot loading-seamless scheduling," combined with hot loading of model weights and a request cache queue, which can complete deployment mode reconstruction in less than 2 seconds, transparently and without the user's awareness, meeting the stringent requirements of high availability and real-time performance in production environments.
[0049] Figure 2 This is a system architecture diagram of a large language model deployment method based on Ascend 910B provided in an embodiment of this application.
[0050] Figure 3 This application provides a flowchart of the deployment mode switching logic for a large language model deployment method based on Ascend 910B.
[0051] Figure 4 This is a schematic diagram of a single-instance pipeline deployment topology of Qwen3-8B on 4 cards, which is a method for deploying large language models based on Ascend 910B provided in this application embodiment.
[0052] Figure 5 This is a schematic diagram of the dual-instance tensor parallel pipeline deployment topology of Qwen3-8B on 4 cards, which is a method for deploying large language models based on Ascend 910B provided in this application embodiment.
[0053] Figure 6 A schematic diagram of a large language model deployment device based on Ascend 910B provided in this application embodiment includes: At least one processor; and, A memory that is communicatively connected to at least one processor; wherein, The memory stores instructions that can be executed by at least one processor, and the instructions are executed by at least one processor to enable at least one processor to perform any of the above-mentioned large language model deployment methods based on Ascend 910B.
[0054] Some embodiments of this application provide a non-volatile computer storage medium for deploying a large language model based on Ascend 910B, which stores computer-executable instructions that can execute any of the above-mentioned methods for deploying a large language model based on Ascend 910B.
[0055] The various embodiments in this application are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the device and medium embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the description of the method embodiments.
[0056] The devices and media provided in this application are one-to-one with the methods. Therefore, the devices and media also have similar beneficial technical effects as their corresponding methods. Since the beneficial technical effects of the methods have been described in detail above, the beneficial technical effects of the devices and media will not be repeated here.
[0057] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0058] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.
[0059] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.
[0060] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.
[0061] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.
[0062] Memory may include non-persistent storage in computer-readable media, random access memory (RAM), and non-volatile memory such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.
[0063] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.
[0064] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.
[0065] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, or improvements made within the technical principles of this application should fall within the protection scope of this application.
Claims
1. A method for deploying a large language model based on Ascend 910B, characterized in that, The method includes: On a single Ascend 910B chip, benchmark performance tests were conducted on a target large language model within a preset concurrency range to obtain a set of performance parameters corresponding to different concurrency levels. The target large language model is a large language model deployed on multiple GPUs. The set of performance parameters includes P99 latency, token throughput, video memory utilization, and computing core utilization. Based on the set of performance parameters, determine the low-concurrency threshold and high-concurrency threshold for deploying large language models; When the target large language model is running in any deployment mode, the load data of the request queue is periodically collected through a preset sliding window, and the hardware status data of each Ascend 910B chip is monitored; the load data includes at least the average number of concurrent requests within the collection period; the deployment mode includes multi-card single instance mode and multi-card multi-instance mode. The average number of concurrent requests is compared with the low concurrency threshold and the high concurrency threshold respectively to determine the load assessment result; Based on the load assessment results and the hardware status data, the deployment mode of the target large language model is determined.
2. The method according to claim 1, characterized in that, The method further includes: When the deployment mode needs to be switched, the topology and resource allocation of the target large language model on each Ascend 910B chip are determined according to the deployment mode to be switched. Based on the topology and resource allocation, determine the differences between the deployment mode to be switched and the current deployment mode; Call the Ascend computing architecture interface to switch the differences to complete the deployment mode switch.
3. The method according to claim 1, characterized in that, The step of determining the low-concurrency threshold and high-concurrency threshold for deploying large language models based on the performance parameter set specifically includes: Based on the set of performance parameters, fit the memory utilization-concurrency curve and the latency-throughput model respectively; Based on the memory usage-concurrency curve, the concurrency level corresponding to when the memory usage reaches a preset danger threshold is determined as the low concurrency threshold. Based on the latency-throughput model and the communication bandwidth data between each Ascend 910B chip, the concurrency level corresponding to the saturation state of the communication bandwidth utilization between each Ascend 910B chip in the multi-card single-instance mode is determined as the high concurrency threshold.
4. The method according to claim 1, characterized in that, The step of comparing the average number of concurrent requests with the low concurrency threshold and the high concurrency threshold respectively to determine the load assessment result specifically includes: When the average number of concurrent requests is less than the low concurrency threshold for a first preset duration, the load assessment result is determined to be a low load state. When the average number of concurrent requests continues to exceed the high concurrency threshold for a second preset duration, the load assessment result is determined to be a high load state; When the average number of concurrent requests is greater than or equal to the low concurrency threshold and less than or equal to the high concurrency threshold, the load assessment result is determined to be an intermediate load state.
5. The method according to claim 1, characterized in that, The step of determining the deployment mode of the target large language model based on the load assessment results and the hardware status data specifically includes: When the evaluation result indicates a low load state and the hardware status data does not exceed the preset normal data threshold, the deployment mode is determined to be a multi-card single instance mode. When the evaluation result indicates a high load state and the hardware status data does not exceed the preset normal data threshold, the deployment mode is determined to be a multi-card multi-instance mode. When the evaluation result is in the intermediate load state and the hardware status data does not exceed the preset normal data threshold, the current deployment mode of the target large language model is maintained. When the hardware status data exceeds the preset normal data threshold, the deployment mode of the current target large language model is switched directly; the preset normal data threshold includes at least one of the following: the preset threshold for the memory utilization rate of a single Ascend 910B chip, the preset threshold for the communication bandwidth utilization rate between each Ascend 910B chip, and the preset threshold for the utilization rate of the computing core.
6. The method according to claim 1, characterized in that, Before the target large language model is run in any deployment mode, the method further includes: When the deployment mode is multi-card single instance mode, according to the preset parallel strategy, the graph engine toolchain in the Ascend architecture is called to segment and deploy the target large language model to multiple Ascend 910B chips; the number of multiple Ascend 910B chips is a preset number greater than 1. When the deployment mode is multi-card multi-instance mode, the Ascend MindIE toolchain is called according to the preset subgroup division rules to load the target large language model instance on multiple subgroups respectively, and deploy each target large language model instance to the Ascend 910B chip of the corresponding subgroup.
7. The method according to claim 1, characterized in that, The method further includes the following after the target large language model is run in any deployment mode: When the deployment mode is multi-card single instance mode, the inference request is distributed to the idle pipeline stage through the pipeline stage-aware scheduling algorithm; When the deployment mode is multi-GPU multi-instance mode, the inference requests are distributed to the target large language model instance with the lowest GPU memory usage through a round-robin scheduling algorithm.
8. The method according to claim 2, characterized in that, After calling the Ascend computing architecture interface and switching the difference section to complete the deployment mode switch, the method further includes: According to the preset monitoring cycle, the load indicators of the target large language model and the hardware status indicators of the Ascend 910B chip are collected. If the load index and / or hardware status index are not within the preset range, at least one of the low concurrency threshold and high concurrency threshold is fine-tuned according to the preset fine-tuning ratio.
9. A large language model deployment device based on Ascend 910B, characterized in that, include: At least one processor; as well as, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the large language model deployment method based on Ascend 910B as described in any one of claims 1-8.
10. A large language model deployment storage medium based on Ascend 910B, storing computer-executable instructions, characterized in that, The computer-executable instructions are capable of executing the large language model deployment method based on Ascend 910B as described in any one of claims 1-8.