Method and device for parallel accelerated calculation of multiple batch data based on mercuric chloride NPU (Network Processing Unit) equipment

Abstract

The invention relates to the technical field of deep learning inference acceleration, and discloses a mercuric chloride NPU equipment-based multi-batch data parallel acceleration calculation method and device, and the method comprises the steps: converting training models of different deep learning frameworks into mercuric chloride NPU inference models; the mercuration NPU inference model is loaded on the mercuration NPU equipment; dynamically segmenting the computing resources of the mercuration NPU equipment to obtain a plurality of computing power partitions; performing combination processing on the plurality of batch data to generate batch input data corresponding to the computing power partitions, and distributing the batch input data to the corresponding computing power partitions; and performing parallel reasoning calculation on the distributed batch input data in each calculation power partition through the mercuric chloride NPU reasoning model to obtain a reasoning result, and aiming at overcoming the problems of compatibility and parameter management of the existing framework, static resource allocation and low multi-batch parallel efficiency.

Description

Technical Field

[0001] This invention relates to the field of deep learning inference acceleration technology, and discloses a method and apparatus for parallel acceleration of multiple batch data based on Ascend NPU devices. Background Technology

[0002] With the widespread adoption of deep learning technology across various industries, the performance requirements for the inference stage are increasing—not only must massive amounts of data be processed, but low latency and high throughput must also be guaranteed. Ascend NPU, as a high-performance computing hardware optimized for deep learning, has become one of the core hardware platforms for deep learning inference thanks to its dedicated architecture for neural network operations.

[0003] However, existing inference solutions based on Ascend NPU have the following key pain points: I. Deep learning training models are mostly developed based on different frameworks such as PyTorch, Caffe, MindSpore, and PaddlePaddle, and need to be converted to an inference model format supported by Ascend NPU before they can run. Existing conversion solutions are mostly single-transformation modes, lacking a unified parameter storage and management mechanism. The adaptability of the converted model is poor, and subsequent adjustments require re-execution of the entire conversion process, which is inefficient.

[0004] Second, Ascend NPUs primarily employ a static partitioning strategy for resource allocation—that is, the number and size of partitions are predetermined and cannot be dynamically adjusted based on the real-time computing power status of the NPU and the actual computing power requirements of the inference model. This leads to two extreme situations: if the partitions are too large, the NPU's computing power will be idle; if the partitions are too small, the model's inference load will be overloaded, and may even trigger a memory overflow.

[0005] Third, when faced with inference requirements for multiple batches of data, existing solutions lack a targeted data-resource matching mechanism. They either pile all batches of data into the NPU in an unordered manner, causing the load to be concentrated in a few computing units, or they simply distribute the data evenly, ignoring the format differences of each batch of data, which requires additional time for format adaptation. Summary of the Invention

[0006] The purpose of this invention is to overcome the existing framework compatibility and parameter management problems, resource allocation staticization problems, and low efficiency of multi-batch parallel processing, and to provide a method and apparatus for accelerating multi-batch data parallel processing based on Ascend NPU devices.

[0007] To achieve the above-mentioned technical effects, the technical solution adopted by the present invention is as follows: A method for accelerating parallel computation of multiple batches of data based on Ascend NPU devices includes the following steps: Step 1: Convert the training models of different deep learning frameworks into Ascend NPU inference models using the model conversion script; Step 2: Load the Ascend NPU inference model onto the Ascend NPU device; Step 3: Based on the current computing power status of the Ascend NPU device and the computing power requirements of the Ascend NPU inference model, dynamically divide the computing resources of the Ascend NPU device to obtain multiple computing power partitions; Step 4: Combine multiple batch data to generate batch input data corresponding to the computing power partition, and distribute the batch input data to the corresponding computing power partition; Step 5: Using the Ascend NPU inference model, perform parallel inference calculations on the distributed batch input data within each computing power partition to obtain the inference results.

[0008] As a preferred embodiment, the deep learning framework includes at least one of PyTorch, Caffe, MindSpore, and PaddlePaddle.

[0009] As a preferred embodiment, a configuration file is generated during the conversion process in step one, and the configuration file is used to store the parameters of the model conversion.

[0010] As a preferred implementation, the step of dynamically allocating computing resources includes: monitoring the remaining computing power of the Ascend NPU device in real time, and dynamically adjusting the number and size of the computing power partitions based on the monitoring results and the computing power requirements of the inference model.

[0011] As a preferred implementation, it also includes: real-time monitoring of the computing status of each computing power partition and the overall performance of the Ascend NPU device; Based on the monitoring results, dynamically adjust the distribution strategy for subsequent batch input data or the partitioning strategy for computing power partitions.

[0012] This invention also proposes a multi-batch data parallel acceleration computing device based on the Ascend NPU device, comprising: The model conversion module is used to convert training models from different deep learning frameworks into Ascend NPU inference models using model conversion scripts. A model loading module is used to load the Ascend NPU inference model on the Ascend NPU device; The resource dynamic allocation module is used to dynamically allocate the computing resources of the Ascend NPU device according to the current computing power status of the Ascend NPU device and the computing power requirements of the Ascend NPU inference model, to obtain multiple computing power partitions; The data combination and distribution module is used to combine multiple batch data to generate batch input data corresponding to the computing power partition, and distribute the batch input data to the corresponding computing power partition. The parallel inference computing module is used to perform parallel inference computing on the distributed batch input data in each computing power partition through the Ascend NPU inference model to obtain inference results.

[0013] As a preferred embodiment, the deep learning framework includes at least one of PyTorch, Caffe, MindSpore, and PaddlePaddle.

[0014] In a preferred embodiment, the model conversion module is also used to generate a configuration file during the conversion process, the configuration file being used to store the parameters of the model conversion.

[0015] As a preferred implementation, the resource dynamic allocation module includes: The monitoring unit is used to monitor the remaining computing power of the Ascend NPU device in real time. The adjustment unit is used to dynamically adjust the number and size of the computing power partitions based on the monitoring results and the computing power requirements of the inference model.

[0016] As a preferred implementation scheme, it also includes: The monitoring module is used to monitor the computing status of each computing power partition and the overall performance of the Ascend NPU device in real time. The strategy adjustment module is used to dynamically adjust the distribution strategy of subsequent batch input data or the partitioning strategy of computing power based on monitoring results.

[0017] Compared with the prior art, the beneficial effects of this invention are: I. Supports deep learning frameworks such as PyTorch, Caffe, MindSpore, and PaddlePaddle. Through unified model conversion scripts and configuration file management, it achieves efficient adaptation of multi-framework training models to Ascend NPU inference models, eliminating the need to develop dedicated conversion tools for different frameworks and broadening application scenarios.

[0018] Second, abandoning the traditional static resource allocation strategy, we dynamically adjust the number and size of computing power partitions by monitoring the NPU computing power status and model requirements in real time, ensuring that the NPU computing power resources are allocated on demand—avoiding idle computing power and preventing overload, thus improving resource utilization.

[0019] Third, synchronous inference of multiple batches of data is achieved through a logic of data combination, targeted distribution, and multi-partition parallel processing. Compared to the traditional single-partition serial processing scheme, the inference throughput is improved under the same hardware conditions.

[0020] Fourth, through real-time monitoring and dynamic strategy adjustment, load fluctuations during NPU inference can be addressed, ensuring stable operation of model inference under different working conditions and avoiding inference interruptions or accuracy degradation caused by sudden load changes. Attached Figure Description

[0021] Figure 1 This is a logical block diagram of the parallel accelerated computing method for multiple batch data based on the Ascend NPU device of the present invention; Figure 2 This is a block diagram of the multi-batch data parallel acceleration computing device based on the Ascend NPU device of the present invention; Figure 3 This is a diagram showing the internal structure of the resource dynamic partitioning module of the present invention. Attached image description: 1. Model conversion module; 2. Model loading module; 3. Dynamic resource allocation module; 31. Monitoring unit; 32. Adjustment unit; 4. Data combination and distribution module; 5. Parallel inference computing module; 6. Monitoring module; 7. Strategy adjustment module. Detailed Implementation

[0023] The present invention will now be described in further detail with reference to the embodiments and accompanying drawings. However, this should not be construed as limiting the scope of the above-described subject matter of the present invention to the following embodiments; all technologies implemented based on the content of the present invention fall within the scope of the present invention.

[0024] like Figure 1 As shown in Example 1, a method for parallel acceleration of multiple batch data based on Ascend NPU devices includes the following steps: Step 1: Model Conversion—Adapting Multi-Frame Training Models to Ascend NPU Pre-developed model conversion scripts, such as scripts based on Ascend ATC tools, are used to convert training models from different deep learning frameworks (e.g., PyTorch's .pth format, Caffe's .caffemodel format, MindSpore's .mindir format, and PaddlePaddle's .pdparams format) into inference model formats supported by Ascend NPU (e.g., .om format). During the conversion process, a configuration file (e.g., .cfg format) is generated synchronously. This file stores key parameters for model conversion, including but not limited to: input data dimensions (e.g., [batch_size, height, width, channel] in NHWC format), inference accuracy (e.g., FP16, FP32, INT8), operator optimization strategies (e.g., fusing convolution and BatchNorm operators), and output node definitions. The existence of the configuration file avoids repeated conversions during subsequent model adjustments; only parameter modifications are needed to regenerate the adapted model.

[0025] Step 2: Model Loading—Deploying the Inference Model to the Ascend NPU On the Ascend NPU device, the Ascend NPU inference model generated in step one is loaded through the Ascend device driver interface. During the loading process, the NPU automatically loads the model's weight data and operator logic into dedicated video memory and completes the initialization of the inference engine, preparing the hardware for subsequent parallel computing.

[0026] Step 3: Dynamic Resource Allocation—On-Demand Allocation of NPU Computing Resources Based on the current computing power status of the Ascend NPU device and the computing power requirements of the Ascend NPU inference model, the NPU's computing resources are dynamically partitioned to obtain multiple independent computing power partitions. The specific implementation logic is as follows: Real-time computing power monitoring: The current computing power status of the NPU can be obtained in real time through the Ascend device monitoring tool. Key indicators include: percentage of remaining computing power (such as the current proportion of idle computing units), used video memory / remaining video memory size, and computing unit load rate. Computing power requirement matching: Based on the configuration file generated in step one, analyze the computing power requirement of the inference model - that is, the minimum computing power and minimum video memory required for a single inference of the model; Dynamic partitioning adjustment: Based on the monitoring results and computing power requirements, the number and size of computing power partitions are dynamically determined. For example, if the NPU currently has 80% remaining computing power, and the model requires 20% computing power for a single inference, then the system is divided into 4 computing power partitions, with each partition allocated 20% computing power and corresponding GPU memory. If the NPU's remaining computing power is subsequently monitored to drop to 60%, the number of partitions is automatically adjusted to 3 to ensure that the resources of each partition meet the model's inference requirements.

[0027] Step 4: Batch Data Combination and Distribution—Precise Matching of Data and Resources The multiple input batches of data are combined and processed to generate batch input data that corresponds one-to-one with the computing power partitions obtained in step three, and then distributed to the corresponding computing power partitions. The specific operations are as follows: Data combination processing: To address potential format differences among multiple batches of data, such as some batches being RGB images and others being grayscale images, or inconsistent resolutions, the format is first standardized, for example, converting grayscale images to 3-channel RGB format and standardizing the resolution to the 224×224 required by the model input. Then, the data is grouped according to the number of computing power partitions—the total amount of data in each group matches the computing power capacity of the corresponding partition. For example, if the maximum batch size supported by the partition computing power is 64, then two 32-batch data sets are combined into one 64-batch input data set. Targeted distribution: Through the data scheduling module, each batch of input data is distributed to the corresponding computing power partition, ensuring that the amount of input data in each partition does not exceed its resource capacity limit and avoiding load imbalance.

[0028] Step 5: Parallel Inference Computation—Simultaneous Execution of Inference Across Multiple Partitions Within each computing power partition, the Ascend NPU inference model, loaded in step two, performs parallel inference computation on the distributed batch input data. Since each computing power partition is independent and the resource allocation meets the model requirements, all partitions can perform inference operations synchronously (such as calculating the class probability of each image for an image classification model, and calculating the semantic vector of text for an NLP model). Finally, each partition outputs an independent inference result.

[0029] To address dynamic load changes during NPU inference, this method also includes the following optimization steps: Real-time monitoring: Through the Ascend device monitoring interface, real-time status data is collected in two dimensions: ① the computing status of each computing power partition; ② the overall performance of the NPU device; Strategy Adjustment: Based on monitoring results, dynamically adjust the distribution strategy for subsequent batch input data or the partitioning strategy for computing power partitions. For example, if the inference time of partition 1 is found to be twice that of other partitions, reduce the number of batches in partition 1 in the next distribution round (e.g., from 64-batch to 32-batch); if the overall NPU temperature is found to be too high, appropriately reduce the number of computing power partitions (e.g., from 4 to 2) to reduce the overall load.

[0030] like Figure 2 , 3 As shown in Embodiment 2, a multi-batch data parallel acceleration computing device based on an Ascend NPU device includes: Model conversion module 1 is used to convert training models from different deep learning frameworks into Ascend NPU inference models using model conversion scripts. Model loading module 2 is used to load the Ascend NPU inference model on the Ascend NPU device; The resource dynamic allocation module 3 is used to dynamically allocate the computing resources of the Ascend NPU device according to the current computing power status of the Ascend NPU device and the computing power requirements of the Ascend NPU inference model, to obtain multiple computing power partitions; the resource dynamic allocation module 3 includes: a monitoring unit 31, used to monitor the remaining computing power of the Ascend NPU device in real time; and an adjustment unit 32, used to dynamically adjust the number and size of the computing power partitions according to the monitoring results and the computing power requirements of the inference model. The data combination and distribution module 4 is used to combine multiple batch data to generate batch input data corresponding to the computing power partition, and distribute the batch input data to the corresponding computing power partition. Parallel inference computing module 5 is used to perform parallel inference computing on the distributed batch input data in each computing power partition through the Ascend NPU inference model to obtain inference results.

[0031] It also includes: a monitoring module 6, used to monitor the computing status of each computing power partition and the overall performance of the Ascend NPU device in real time; The strategy adjustment module 7 is used to dynamically adjust the distribution strategy of subsequent batch input data or the partitioning strategy of computing power partitions based on monitoring results.

[0032] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for parallel acceleration computing of multiple batches of data based on Ascend NPU devices, characterized in that, Includes the following steps: Step 1: Convert the training models of different deep learning frameworks into Ascend NPU inference models using the model conversion script; Step 2: Load the Ascend NPU inference model onto the Ascend NPU device; Step 3: Based on the current computing power status of the Ascend NPU device and the computing power requirements of the Ascend NPU inference model, dynamically divide the computing resources of the Ascend NPU device to obtain multiple computing power partitions; Step 4: Combine multiple batch data to generate batch input data corresponding to the computing power partition, and distribute the batch input data to the corresponding computing power partition; Step 5: Using the Ascend NPU inference model, perform parallel inference calculations on the distributed batch input data within each computing power partition to obtain the inference results.

2. The method for parallel acceleration of multiple batch data based on Ascend NPU device according to claim 1, characterized in that, The deep learning framework includes at least one of PyTorch, Caffe, MindSpore, and PaddlePaddle.

3. The method for parallel acceleration of multiple batch data based on Ascend NPU device according to claim 2, characterized in that, A configuration file is generated during the conversion process in step one. This configuration file is used to store the parameters for model conversion.

4. The method for parallel acceleration of multiple batch data based on Ascend NPU device according to claim 3, characterized in that, The step of dynamically allocating computing resources includes: monitoring the remaining computing power of the Ascend NPU device in real time, and dynamically adjusting the number and size of the computing power partitions based on the monitoring results and the computing power requirements of the inference model.

5. The method for parallel acceleration of multiple batch data based on Ascend NPU device according to claim 4, characterized in that, Also includes: Real-time monitoring of the computing status of each computing power partition and the overall performance of the Ascend NPU device; Based on the monitoring results, dynamically adjust the distribution strategy for subsequent batch input data or the partitioning strategy for computing power partitions.

6. A multi-batch data parallel acceleration computing device based on Ascend NPU, characterized in that, include: The model conversion module (1) is used to convert training models of different deep learning frameworks into Ascend NPU inference models through model conversion scripts; The model loading module (2) is used to load the Ascend NPU inference model on the Ascend NPU device; The resource dynamic partitioning module (3) is used to dynamically partition the computing resources of the Ascend NPU device according to the current computing power status of the Ascend NPU device and the computing power requirements of the Ascend NPU inference model, and obtain multiple computing power partitions. The data combination and distribution module (4) is used to combine multiple batch data to generate batch input data corresponding to the computing power partition, and distribute the batch input data to the corresponding computing power partition. The parallel inference computing module (5) is used to perform parallel inference computing on the distributed batch input data in each computing power partition through the Ascend NPU inference model to obtain the inference result.

7. The multi-batch data parallel acceleration computing device based on the Ascend NPU device according to claim 6, characterized in that, The deep learning framework includes at least one of PyTorch, Caffe, MindSpore, and PaddlePaddle.

8. The multi-batch data parallel acceleration computing device based on the Ascend NPU device according to claim 7, characterized in that, The model conversion module (1) is also used to generate a configuration file during the conversion process, which is used to store the parameters of the model conversion.

9. The multi-batch data parallel acceleration computing device based on the Ascend NPU device according to claim 8, characterized in that, The resource dynamic partitioning module (3) includes: The monitoring unit (31) is used to monitor the remaining computing power of the Ascend NPU device in real time; The adjustment unit (32) is used to dynamically adjust the number and size of the computing power partitions according to the monitoring results and the computing power requirements of the inference model.

10. The multi-batch data parallel acceleration computing device based on the Ascend NPU device according to claim 9, characterized in that, Also includes: The monitoring module (6) is used to monitor the computing status of each computing power partition and the overall performance of the Ascend NPU device in real time. The strategy adjustment module (7) is used to dynamically adjust the distribution strategy of subsequent batch input data or the partitioning strategy of computing power based on the monitoring results.

Images