AI computing power enabled subsystem

Abstract

The invention discloses an AI computing power enabling subsystem which comprises heterogeneous computing architecture software used for supporting development of self-defined operators of developers; the AI framework software is used for being provided for the step 2 to construct the AI large model, and the AI computing power enabling subsystem is set to be in a mode of combining the heterogeneous computing architecture software and the AI framework software, so that the capability of each functional unit of the heterogeneous computing architecture, the capability of each functional unit of the AI framework, the capability of each functional unit of the heterogeneous computing architecture and the capability of each functional unit of the AI framework can be simply and effectively realized. According to the method, the AI computing power enabling engine is comprehensively formed according to the comprehensive performance optimization scheme of the heterogeneous computing architecture and the AI framework, the data processing capacity of an AI chip can be optimized, the throughput rate of an AI computing cluster subsystem is increased, and developers are helped to improve the development and training efficiency. According to the AI computing power enabling subsystem, through the arrangement of heterogeneous computing architecture software and AI framework software, a developer can be effectively enabled to focus on adjustment, optimization and improvement of an artificial intelligence algorithm.

Description

Technical Field

[0001] The present invention relates to an enabling subsystem, and more specifically to an AI computing power enabling subsystem. Background Art

[0002] The AI ​​computing power enabling subsystem of the AI ​​computing power center is oriented to the AI ​​computing cluster subsystem to enable hardware devices; it is oriented to developers and is used for the development and training of AI algorithm models. The AI ​​computing power enabling subsystem has the following features: 1. It supports CPU, NPU and other architectures. 2. It supports backward compatibility and evolution. 3. The platform is scalable and adaptable to different computing power and memory changes. 4. It supports the AI ​​development framework, allowing developers to focus more on the tuning and improvement of artificial intelligence algorithms.

[0003] Currently, the existing technology does not provide the above-mentioned AI computing power enabling subsystem, so the existing AI computing power center cannot have the following features: 1. Support CPU, NPU and other architectures. 2. Support backward compatibility and evolution. 3. The platform is scalable and adaptable to different computing power and memory changes. 4. Support AI development framework, allowing developers to focus more on tuning and improving artificial intelligence algorithms. Summary of the invention

[0004] In view of the shortcomings of the prior art, the purpose of the present invention is to provide an AI computing power enabling subsystem that can be well applied to AI computing power centers and allows developers to focus more on the tuning and improvement of artificial intelligence algorithms.

[0005] To achieve the above-mentioned purpose, the present invention provides the following technical solution: an AI computing power enabling subsystem, characterized in that it includes:

[0006] Heterogeneous computing architecture software to support developers in developing custom operators;

[0007] AI framework software is used to provide step 2 for building a large AI model.

[0008] As a further improvement of the present invention, the heterogeneous computing architecture software includes:

[0009] The unified programming interface Ascend computing language library is used to provide unified and standardized interface components, including device management, context management, stream management, memory management, model loading and execution, operator loading and execution, media data processing, and graph management API libraries, providing users with programming interfaces that enable the capabilities of each component for users to develop deep neural network applications;

[0010] The graph optimization and compilation module is used to provide a unified IR interface for different front-ends, support the parsing / optimization / compilation of the computational graphs expressed by TensorFlow / Caffe / MindSpore, and provide the ability to optimize the deployment of the back-end computing engine to give full play to the performance of the device;

[0011] The computing execution engine is responsible for the scheduling and execution of various types of operators in the neural network, memory and other resources, communication management, etc., and outputs the computing results;

[0012] Operator compilation module and operator library, used to provide Ascend operator development and compilation capabilities, enabling chip capabilities; computing library, used to provide computing libraries such as neural networks, basic linear algebra, collective communication, and media data preprocessing, to support users to use and build AI models;

[0013] The computing tuning engine includes two plug-ins: the AutoTune plug-in, which supports both single operator and whole-graph tuning modes. It can automatically generate a knowledge base based on the model and data features, and apply it to the actual deployment process to obtain high performance. The model compression plug-in provides model quantization and sparsity capabilities in the form of a framework plug-in, supports Calibration and ReTrain, and realizes model miniaturization.

[0014] As a further improvement of the present invention, the graph optimization and compilation module includes:

[0015] GE (GraphEngine) module, which serves as the control center for graph compilation and operation, and provides graph operation environment management, graph execution engine management, operator library management, subgraph optimization management, graph operation management, and graph execution control;

[0016] FE (FusionEngine) module, which is used to manage the fusion rules of operators, match IRGraph and perform operator fusion operations according to the fusion rules, manage the operators supported by the system and maintain the operator list, and support custom operators;

[0017] AICPUE (AICPUEngine) module, which is responsible for AICPU operator information management;

[0018] HCCL (Huawei Collective Communication Library) module: This HCCL module is responsible for the operator information management of HCCL, and is used to provide HCCL to implement the gradient aggregation function of all workers involved in parallel computing, providing a data parallel solution for Ascend multi-machine and multi-card training.

[0019] As a further improvement of the present invention, the calculation execution engine includes:

[0020] Runtime module, which is used to provide resource management channels for neural network tasks, including storage management, device management, execution flow management, event management, kernel function execution and other functions;

[0021] The TS module is responsible for the management, scheduling, and execution of the computational graph Task sequence.

[0022] As a further improvement of the present invention, the operator compilation module includes:

[0023] The TBE module is used to automatically compile and generate operators by describing the algorithm implementation and optimization scheduling of operators through DSL. At the same time, this module is also used as a development tool for custom operators for ecosystem developers to use;

[0024] Among them, the operator library is the Ascend neural network acceleration library, which has rich built-in operators to support neural network training and reasoning acceleration. Through the setting of the above modules and operator libraries, it is possible to simply and effectively provide Ascend operator development and compilation capabilities and enable chip capabilities.

[0025] As a further improvement of the present invention, the calculation library includes:

[0026] Neural network library, used as the Ascend neural network acceleration library, with rich built-in operators to support neural network training and reasoning acceleration;

[0027] Basic Linear Algebra Set, a numerical library for standardizing and publishing basic linear algebra operations;

[0028] Collective communication library, used to store HCCL operator information and to manage HCCL operator information;

[0029] The media data preprocessing module is used to implement video decoding, video encoding, JPEG decoding, JPEG encoding, PNG decoding, and VPC. The codec library performs image preprocessing through hardware acceleration, reduces CPU usage, and improves the parallel processing capability of image preprocessing.

[0030] As a further improvement of the present invention, the computing optimization engine includes:

[0031] The AutoTune module supports two tuning modes: single operator and whole graph. It can automatically generate a knowledge base based on the model and data features, and apply it to the actual deployment process to obtain high performance.

[0032] The model compression module provides model quantization and sparsity capabilities in the form of a framework plug-in, supports Calibration and ReTrain, and realizes model miniaturization.

[0033] The beneficial effect of the present invention is that, through the setting of heterogeneous computing architecture software and AI framework software, it is possible to effectively realize the capabilities of each functional unit of the heterogeneous computing architecture, the capabilities of each functional unit of the AI ​​framework, and the comprehensive performance optimization plan for the heterogeneous computing architecture and AI framework, and comprehensively form an AI computing power enabling engine, which can optimize the data processing capabilities of AI chips, improve the throughput of AI computing cluster subsystems, and help developers improve development and training efficiency. DETAILED DESCRIPTION

[0034] The present invention will be further described in detail with reference to the given embodiments below.

[0035] An AI computing power enabling subsystem of this embodiment includes:

[0036] Heterogeneous computing architecture software to support developers in developing custom operators;

[0037] The AI ​​framework software is used to provide the AI ​​big model for step 2. By setting the AI ​​computing power enabling subsystem to a combination of heterogeneous computing architecture software and AI framework software, it is possible to simply and effectively realize the capabilities of each functional unit of the heterogeneous computing architecture, the capabilities of each functional unit of the AI ​​framework, and the comprehensive performance optimization plan for the heterogeneous computing architecture and AI framework. The comprehensive composition of the AI ​​computing power enabling engine can optimize the data processing capabilities of the AI ​​chip, improve the throughput of the AI ​​computing cluster subsystem, and help developers improve development and training efficiency.

[0038] As a specific implementation of the improvement, the heterogeneous computing architecture software includes:

[0039] The unified programming interface Ascend computing language library is used to provide unified and standardized interface components, including device management, context management, stream management, memory management, model loading and execution, operator loading and execution, media data processing, and graph management API libraries, providing users with programming interfaces that enable the capabilities of each component for users to develop deep neural network applications;

[0040] The graph optimization and compilation module is used to provide a unified IR interface for different front-ends, support the parsing / optimization / compilation of the computational graphs expressed by TensorFlow / Caffe / MindSpore, and provide the ability to optimize the deployment of the back-end computing engine to give full play to the performance of the device;

[0041] The computing execution engine is responsible for the scheduling and execution of various types of operators in the neural network, memory and other resources, communication management, etc., and outputs the computing results;

[0042] Operator compilation module and operator library, used to provide Ascend operator development and compilation capabilities, enabling chip capabilities; computing library, used to provide computing libraries such as neural networks, basic linear algebra, collective communication, and media data preprocessing, to support users to use and build AI models;

[0043] The computing tuning engine includes two plug-ins: the AutoTune plug-in, which supports single operator and whole graph tuning modes. It can automatically generate a knowledge base based on the model and data features, and apply it to the actual deployment process to obtain high performance; the model compression plug-in provides model quantization and sparsity capabilities in the form of a framework plug-in, supports Calibration and ReTrain, and realizes model miniaturization. In this way, through the settings of the above modules, it is possible to simply and effectively provide a multi-level programming interface, with the advantages of full scenarios, low thresholds, and high performance, to support users to quickly build AI applications and businesses based on the Ascend platform.

[0044] As a specific implementation of the improvement, the graph optimization and compilation module includes:

[0045] GE (GraphEngine) module, which serves as the control center for graph compilation and operation, and provides graph operation environment management, graph execution engine management, operator library management, subgraph optimization management, graph operation management, and graph execution control;

[0046] FE (FusionEngine) module, which is used to manage the fusion rules of operators, match IRGraph and perform operator fusion operations according to the fusion rules, manage the operators supported by the system and maintain the operator list, and support custom operators;

[0047] AICPUE (AICPUEngine) module, which is responsible for AICPU operator information management;

[0048] HCCL (Huawei Collective Communication Library) module: This HCCL module is responsible for the operator information management of HCCL, and is used to provide HCCL to implement the gradient aggregation function of all workers involved in parallel computing, and provide a data parallel solution for Ascend multi-machine and multi-card training. Through the settings of the above modules, graph optimization and compilation operations can be effectively implemented.

[0049] As a specific implementation of the improvement, the calculation execution engine includes:

[0050] The Runtime module is used to provide resource management channels for neural network tasks, including storage (Memory) management, device (Device) management, execution flow (Stream) management, and event (Event)

[0051] Management, kernel function execution and other functions;

[0052] The TS module is responsible for the management, scheduling, and execution of the Task sequence of the computational graph. By setting up the above modules, the computation can be simply and effectively implemented, and the computation results can be output.

[0053] As a specific implementation of the improvement, the operator compilation module includes:

[0054] The TBE module is used to automatically compile and generate operators by describing the algorithm implementation and optimization scheduling of operators through DSL (domain-specific language). At the same time, this module is also used as a development tool for custom operators for ecosystem developers to use;

[0055] Among them, the operator library is the Ascend neural network acceleration library, which has rich built-in operators to support neural network training and reasoning acceleration. Through the setting of the above modules and operator libraries, it is possible to simply and effectively provide Ascend operator development and compilation capabilities and enable chip capabilities.

[0056] As a specific implementation of the improvement, the calculation library includes:

[0057] Neural network library, used as the Ascend neural network acceleration library, with rich built-in operators to support neural network training and reasoning acceleration;

[0058] Basic linear algebra set, a numerical library for standardizing the release of basic linear algebra operations. BLAS (Basic Linear Algebra Program Set) is an application program interface (API) standard that is used to standardize the release of basic linear algebra operations (such as vector or matrix multiplication). In the field of high-performance computing, BLAS is widely used. In AI chip enablement, some BLAS library implementations are also provided, and are constantly iterating and supplementing. Provide a standard BLAS library that users can call directly, shielding the underlying detail differences, and thus obtaining the best computing experience and effect on the AI ​​platform;

[0059] Collective Communication Library, which is used to store HCCL (collective communication library) operator information and is responsible for HCCL operator information management. HCCL implements the gradient aggregation (allreduce) function of all workers participating in parallel computing and provides a data parallel solution for Ascend multi-machine and multi-card training.

[0060] The media data preprocessing module is used to implement video decoding, video encoding, JPEG decoding, JPEG encoding, PNG decoding, and VPC. The codec library performs image preprocessing through hardware acceleration, reduces CPU usage, and improves the parallel processing capability of image preprocessing.

[0061] As a specific implementation of the improvement, the computing optimization engine includes:

[0062] The AutoTune module supports two tuning modes: single operator and whole graph. It can automatically generate a knowledge base based on the model and data features, and apply it to the actual deployment process to obtain high performance.

[0063] The model compression module provides model quantization and sparsity capabilities in the form of a framework plug-in, supports Calibration, ReTrain, etc., and realizes the miniaturization of the model. In this way, through the setting of the AutoTune module, Autotune can complete the deep performance optimization of the operator for specific model specifications. Users only need to make a small investment to improve the performance of the model; in the model conversion or model training stage, the user is provided with the function of automatic tuning, which can automatically complete the automatic performance optimization of the operator in the model and improve the end-to-end running speed of the model; Autotune can automatically complete the performance optimization of the model and output the knowledge base file at the same time. Users can deploy the knowledge base file to a new environment, and the same model can obtain the previously tuned performance without turning on Autotune again. The AI ​​chip enables the provision of model compression tools, which currently include model quantization and tensor decomposition functions to improve the model inference performance. Model quantization provides two quantization modes: post-training quantization and quantization-aware training. The tensor decomposition function supports decomposing some convolutional layers in the original model into several layers with smaller computational and parameter amounts. Post-training quantization refers to the INT8 quantization of the weights and data of the trained model with the help of a small amount of calibration data, thereby accelerating the model inference speed. The post-training quantization function in the model compression tool can complete the model quantization with only a small amount of data in resource-constrained scenarios, which can provide better business capabilities, better adapt to customers' hardware resources, and better serve their business applications.

[0064] The above is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should also be regarded as the protection scope of the present invention.

Claims

1. An AI computing power enabling subsystem, characterized in that: include: Heterogeneous computing architecture software to support developers in developing custom operators; AI framework software is used to provide step 2 for building a large AI model.

2. The AI ​​computing power enabling subsystem according to claim 1, characterized in that: The heterogeneous computing architecture software includes: The unified programming interface Ascend computing language library is used to provide unified and standardized interface components, including device management, context management, stream management, memory management, model loading and execution, operator loading and execution, media data processing, and graph management API libraries, providing users with programming interfaces that enable the capabilities of each component for users to develop deep neural network applications; The graph optimization and compilation module is used to provide a unified IR interface for different front-ends, support the parsing / optimization / compilation of the computational graphs expressed by TensorFlow / Caffe / MindSpore, and provide the ability to optimize the deployment of the back-end computing engine to give full play to the performance of the device; The computing execution engine is responsible for the scheduling and execution of various types of operators in the neural network, memory and other resources, communication management, etc., and outputs the computing results; Operator compilation module and operator library, used to provide Ascend operator development and compilation capabilities, enabling chip capabilities; computing library, used to provide computing libraries such as neural networks, basic linear algebra, collective communication, and media data preprocessing, to support users to use and build AI models; The computing tuning engine includes two plug-ins: the AutoTune plug-in, which supports both single operator and whole-graph tuning modes. It can automatically generate a knowledge base based on the model and data features, and apply it to the actual deployment process to obtain high performance. The model compression plug-in provides model quantization and sparsity capabilities in the form of a framework plug-in, supports Calibration and ReTrain, and realizes model miniaturization.

3. The AI ​​computing power enabling subsystem according to claim 2, characterized in that: The graph optimization and compilation module includes: GE (GraphEngine) module, which serves as the control center for graph compilation and operation, and provides graph operation environment management, graph execution engine management, operator library management, subgraph optimization management, graph operation management, and graph execution control; FE (FusionEngine) module, which is used to manage the fusion rules of operators, match IRGraph and perform operator fusion operations according to the fusion rules, manage the operators supported by the system and maintain the operator list, and support custom operators; AICPUE (AICPUEngine) module, which is responsible for AICPU operator information management; HCCL (Huawei Collective Communication Library) module: This HCCL module is responsible for the operator information management of HCCL, and is used to provide HCCL to implement the gradient aggregation function of all workers involved in parallel computing, providing a data parallel solution for Ascend multi-machine and multi-card training.

4. The AI ​​computing power enabling subsystem according to claim 3, characterized in that: The computing execution engine comprises: Runtime module, which is used to provide resource management channels for neural network tasks, including storage management, device management, execution flow management, event management, kernel function execution and other functions; The TS module is responsible for the management, scheduling, and execution of the computational graph Task sequence.

5. The AI ​​computing power enabling subsystem according to claim 4, characterized in that: The operator compilation module includes: The TBE module is used to automatically compile and generate operators by describing the algorithm implementation and optimization scheduling of operators through DSL. At the same time, this module is also used as a development tool for custom operators for ecosystem developers to use; Among them, the operator library is the Ascend neural network acceleration library, which has rich built-in operators to support neural network training and reasoning acceleration. Through the setting of the above modules and operator libraries, it is possible to simply and effectively provide Ascend operator development and compilation capabilities and enable chip capabilities.

6. The AI ​​computing power enabling subsystem according to claim 5, characterized in that: The computing library includes: a neural network library, which is used as an Ascend neural network acceleration library, with rich built-in operators to support neural network training and reasoning acceleration; Basic Linear Algebra Set, a numerical library for standardizing and publishing basic linear algebra operations; Collective communication library, used to store HCCL operator information and to manage HCCL operator information; The media data preprocessing module is used to implement video decoding, video encoding, JPEG decoding, JPEG encoding, PNG decoding, and VPC. The codec library performs image preprocessing through hardware acceleration, reduces CPU usage, and improves the parallel processing capability of image preprocessing.

7. The AI ​​computing power enabling subsystem according to claim 6, characterized in that: The computing optimization engine includes: The AutoTune module supports two tuning modes: single operator and whole graph. It can automatically generate a knowledge base based on the model and data features, and apply it to the actual deployment process to obtain high performance. The model compression module provides model quantization and sparsity capabilities in the form of a framework plug-in, supports Calibration and ReTrain, and realizes model miniaturization.