Hardware accelerator VTA model deployment scheme support extension method

Abstract

The invention relates to a hardware accelerator VTA model deployment scheme support extension method and device, electronic equipment and a storage medium. The method comprises the steps that a pre-training model is imported into a deep learning compiler, a hardware accelerator is used as a target operation model to compile and generate a low-level intermediate representation, the hardware accelerator compiles and generates an operation environment, and the hardware accelerator receives the introduced low-level intermediate representation and generates an instruction stream; processing the instruction flow queue, and driving a model reasoning process on hardware; and the hardware accelerator calculates the input and weight data loaded to the on-chip cache to obtain a reasoning result of the model, and post-processes the reasoning result of the model and generates a visual result for output. According to the method, the model reasoning efficiency on the target hardware platform is improved, the logic completeness of the design during the running of the hardware accelerator VTA is improved, and the generalization degree of the hardware accelerator VTA is improved.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the field of deep learning compiler and hardware accelerator, and in particular, to a model deployment scheme support extension method and device of hardware accelerator VTA, electronic equipment and computer readable storage medium. BACKGROUND

[0002] Regardless of the model training process, the trained model is finally deployed to the actual scene. The development and rapid landing of deep learning models have become the research focus in this field. Deep learning compilers enable developers to deploy models of various deep learning training frameworks on various hardware, convert models described by deep learning frameworks into effective code implementations on hardware platforms, and reduce the burden of manual optimization of models. Neural network models generally have high computation and storage complexity, and the appropriate computing platform needs to be selected during model deployment. As a chip or board designed specifically for accelerating deep learning computation, deep learning accelerators are common model deployment targets. Since the current deep learning compiler and deep learning model deployment scheme are still in the development stage, the ability to support potential model deployment optimization schemes is particularly important for deep learning accelerators.

[0003] The current deep learning compilers mainly include TensorFlow XLA, TensorComprehension, Glow, MLIR, TVM, etc. Some mainstream deep learning compilers have been widely used in actual model deployment processes.

[0004] Although the current popular open source deep learning compiler framework has its own design advantages and focuses, the design architecture of its general multi-level intermediate representation and the front-end and back-end optimization are similar. In addition to developing new deep learning compiler frameworks, the main research direction in this field is to use widely recognized open source deep learning compiler frameworks to complete model deployment and develop and optimize on this basis to improve model inference performance. Among them, the open source deep learning compiler TVM has advantages in universality and flexibility, has an advanced automatic tuning strategy, and becomes the preferred choice of many researchers. However, the deep learning compiler TVM currently still has problems such as insufficient scheduling optimization design, and the model deployment inference cannot achieve ideal performance, and there is optimization space in the compilation process.

[0005] On the other hand, in addition to CPU and GPU, FPGA is gradually becoming a candidate platform for high-energy efficient neural network processing. FPGA can cooperate with the hardware design of a specific model according to the computation process of the neural network to achieve high parallelism and simplify the logic.

[0006] VTA is a fully open-source, general-purpose, customizable deep learning accelerator with a complete TVM-based compiler stack. Together, TVM and VTA form an end-to-end deep learning system stack, including hardware design, driver, JIT runtime, and TVM-based optimized compiler stack. VTA integrates the most prominent and common features of mainstream deep learning accelerators, which is a modular hardware accelerator that simplifies the workflow of deploying deep learning models to FPGAs. Although VTA basically reflects the design experience of deep learning accelerators, in order to simplify the hardware design, VTA is coupled with the compilation strategy of TVM in many ways, and the support for emerging model deployment optimization solutions is insufficient, and the inference cannot reach the limit performance optimization. Therefore, VTA also needs to be updated and designed to support the latest research results of deep learning model deployment solutions.

[0007] Therefore, one or more methods are needed to solve the above problems.

[0008] It should be noted that the information disclosed in the above background section is only used to strengthen the understanding of the background of the present disclosure, and therefore can include information that does not constitute prior art known to those of ordinary skill in the art. SUMMARY

[0009] The purpose of the present disclosure is to provide a model deployment solution support extension method, device, electronic equipment and computer readable storage medium of hardware accelerator VTA, so as to at least overcome one or more problems caused by the limitations and defects of the related art to some extent.

[0010] According to one aspect of the present disclosure, a model deployment solution support extension method of hardware accelerator VTA is provided, comprising:

[0011] Importing a pre-trained model into a deep learning compiler, running model compilation with a hardware accelerator as a target, generating a low-level intermediate representation, and a hardware accelerator compilation generating a running environment, the hardware accelerator receiving incoming low-level intermediate representation and generating an instruction stream;

[0012] Processing the instruction stream queue to drive the model inference process on the hardware;

[0013] The hardware accelerator calculates the input and weight data loaded into the on-chip cache to obtain the inference result of the model, and post-processes the model inference result and generates a visual result output.

[0014] In an exemplary embodiment of the present disclosure, the method further comprises:

[0015] Preparing a deep learning model to be deployed to a hardware platform, importing a pre-trained model to a deep learning compiler, running model compilation targeting a hardware accelerator, converting the deep learning model into a high-level graph structure for graph-level optimization, and reducing to a low-level intermediate representation for operator-level optimization to generate a low-level intermediate representation for model inference on the hardware accelerator.

[0016] In an example embodiment of the present disclosure, the method further comprises:

[0017] Defining configurable parameters of the hardware accelerator, and compiling to generate a runtime environment of the deep learning compiler and the hardware accelerator.

[0018] In an example embodiment of the present disclosure, the method further comprises:

[0019] The hardware accelerator receives the low-level intermediate representation, and determines a compilation optimization scheme based on the low-level intermediate representation, and generates an instruction stream based on the low-level intermediate representation of the single-variable data loading scheme and the multi-variable data loading scheme, respectively, using the instruction generation scheme of the single-variable data loading scheme and the multi-variable data loading scheme.

[0020] In an example embodiment of the present disclosure, the method further comprises:

[0021] Processing the instruction stream queue, partially decoding the instructions by the instruction fetch module, and pushing the instructions into the command queue, which enters the loading, computing, and storage modules.

[0022] In an example embodiment of the present disclosure, the method further comprises:

[0023] The hardware accelerator driver drives the model inference process on the hardware through the processed instruction stream queue, and the hardware module completes the loading, computing, and storage processes of the data.

[0024] In an example embodiment of the present disclosure, the method further comprises:

[0025] The hardware accelerator uses the GEMM kernel of the hardware accelerator to perform dense linear algebra computation, uses the tensor ALU of the hardware accelerator to perform general computation, and performs computation on the input and weight data loaded into the on-chip cache to obtain the inference result of the model and perform post-processing.

[0026] After the post-processing of the model inference computation result, a visualized result is generated and output to the user.

[0027] In one aspect of the present disclosure, a hardware accelerator VTA model deployment scheme supporting extension device is provided, comprising:

[0028] An instruction stream generation module is configured to import a pre-trained model into a deep learning compiler, to compile the model for a hardware accelerator, to generate a low-level intermediate representation, and to generate a runtime environment for the hardware accelerator, wherein the hardware accelerator receives the incoming low-level intermediate representation and generates an instruction stream.

[0029] A model inference process module is configured to process the instruction stream queue and to drive a model inference process on the hardware.

[0030] An inference result generation module is configured to perform computation on input and weight data loaded into an on-chip cache by the hardware accelerator, to obtain an inference result of the model, and to perform post-processing on the inference result of the model and to generate a visualized result output.

[0031] In one aspect of the present disclosure, an electronic device is provided, comprising:

[0032] a processor; and

[0033] a memory having computer readable instructions stored thereon, wherein the computer readable instructions, when executed by the processor, implement the method according to any one of the above.

[0034] In one aspect of the present disclosure, a computer readable storage medium is provided, having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the method according to any one of the above.

[0035] In one of the exemplary embodiments of the present disclosure, a model deployment scheme of a hardware accelerator VTA supports an extension method, which includes: importing a pre-trained model into a deep learning compiler, compiling the model for a hardware accelerator, generating a low-level intermediate representation, and generating a runtime environment for the hardware accelerator, wherein the hardware accelerator receives the incoming low-level intermediate representation and generates an instruction stream; processing the instruction stream queue and driving a model inference process on the hardware; performing computation on input and weight data loaded into an on-chip cache by the hardware accelerator, obtaining an inference result of the model, and performing post-processing on the inference result of the model and generating a visualized result output. The present disclosure helps to improve the model inference efficiency on a target hardware platform, to improve the logical completeness of the runtime design of the hardware accelerator VTA, and to improve the generalization degree of the hardware accelerator VTA.

[0036] It should be understood that the foregoing general description and the following detailed description are only exemplary and explanatory, and are not limiting of the present disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0037] The above and other features and advantages of the present disclosure will become more apparent by describing in detail exemplary embodiments thereof with reference to the attached drawings.

[0038] Figure 1A flowchart of a model deployment scheme of a hardware accelerator VTA supporting an extension method is shown according to an example embodiment of the present disclosure;

[0039] Figure 2 A system structure diagram of a hardware accelerator runtime supporting an extension method of a model deployment scheme of a hardware accelerator VTA supporting an extension method is shown according to an example embodiment of the present disclosure;

[0040] Figure 3 An execution relationship design diagram of a hardware accelerator runtime supporting an extension method of a model deployment scheme of a hardware accelerator VTA supporting an extension method is shown according to an example embodiment of the present disclosure;

[0041] Figure 4 A schematic block diagram of a model deployment scheme of a hardware accelerator VTA supporting an extension device is shown according to an example embodiment of the present disclosure;

[0042] Figure 5 A block diagram of an electronic device is schematically shown according to an example embodiment of the present disclosure;

[0043] Figure 6 A schematic diagram of a computer readable storage medium is schematically shown according to an example embodiment of the present disclosure. DETAILED DESCRIPTION

[0044] Example embodiments now will be described more fully hereinafter with reference to the accompanying drawings. Example embodiments, however, can be implemented in many different forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of example embodiments to those skilled in the art. Like reference numerals refer to like elements throughout the several views.

[0045] Moreover, the described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided to give a thorough understanding of embodiments of the present disclosure. One skilled in the relevant art will recognize, however, that the techniques described herein can be practiced without one or more of the specific details, or with other methods, components, materials, and so forth. In other instances, well-known structures, materials, or operations are not shown or described in detail in order to avoid obscuring aspects of the present disclosure.

[0046] The block diagrams shown in the drawings are merely functional entities and do not necessarily have to correspond to physically independent entities. That is, the functional entities can be implemented in the form of software, or in one or more software-hardened modules, or in different network and / or processor devices and / or microcontroller devices.

[0047] In the present example embodiment, a model deployment scheme support extension method of a hardware accelerator VTA is first provided; as shown in Figure 1 The model deployment scheme support extension method of the hardware accelerator VTA can include the following steps:

[0048] In step S110, a pre-trained model is imported into a deep learning compiler to compile the model for a hardware accelerator, generate a low-level intermediate representation, and generate a running environment for hardware accelerator compilation. The hardware accelerator receives the incoming low-level intermediate representation and generates an instruction stream.

[0049] In step S120, the instruction stream queue is processed to drive the model inference process on the hardware.

[0050] In step S130, the hardware accelerator performs calculations on the input and weight data loaded into the on-chip cache to obtain the inference result of the model, and performs post-processing on the model inference result and generates a visual result output.

[0051] The model deployment scheme support extension method of the hardware accelerator VTA in the example embodiment of the present disclosure includes: importing a pre-trained model into a deep learning compiler to compile the model for a hardware accelerator, generating a low-level intermediate representation, and generating a running environment for hardware accelerator compilation. The hardware accelerator receives the incoming low-level intermediate representation and generates an instruction stream. The instruction stream queue is processed to drive the model inference process on the hardware. The hardware accelerator performs calculations on the input and weight data loaded into the on-chip cache to obtain the inference result of the model, and performs post-processing on the model inference result and generates a visual result output. The present disclosure helps to improve the model inference efficiency on the target hardware platform, improve the logical completeness of the hardware accelerator VTA runtime design, and improve the generalization degree of the hardware accelerator VTA.

[0052] In the following, the model deployment scheme support extension method of the hardware accelerator VTA in the present example embodiment will be further described.

[0053] Embodiment One:

[0054] In step S110, the pre-trained model can be imported into a deep learning compiler, the model is compiled for a hardware accelerator, a low-level intermediate representation is generated, a hardware accelerator is compiled to generate a running environment, and the hardware accelerator receives the incoming low-level intermediate representation and generates an instruction stream.

[0055] In an embodiment of the present example, the method further comprises:

[0056] A deep learning model to be deployed to a hardware platform is prepared, a pre-trained model is imported into a deep learning compiler, the model is compiled for a hardware accelerator, the deep learning model is converted into a high-level graph structure for graph-level optimization, and is then reduced to a low-level intermediate representation for operator-level optimization, and a low-level intermediate representation is generated for model inference on the hardware accelerator.

[0057] In an embodiment of the present example, the method further comprises:

[0058] The configurable parameters of the hardware accelerator are defined, and a runtime environment of the deep learning compiler and the hardware accelerator is compiled and generated.

[0059] In an embodiment of the present example, the method further comprises:

[0060] The hardware accelerator receives the low-level intermediate representation, and determines the compilation optimization scheme based on the low-level intermediate representation, and generates an instruction stream based on the low-level intermediate representation of the single-variable data loading scheme and the multi-variable data loading scheme, respectively.

[0061] In step S120, the instruction stream queue can be processed to drive the model inference process on the hardware.

[0062] In an embodiment of the present example, the method further comprises:

[0063] The instruction stream queue is processed, the instructions are partially decoded by a fetch module, and the instructions are pushed into a command queue, and the command queue enters a loading, computing, and storage module.

[0064] In an embodiment of the present example, the method further comprises:

[0065] The hardware accelerator drives the model inference process on the hardware through the processed instruction stream queue, and the hardware module completes the loading, computing, and storage processes of the data.

[0066] In step S130, the hardware accelerator can calculate the input and weight data loaded into the on-chip cache to obtain the inference result of the model, and post-process the model inference result to generate a visualized result output.

[0067] In an embodiment of the present example, the method further comprises:

[0068] The hardware accelerator performs dense linear algebra computation using the GEMM kernel of the hardware accelerator, performs general computation using the tensor ALU of the hardware accelerator, performs computation on input and weight data loaded into on-chip cache, obtains inference result of the model and performs post-processing;

[0069] After the model inference computation result is post-processed, a visualized result is generated and output to the user.

[0070] In an embodiment of the present example, an extended method is provided for the model deployment scheme type supported by the hardware accelerator VTA when deploying the deep learning model to the hardware accelerator VTA based on the deep learning compiler. The extended method supports the model deployment scheme of the hardware accelerator VTA, provides correct instruction stream generation and model inference support for the multi-variable cache address offset that the hardware accelerator VTA originally does not support, removes the restriction on generating low-level intermediate representation for the default compilation scheme when performing model inference on the hardware accelerator VTA, allows the deep learning compiler TVM to adopt an optimized and more flexible storage resource allocation scheme, helps to improve the model inference efficiency on the target hardware platform, improves the logical completeness of the hardware accelerator VTA runtime design, and improves the generalization degree of the hardware accelerator VTA.

[0071] Embodiment two:

[0072] In an embodiment of the present example, Figure 2 For the structural diagram of the model deployment scheme support extension method and system of the hardware accelerator VTA in an embodiment, the support extension method and system provided in the present embodiment can be divided into a deep learning compiler and a hardware accelerator, including a deep learning model compilation device, a hardware accelerator structure configuration device, a hardware accelerator instruction generation device, a hardware accelerator instruction processing device, a hardware accelerator driving device, and a model inference result generation device.

[0073] In the embodiment of the present example, the deep learning model compilation device, i.e., the deep learning compiler TVM, provides deep learning model compilation for model deployment targeting hardware accelerators, generates low-level intermediate representations required for model inference on hardware accelerators; the hardware accelerator structure configuration device, i.e., the hardware accelerator VTA, designs the architecture of the hardware accelerator, which can flexibly configure the size of the computing array, the width of the data type, and the size of the on-chip cache. The user needs to determine the parameter configuration of the hardware accelerator according to the actual resources of the target hardware platform; the hardware accelerator instruction generation device generates instructions for data loading, computing, and storage functions and microcode instructions for recording address index offsets based on the incoming low-level intermediate representations. The instruction fetching module of the hardware accelerator instruction processing device partially decodes the instructions and pushes them into the command queue. The command queue enters the loading, computing, and storage modules, which describe the three-stage task flow of loading, computing, and storing, and complete the specific implementation of the inference computing process. The hardware accelerator driver device, as a driver interface device used at runtime, includes storage management and instruction driving, and is responsible for driving the instruction stream generated by the hardware accelerator to perform specific model inference operations. The model inference result generation device completes the post-processing of the final model inference result and visualizes the output of the model inference result.

[0074] In the embodiment of the present example, the model deployment scheme provided by the present embodiment supports the following steps:

[0075] Step one: prepare the deep learning model to be deployed to the hardware platform, import the pre-trained model into the deep learning compiler TVM, and run the model compilation with the hardware accelerator VTA as the target. Convert the deep learning model into a high-level graph structure for graph-level optimization, and then reduce it to a low-level intermediate representation for operator-level optimization to generate a low-level intermediate representation for model inference on the hardware accelerator;

[0076] Step two: define the configurable parameters of the hardware accelerator VTA and compile the runtime environment of the deep learning compiler TVM and the hardware accelerator VTA;

[0077] Step three: the hardware accelerator receives the incoming low-level intermediate representation and judges the compilation optimization scheme of the application based on it. The low-level intermediate representations of the original single-variable data loading scheme and the optimized multi-variable data loading scheme of the application are generated by the original and extended instruction generation schemes, respectively, to generate instruction streams;

[0078] Step four: process the generated instruction stream queue, the instruction fetching module partially decodes the instructions and pushes them into the command queue. These command queues enter the loading, computing, and storage modules;

[0079] Step five: the hardware accelerator drives the model inference process on the hardware through the processed instruction stream queue, and the hardware module completes the loading, calculation and storage of data;

[0080] Step six: the hardware accelerator VTA uses its GEMM kernel to perform dense linear algebra calculation and its tensor ALU to perform general calculation, and calculates the input and weight data loaded into the on-chip cache to obtain the inference result of the model and perform post-processing;

[0081] Step seven: after the post-processing of the model inference calculation result is completed, the visualized result is generated and output to the user. For the same model with the same input, the original compilation scheme and the optimized compilation scheme are applied respectively, and the results of the model inference should be exactly the same, but the inference efficiency of the model on the hardware should be improved.

[0082] In the embodiment of the present example, the embodiments provided by the present application have the following technical effects:

[0083] As the target platform of model deployment, the hardware accelerator VTA receives the low-level intermediate representation obtained by the deep learning compiler compiling and converting the model, generates an instruction stream and drives the model inference, and then visualizes and outputs the model inference result. At present, the processing of the low-level intermediate representation and the instruction stream generation scheme design of the hardware accelerator are coupled with the default compilation scheme of the deep learning compiler TVM, and the compilation optimization of the multi-variable data loading scheme is not considered. Therefore, for the same deep learning model, if the default compilation scheme is applied for compilation and deployment to the hardware accelerator VTA, the correct inference result can be obtained, but if a more flexible multi-variable data loading scheme is applied for model compilation, the original runtime instruction generation scheme of the hardware accelerator VTA cannot implement the optimization scheme, the generated instruction stream cannot calculate the correct result, and the design goal of improving the model inference efficiency of the optimization scheme cannot be achieved.

[0084] In order to further improve the inference efficiency of the model on the hardware, the runtime design of the hardware accelerator VTA is supplemented and expanded, so that the hardware accelerator can support the optimized compilation scheme, and the embodiment of the application provides a hardware accelerator VTA model deployment scheme support expansion method, process and system, which is used to enhance the logical completeness of the hardware accelerator runtime design architecture and provide hardware design support for potential model deployment optimization scheme. The running environment of the deep learning compiler needs to be built, and the customizable parameters of the hardware accelerator need to be configured and compiled to generate, and after importing the model from the front end, the hardware accelerator VTA is used as the target to run the model compilation, the deep learning model is converted into a high-level graph structure for graph-level optimization, and then reduced to a low-level intermediate representation for operator-level optimization, and the low-level intermediate representation is generated for model inference on the hardware accelerator. Specifically, the computer predeploys the environment related to the deep learning compiler and the driver and runtime environment of the hardware accelerator, and when the model compilation and deployment start, the automatic deployment and test script is executed to complete the compilation and conversion of the deep learning model; the low-level intermediate representation is automatically transmitted to the hardware accelerator to generate execution instructions and the model inference on the hardware, the model inference result generation and visual output are completed, and the user can observe the model deployment progress and operation result, and the corresponding result will be output on the terminal after the model inference is completed.

[0085] The hardware accelerator instruction generation device generates instructions based on the low-level intermediate representation generated by the deep learning model compilation device, determines the compilation scheme based on the input parameters, generates instructions including instructions for completing data loading, calculation and storage functions, and microcode instructions for recording address index offset. Among them, the generation of microcode instructions by the hardware accelerator plays an important role in the correctness of the model inference result. If the microcode instructions cannot be correctly generated and loaded, the input and weight data cannot be loaded to the scheme preset address index during the model inference, and the model inference result will be wrong. Specifically, the hardware accelerator instruction generation device in the embodiment of the application provides extension support for the original instruction generation process of the hardware accelerator, such as Figure 3The low-level intermediate representation passed to the hardware accelerator by the deep learning compiler adopts a hardware accelerator original microcode instruction queue generation scheme when a default single-variable compilation scheme design is applied, and adopts an extended microcode instruction queue generation scheme when an optimized multi-variable compilation scheme design is applied. After the microcode instructions are generated respectively, the hardware accelerator continues to generate the instruction stream according to the low-level intermediate representation passed in. The low-level intermediate representation generated by the deep learning compiler is also different for the single-variable and multi-variable compilation schemes, and the specific calculation process adopted during model inference is different, but the calculation results are completely the same. The model inference using the optimized compilation scheme can achieve higher inference efficiency and reduce the model inference time. Through the supplementary processing of the address index offset parameter in the low-level intermediate representation and the extension support of the microcode instruction generation process, the range of model deployment optimization schemes that can be supported by the hardware accelerator VTA is expanded, and the limitation that the original hardware accelerator cannot provide flexible support for the optimized deep learning model compilation scheme is improved. The design framework of the hardware accelerator is supplemented.

[0086] In the model deployment scheme support extension method provided by the embodiment of the present application, the deep learning compiler selects the target platform as the hardware accelerator VTA when running the model compilation conversion process according to the target platform to which the model is to be deployed. In the model compilation optimization process, the deep learning model imported by the front end needs to be converted into a multi-level intermediate representation. Many of the conversions and optimizations are independent of the target platform, and the instruction stream generation, instruction stream processing, model inference hardware driving, and result generation and output are completed on the hardware accelerator. The structural design of the hardware accelerator VTA summarizes the characteristics of existing deep learning accelerators, and from the perspective of hardware and software collaborative design, it contains a simulator and an FPGA deployment module. Because the simplified hardware design needs more software support, the hardware accelerator VTA is collaboratively designed in the deep learning compiler TVM compilation stack, so when the user optimizes, extends or improves the design of the compilation process of the deep learning compiler, the hardware accelerator also needs to be improved and extended with the new design scheme. The model deployment support extension method aims to supplement the design architecture of the hardware accelerator, so that the user can automatically provide hardware support for new optimization schemes when applying or developing optimized model deployment schemes, and improve the model inference efficiency.

[0087] In the model deployment scheme support extension method provided by the embodiment of the present application, the model deployment scheme can be a deep learning model compilation stack composed of the deep learning compiler TVM and the hardware accelerator VTA, or other alternative schemes for deploying deep learning models. The support extension method is universal for different model deployment schemes and accelerators.

[0088] In the embodiment of the present example, the deep learning model compilation process is visualized together with the model inference result on the target platform, so that the user can pay attention to the model deployment inference progress in real time. For different models, their inference results can be different. The types of output model inference results include but are not limited to images and bounding boxes, model inference results and corresponding probabilities, data access amount of model inference, etc.

[0089] In the embodiment of the present example, the model deployment scheme provided by the present application supports an extension method, which can be applied to but is not limited to the hardware accelerator VTA and can support but is not limited to the compilation optimization scheme based on the deep learning compiler TVM. For other deep learning accelerators that lack similar support and other model deployment schemes that apply optimized data loading designs, the extension support scheme described in the present application is also effective.

[0090] It should be noted that although the steps of the method in the present disclosure are described in a specific order in the accompanying drawings, this does not require or imply that the steps must be performed in this specific order, or that all the steps shown must be performed to achieve the desired result. In addition or alternatively, some steps can be omitted, multiple steps can be combined into one step, and / or one step can be divided into multiple steps, etc.

[0091] In addition, in the embodiment of the present example, a hardware accelerator VTA model deployment scheme support extension device is also provided. Referring to Figure 4 As shown in the figure, the hardware accelerator VTA model deployment scheme support extension device 400 can include an instruction stream generation module 410, a model inference process module 420, and an inference result generation module 430. Wherein:

[0092] The instruction stream generation module 410 is configured to import a pre-trained model into a deep learning compiler, run model compilation with a hardware accelerator as a target, generate a low-level intermediate representation, and generate a running environment for hardware accelerator compilation. The hardware accelerator receives incoming low-level intermediate representations and generates an instruction stream;

[0093] The model inference process module 420 is configured to process the instruction stream queue and drive the model inference process on the hardware;

[0094] The inference result generation module 430 is configured to perform calculations on input and weight data loaded into on-chip cache by the hardware accelerator, obtain the inference result of the model, and perform post-processing on the model inference result and generate a visualized result output.

[0095] The details of the model deployment scheme support extension of the hardware accelerator VTA of each of the above have been described in the corresponding model deployment scheme support extension method of the hardware accelerator VTA, and thus will not be repeated here.

[0096] It should be noted that although several modules or units of the model deployment scheme support extension device 400 of the hardware accelerator VTA are mentioned in the above detailed description, such a division is not mandatory. In fact, according to embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided into modules or units embodied by multiple modules or units.

[0097] In addition, in the exemplary embodiments of the present disclosure, an electronic device capable of implementing the above method is also provided.

[0098] Those skilled in the art can understand that various aspects of the present application can be implemented as a system, method or program product. Therefore, various aspects of the present application can be embodied as a complete hardware embodiment, a complete software embodiment (including firmware, microcode, etc.), or an embodiment combined with hardware and software aspects, which can be collectively referred to as "circuitry", "module" or "system" here.

[0099] The electronic device 500 according to such an embodiment of the present application will be described below with reference to Figure 5 Figure 5 The electronic device 500 shown is merely an example and should not impose any limitation on the functions and use range of the embodiments of the present application.

[0100] As shown in Figure 5 The electronic device 500 is in the form of a general computing device. The components of the electronic device 500 can include, but are not limited to, the at least one processing unit 510 described above, the at least one storage unit 520 described above, a bus 530 connecting different system components including the storage unit 520 and the processing unit 510, and a display unit 540.

[0101] The storage unit stores program code that can be executed by the processing unit 510, so that the processing unit 510 performs the steps according to various exemplary embodiments of the present application described in the "Exemplary Method" section of the present specification. For example, the processing unit 510 can perform steps S110 to S130 as shown in Figure 1

[0102] ​​The storage unit 520 can include a readable medium in the form of volatile storage such as random access memory (RAM) 5201 and / or cache memory 5202, and also can include a non-volatile storage such as read only memory (ROM) 5203.

[0103] The storage unit 520 also can include a program / utility 5204 having a set of program modules 5205 such as an operating system, one or more application programs, other program modules, and program data, each of which can give the electronic device 500 its functionality, at least in part. The program modules 5205 can include, but are not limited to, a network environment implementation, either alone or in some combination.

[0104] The bus 530 can represent one or more of several types of bus structures, including a storage bus or bus for storage controller, a peripheral bus, an accelerated graphics port, a processor or local bus using any of a variety of bus architectures.

[0105] The electronic device 500 also can communicate with one or more external devices 570 such as a keyboard or pointing device, a Bluetooth device, etc.; other devices that enable a user to interact with the electronic device 500; and / or any devices (e.g., a router, a modem, a printer, etc.) that enable the electronic device 500 to communicate with one or more other computing devices. Such communication can occur via an input / output (I / O) interface 550. Still yet, the electronic device 500 can communicate with one or more networks, such as a local area network (LAN), a general area network (WAN), and / or the public network, such as the Internet, via a network adapter 560. As depicted, the network adapter 560 communicates with the other components of the electronic device 500 via the bus 530. It should be appreciated that although the network adapter 560 is depicted as a single component, the network adapter 560 can comprise a plurality of components that work in cooperation to provide the network connectivity. Further, it should be appreciated that the bus 530 can be implemented using any type of communications fabric that can provide for communication between the components of the electronic device 500.

[0106] Those skilled in the art will readily appreciate that the example embodiments described herein can be implemented by software and / or firmware in addition to the necessary hardware. Thus, the techniques that would be implemented according to the embodiments of the present disclosure can be embodied as a software product. The software product can be stored in a non-volatile storage medium such as a CD-ROM, a USB, a mobile hard disk, etc. or on a network, and includes a plurality of instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to perform the methods according to the embodiments of the present disclosure.

[0107] In exemplary embodiments of the present disclosure, a computer readable storage medium having stored thereon a program product capable of implementing the above-described method of the specification is also provided. In some possible embodiments, various aspects of the present application can also be implemented in the form of a program product including program code, which, when run on a terminal device, causes the terminal device to perform the steps described in the above "Exemplary Method" section according to various exemplary embodiments of the present application.

[0108] Reference Figure 6 As shown, a program product 600 for implementing the above-described method according to embodiments of the present application is described, which can take the form of a portable compact disc read-only memory (CD-ROM) and include program code, and can be run on a terminal device, such as a personal computer. However, the program product of the present application is not limited thereto, and in the present document, the readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus or device.

[0109] The program product can take any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium, for example, can be, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or any combination thereof. More specific examples (non-exhaustive list) of the readable storage medium include an electrical connection having one or more wires, a portable disc, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0110] The computer readable signal medium can include a data signal propagated in baseband or propagated as a carrier wave in a propagated data signal, in which the readable program code is carried. Such propagated data signal can take various forms, including but not limited to electromagnetic signal, optical signal or any suitable combination thereof. The readable signal medium can also be any readable medium other than the readable storage medium, which can send, propagate or transmit the program for use by or in conjunction with an instruction execution system, apparatus or device.

[0111] The program code contained on the readable medium can be transmitted using any suitable medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination thereof.

[0112] The program code may, for example, be written in any combination of one or more programming languages, including an object oriented programming language such as Java, C++ or the like, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computing device, partly on the user's computing device, as a stand-alone software package, partly on the user's computing device and partly on a remote computing device or entirely on the remote computing device or server. In the latter scenario, the remote computing device can be connected to the user's computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computing device, for example, through the Internet using an Internet Service Provider.

[0113] Furthermore, the above-described diagrams are merely schematic illustrations of the processes included in the method according to the exemplary embodiments of the present application, and are not intended for limiting purposes. It is readily understood that the processes shown in the above-described diagrams do not indicate or limit the time sequence of the processes. In addition, it is readily understood that the processes can be executed, for example, synchronously or asynchronously in a plurality of modules.

[0114] Other embodiments of the disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the features disclosed herein. For example, the specification and examples do not necessarily limit the scope of the disclosure. It is contemplated that any method or system that incorporates elements of the general inventive concepts disclosed herein can reasonably fall within the scope of the disclosure. The specification and examples do not limit the scope of the disclosure. It is contemplated that any method or system that incorporates elements of the general inventive concepts disclosed herein can reasonably fall within the scope of the disclosure. Such embodiments of the disclosure can vary from the disclosure as described and as claimed, and it is understood that this application is intended to cover any alternatives, modifications, and equivalents as can be included within the scope of the disclosure. The true scope and spirit of the disclosure are indicated by the appended claims.

[0115] It is to be understood that the present disclosure is not limited to the precise construction described and as shown in the accompanying drawings, and that changes can be made to the exact construction and arrangement of parts without departing from the scope of the present disclosure. The scope of the present disclosure is limited only by the claims appended hereto.

Claims

1. A method for supporting the expansion of a hardware accelerator VTA model deployment scheme, characterized in that, The method includes: The pre-trained model is imported into the deep learning compiler, and the model is compiled and executed with the hardware accelerator as the target to generate a low-level intermediate representation. The hardware accelerator compiles and generates a runtime environment, and the hardware accelerator receives the input low-level intermediate representation and generates an instruction stream. The instruction stream queue is processed to drive the model inference process on the hardware; The hardware accelerator calculates the input and weight data loaded into the on-chip cache to obtain the inference results of the model, and then performs post-processing on the model inference results to generate visualization results output.

2. The method as described in claim 1, characterized in that, The method further includes: Prepare a deep learning model to be deployed to a hardware platform, import the pre-trained model into a deep learning compiler, run the model compilation with the hardware accelerator as the target, transform the deep learning model into a high-level graph structure for graph-level optimization, and then reduce it to a low-level intermediate representation for operator-level optimization to generate a low-level intermediate representation for model inference on the hardware accelerator.

3. The method as described in claim 2, characterized in that, The method further includes: Define the configurable parameters of the hardware accelerator and compile to generate the runtime environment for the deep learning compiler and hardware accelerator.

4. The method as described in claim 3, characterized in that, The method further includes: The hardware accelerator receives the low-level intermediate representation and determines the compilation optimization scheme of the application based on the low-level intermediate representation. For the low-level intermediate representations of the application's single-variable data loading scheme and multi-variable data loading scheme, it generates instruction streams by adopting the instruction generation schemes of the single-variable data loading scheme and the multi-variable data loading scheme, respectively.

5. The method as described in claim 1, characterized in that, The method further includes: The instruction stream queue is processed by partially decoding the instructions through the instruction fetch module and pushing the instructions to the command queue, which then enters the loading, calculation, and storage module.

6. The method as described in claim 5, characterized in that, The method further includes: The hardware accelerator drives the model inference process on the hardware through the processed instruction stream queue, and the hardware module completes the data loading, calculation and storage process.

7. The method as described in claim 1, characterized in that, The method further includes: The hardware accelerator uses the GEMM kernel of the hardware accelerator to perform intensive linear algebra computations, uses the tensor ALU of the hardware accelerator to perform general computations, performs computations on the input and weight data loaded into the on-chip cache, obtains the inference results of the model, and performs post-processing. After post-processing the model inference calculation results, a visual result is generated and output to the user.

8. A model deployment scheme for a hardware accelerator VTA that supports expansion devices, characterized in that, The device includes: The instruction stream generation module is used to import the pre-trained model into the deep learning compiler, run the model compilation with the hardware accelerator as the target, generate a low-level intermediate representation, and the hardware accelerator compiles and generates a runtime environment. The hardware accelerator receives the input low-level intermediate representation and generates an instruction stream. The model inference process module is used to process the instruction stream queue and drive the model inference process on the hardware. The inference result generation module is used by the hardware accelerator to calculate the input and weight data loaded into the on-chip cache, obtain the inference result of the model, and post-process the model inference result to generate a visualization result output.

9. An electronic device, characterized in that, include Processor; and A memory storing computer-readable instructions that, when executed by the processor, implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by a processor, implements the method according to any one of claims 1 to 7.

Images