DNN universal deployment method and system based on heterogeneous system

Abstract

The invention relates to the technical field of artificial intelligence hardware acceleration, and relates to a DNN universal deployment method and system based on a heterogeneous system. The method comprises the steps of performing fine-grained splitting on a to-be-deployed deep neural network, and dividing the to-be-deployed deep neural network into a plurality of sub-layers; based on hardware parameters of the heterogeneous system, mapping the sub-layer to a computing core of the heterogeneous system, and performing preliminary mapping distribution; modeling communication links of the heterogeneous system to form a preliminary distribution scheme, and estimating delay and power consumption of each communication link; communication delay and power consumption in the data transmission process are monitored in real time, offline iterative optimization is conducted on the distribution scheme through a mathematical programming optimization algorithm, and a final optimization distribution scheme is generated; and based on the final optimal distribution scheme, controlling a calculation core of the heterogeneous system to execute a sub-layer calculation task, and managing a tensor life cycle of a storage module. The method can better adapt to different application requirements, and the autonomy and the operation efficiency of the system are greatly improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence hardware acceleration technology, and more particularly to a general DNN deployment method and system based on heterogeneous systems. Background Art

[0002] At present, with the continuous development of DNN, its parameter scale has shown explosive growth. Taking some advanced image recognition models as an example, the number of parameters can reach billions or even tens of billions. This large-scale parameter growth has brought many severe challenges to the system. From the perspective of power consumption, in order to process such a large number of parameter calculations, hardware devices need to consume a lot of electricity. The server clusters used to run DNN models in data centers consume an astonishing amount of electricity every year, which not only increases operating costs, but also puts higher requirements on energy supply. In terms of latency, a large number of computing tasks prolongs the data processing time. In application scenarios with extremely high real-time requirements, such as autonomous driving and real-time video surveillance, long delays may lead to serious consequences and make it impossible to make accurate decisions in time. In terms of memory resources, large parameters require a large amount of memory space, which makes memory management extremely complicated and prone to problems such as insufficient memory or memory fragmentation.

[0003] In order to meet these challenges, the architecture of DNN accelerators continues to evolve. Early single-core designs have been gradually eliminated, and dedicated architectures have emerged. In recent years, heterogeneous design has become a focus of research and attention. Heterogeneous systems are usually composed of multiple different types of computing cores, such as CPU, GPU, FPGA, ASIC, etc. Each core has its own unique performance advantages and applicable scenarios. By combining different cores, their respective advantages can be fully utilized to improve the overall performance of the system. For example, GPUs are good at processing large-scale parallel computing tasks and are suitable for convolutional layer calculations in DNNs; FPGAs are reconfigurable and can be customized according to different algorithm requirements; ASICs have extremely high efficiency and low power consumption for specific computing tasks.

[0004] In heterogeneous multi-core architectures, it is crucial to design effective layer scheduling methods to improve core utilization. However, existing layer scheduling strategies have many limitations. On the one hand, most existing scheduling strategies are specific to a certain hardware architecture and lack versatility and generality. For example, some scheduling strategies designed for a specific GPU architecture cannot be effectively applied to other types of GPUs or heterogeneous systems containing multiple computing cores, which requires developers to redesign scheduling strategies when facing different hardware platforms, increasing development costs and time. On the other hand, existing scheduling strategies also have a lot of room for improvement in performance. They often fail to fully consider factors such as complex resource distribution, data communication delays, and collaborative work between different computing cores in heterogeneous systems, resulting in low system resource utilization and inability to achieve optimal overall performance.

[0005] In summary, there is an urgent need for a universal and efficient solution to accelerate the deployment of DNN on heterogeneous systems to overcome the shortcomings of existing technologies and meet the growing demand for artificial intelligence applications.

[0006] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present application, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0007] In order to provide a basic understanding of some aspects of the disclosed embodiments, a brief summary is given below. The summary is not an extensive review, nor is it intended to identify key / critical elements or delineate the scope of protection of these embodiments, but rather serves as a prelude to the detailed description that follows.

[0008] The disclosed embodiments provide a general DNN deployment method and system based on heterogeneous systems. During each DNN operation iteration, the delay and power consumption of the allocation scheme are monitored, and the allocation scheme is optimized in real time through mathematical modeling methods, thereby maximizing the use of hardware resources, improving hardware utilization, and enhancing system efficiency and energy efficiency.

[0009] In some embodiments, the method comprises: Fine-grained splitting of the deep neural network to be deployed into multiple sub-layers as the smallest unit for heterogeneous system scheduling, and maintaining the data dependency between sub-layers during the splitting process; Based on the hardware parameters of the heterogeneous system, the sublayer is mapped to the computing core of the heterogeneous system to perform preliminary mapping allocation; Modeling the communication links of the heterogeneous system, forming a preliminary allocation plan, and estimating the delay and power consumption of each communication link; According to the preliminary allocation plan, the communication delay and power consumption in the data transmission process are monitored in real time, and the allocation plan is optimized offline through mathematical programming optimization algorithm to generate a final optimized allocation plan; Based on the final optimized allocation scheme, the computing core of the heterogeneous system is controlled to execute the sub-layer computing tasks, and the tensor life cycle of the storage module is managed.

[0010] Preferably, during the fine-grained splitting process of the deep neural network, the input-output dependency relationship between each sub-layer is maintained through a data dependency matrix.

[0011] Preferably, the delay of the communication link The calculation formula is as follows; , in, Indicates the size of the communication packet in the computing core array, Indicates the data bit width of the dual-end computing core interface, Indicates energy consumption.

[0012] Preferably, the mathematical programming optimization algorithm performs offline iterative optimization on the allocation scheme, taking minimization of total delay and total power consumption as the objective function, and optimizing the allocation variables; The following constraints are imposed during the optimization process: each sublayer is assigned to only one computing core; the allocation scheme must satisfy the hierarchical dependency of the data dependency matrix; the weight storage capacity of each computing core does not exceed the upper limit of the storage module directly connected to it; the communication link delay of a special path does not exceed the preset threshold; The optimization is repeated until the preset number of iterations is reached or the performance index is met.

[0013] Preferably, the data transmission process is specifically performed as follows: According to the allocation scheme, the tensors corresponding to the sub-layers are transferred to the target computing core; If the target storage module has sufficient space, the active variables of all CLs involved are set to 1. After the transfer is completed, the delay and power consumption of all active CLs are accumulated to obtain the estimated total delay and power consumption. If the target storage module has insufficient space, the existing tensors are evicted according to priority, and the priority is calculated based on the following factors: whether the computing core directly connected to the storage module has completed the computing task involving the tensor; the size of the tensor and the time interval between the last use; If there is still insufficient space after eviction, a refresh signal is sent to the off-chip storage, the tensor set is repartitioned and the transmission process is repeated until the tensor transmission is completed and the estimated delay and power consumption are obtained. At this time, the estimated total delay and power consumption are the accumulated delay and power consumption of all active CLs and the delay and power consumption of the tensor refresh signal.

[0014] Preferably, when the storage space is insufficient in the prioritization of the existing tensors, the tensors with the lowest priority and the smallest volume are preferentially evicted.

[0015] In some embodiments, the system includes the following modules: Allocation Manager: used to maintain allocation variables, which are used to indicate the allocation of sublayers on the target core. When the allocation variable set changes, it sends an enable signal and the label of the new target core to the storage board that stores the tensor set required by the sublayer, regenerates the packet header, and activates data transmission; Transmission manager: Maintains the power consumption and bit width of each CL based on CL, monitors the CL status through active variables and packet size, estimates the transmission delay and records it; Storage manager: tracks the complete life cycle of each tensor in memory, performs memory loading and eviction operations, and checks whether the tensor set is stored on the board directly connected to the target core before the target core performs operations. If not, it determines whether the storage space is sufficient. If there is insufficient space, it performs eviction operations. If there is still insufficient space after all tensors are evicted, it sends a tensor refresh signal to the off-chip storage; Planning optimization module: It is a RISCV-based planning optimization accelerator that generates task allocations that map sublayers to computing cores, that is, allocates variable sets, and uses a mathematical programming optimizer to perform offline optimization of heterogeneous system resource deployment. The optimization objective functions are total power consumption and total delay.

[0016] Preferably, the planning optimization module has a built-in hash lookup table for maintaining the mapping relationship between the sub-layer topology structure and the computing core, and constraining the optimization process through a data dependency matrix.

[0017] Preferably, the transmission manager monitors the handshake feedback signals of each computing core through active variables, dynamically updates the active status of the communication link, and when the handshake feedback signal is pulled high, the active variable signal of the corresponding CL is pulled high, and the packet size data is requested and recorded from the sending core of the handshake feedback signal, that is, the recipient of the current communication.

[0018] Preferably, the planning optimization module adopts Gurobi optimizer.

[0019] The embodiments of the present disclosure provide a DNN general deployment method and system based on heterogeneous systems, which can achieve the following technical effects: After the layer allocation is completed, the present invention further refines the scheduling process. Each layer is subdivided into a more fine-grained sublayer (SL). Although this subdivision increases the difficulty of scheduling, it can output the data required for the next layer of operation more quickly and improve the operating efficiency of the system. For application scenarios with relatively low requirements for real-time and accuracy, multiple layers can be integrated and regarded as a whole for operation scheduling. This flexible scheduling method can better adapt to different application requirements.

[0020] In order to solve the problem that the existing dedicated solutions for DNN allocation and deployment on heterogeneous systems lack scalability, the present invention optimizes the allocation scheme in real time by monitoring the delay and power consumption of the allocation scheme during each DNN operation iteration, and then applies the optimized scheme to the heterogeneous system by using mathematical modeling methods. The entire optimization process is completely offline, without the need for additional operations on the host side, which greatly improves the autonomy and operation efficiency of the system.

[0021] The above general description and the following description are exemplary and explanatory only and are not intended to limit the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] One or more embodiments are exemplarily described by corresponding drawings, which do not limit the embodiments. Elements with the same reference numerals in the drawings are shown as similar elements, and the drawings do not constitute a scale limitation, and wherein: Figure 1 It is a schematic flow chart of the method of the present invention; Figure 2 Schematic diagram of fine-grained layer splitting of DNN; Figure 3 A schematic diagram of the topological storage structure of the computing array for heterogeneous systems; Figure 4 A schematic diagram of the many-core topology for computing array topology of heterogeneous systems; Figure 5 It is a schematic diagram of the structure of the computing array module of the heterogeneous system; Figure 6 Schematic diagram of tensor transmission process. DETAILED DESCRIPTION

[0023] In order to be able to understand the features and technical contents of the embodiments of the present disclosure in more detail, the implementation of the embodiments of the present disclosure is described in detail below in conjunction with the accompanying drawings. The attached drawings are for reference only and are not used to limit the embodiments of the present disclosure. In the following technical description, for the convenience of explanation, a full understanding of the disclosed embodiments is provided through multiple details. However, one or more embodiments can still be implemented without these details. In other cases, to simplify the drawings, well-known structures and devices can be simplified for display.

[0024] The terms "first", "second", etc. in the specification and claims of the embodiments of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchanged where appropriate, so as to describe the embodiments of the embodiments of the present disclosure described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions.

[0025] Unless otherwise stated, the term "plurality" means two or more.

[0026] A general deployment method for DNN based on heterogeneous systems. The core of this method lies in the resource evaluation and scheduling of the hardware part, focusing on the communication cost and transmission delay of the data link. In the software part, the DNN to be deployed is firstly fine-grained layer split into a set of multiple sub-layers (SLs). Each SL can be scheduled separately and is the smallest unit for scheduling the entire heterogeneous system. During the splitting process, data dependencies are strictly maintained to ensure the correctness of the calculation. At the same time, according to the hardware parameters of each core, all SLs are mapped to each core to complete the preliminary mapping allocation. In the hardware part, various modules work together to achieve efficient scheduling evaluation.

[0027] Specifically, Figure 1 As shown, including: S1: Fine-grained splitting of the deep neural network to be deployed into multiple sub-layers as the smallest unit for heterogeneous system scheduling, and maintaining the data dependency between sub-layers during the splitting process.

[0028] S2: Based on the hardware parameters of the heterogeneous system, the sublayer is mapped to the computing core of the heterogeneous system to perform preliminary mapping allocation.

[0029] S3: Model the communication links of the heterogeneous system, form a preliminary allocation plan, and estimate the delay and power consumption of each communication link.

[0030] S4: According to the preliminary allocation plan, the communication delay and power consumption during the data transmission process are monitored in real time, and the allocation plan is optimized offline through mathematical programming optimization algorithm to generate a final optimized allocation plan.

[0031] S5: Based on the final optimized allocation plan, control the computing core of the heterogeneous system to execute the sub-layer computing tasks and manage the tensor life cycle of the storage module.

[0032] As a refinement of the above embodiment, during the fine-grained splitting process of the deep neural network, the input-output dependency relationship between each sub-layer is maintained through a data dependency matrix.

[0033] As a refinement of the above embodiment, Figure 2 As shown in the figure, each transmission link between cores is called a CL (communication,link). The bandwidth performance of the CL can be predetermined when forming a heterogeneous system, that is, the data bit width of the dual-end computing core interface, the delay of the communication link The calculation formula is as follows; , in, Indicates the size of the communication packet in the computing core array, Indicates the data bit width of the dual-end computing core interface, Represents energy consumption (process, length, clock frequency). By estimating the manufacturing process, physical link length and clock frequency, the power consumption of each CL can be estimated during system initialization and maintained in the transmission manager in the form of a lookup table.

[0034] As a refinement of the above embodiment, the mathematical programming optimization algorithm performs offline iterative optimization on the allocation scheme, taking minimization of total delay and total power consumption as the objective function, and optimizing the allocation variables; The following constraints are imposed during the optimization process: each sublayer is assigned to only one computing core; the allocation scheme must satisfy the hierarchical dependency of the data dependency matrix; the weight storage capacity of each computing core does not exceed the upper limit of the storage module directly connected to it; the communication link delay of a special path does not exceed the preset threshold; The optimization is repeated until the preset number of iterations is reached or the performance index is met.

[0035] As a refinement of the above embodiment, the data transmission process is specifically as follows: According to the allocation scheme, the tensors corresponding to the sub-layers are transferred to the target computing core; If the target storage module has sufficient space, the active variables of all CLs involved are set to 1. After the transfer is completed, the delay and power consumption of all active CLs are accumulated to obtain the estimated total delay and power consumption. If the target storage module has insufficient space, the existing tensors are evicted according to priority, and the priority is calculated based on the following factors: whether the computing core directly connected to the storage module has completed the computing task involving the tensor; the size of the tensor and the time interval between the last use; If there is still insufficient space after eviction, a refresh signal is sent to the off-chip storage, the tensor set is repartitioned and the transmission process is repeated until the tensor transmission is completed and the estimated delay and power consumption are obtained. At this time, the estimated total delay and power consumption are the accumulated delay and power consumption of all active CLs and the delay and power consumption of the tensor refresh signal.

[0036] As a refinement of the above embodiment, when the storage space is insufficient in the prioritization of existing tensors, the tensor with the lowest priority and the smallest volume is preferentially evicted.

[0037] Example 2 A DNN general deployment system based on heterogeneous systems, including the following modules: Allocation Manager: It is used to maintain allocation variables, which are used to indicate the allocation of sub-layers on the target core. When the allocation variable set changes, it sends an enable signal and the label of the new target core to the storage block that stores the tensor set required by the sub-layer, regenerates the packet header, and activates data transmission.

[0038] Transmission Manager: Maintains the power consumption and bit width of each CL based on the CL dimension, monitors the CL status through active variables and packet size, estimates the transmission delay and records it.

[0039] Storage Manager: Tracks the complete life cycle of each tensor in memory, performs memory loading and eviction operations, and checks whether the tensor set is stored on the board directly connected to the target core before the target core performs operations. If not, it determines whether the storage space is sufficient. If there is insufficient space, it performs eviction operations. If there is still insufficient space after all tensors are evicted, it sends a tensor refresh signal to the off-chip storage.

[0040] Planning optimization module: It is a RISCV-based planning optimization accelerator that generates task allocations that map sublayers to computing cores, that is, allocates variable sets, and uses a mathematical programming optimizer to perform offline optimization of heterogeneous system resource deployment. The optimization objective functions are total power consumption and total delay.

[0041] As a refinement of the above embodiment, Figure 5 The allocation manager is the allocation management module of the entire heterogeneous system, mainly maintaining allocation variables ,(show The allocation on core j. If it is 1, core j is responsible for executing The computational tasks Indicates sublayers). Assigned to core j, the input tensor set Tinput and the weight tensor set Tweight together constitute The required tensor set .when The collection changes, indicating that the allocation mapping relationship has changed. The allocation manager sends a The storage module sends an enable signal and the label of the new destination core j, regenerates the packet header, and activates data transmission. The specific data path is determined by the routing strategy and on-chip network.

[0042] As a refinement of the above embodiment, Figure 5 The transmission manager in the example is an adaptive hardware module designed for the above scheduling evaluation scheme. The transmission manager maintains the power consumption and bit width of each CL based on the CL dimension, and monitors the CL status through the active variable act and the packet size packet_size. The active variable act is modified by monitoring the handshake feedback Response signal of each computing core. When the Response signal is pulled high, it means that an inter-core communication has completed the handshake and started. Then the act signal of the corresponding CL is pulled high, and the packet size data is requested and recorded from the core that sends the Response signal, that is, the receiver of the current communication. Subsequently, the transmission manager estimates the transmission delay and records it.

[0043] As a refinement of the above embodiment, Figure 5 The storage manager in is another additional hardware module designed for the storage side. Its purpose is to track the complete life cycle of each tensor in memory and perform memory loading and eviction operations. Before performing related operations, the core j needs to be moved to the target core. Before starting, Figure 2 As shown in the overall architecture of heterogeneous systems, the core and storage modules are not fully connected. The storage manager first checks If it is stored on the board directly connected to core j, that is, the input and output value storage IOMem_j and the weight value storage WeightMem_j, the cost is only the storage access consumption e_memor and the access latency t_memor. Otherwise, the storage manager determines whether the storage space is sufficient to accommodate the upcoming tensor set.

[0044] If there is enough space, the tensor is moved to the target memory. The process of moving is controlled by the allocation manager and follows the preset transfer strategy. At the same time, the transfer manager sets the active variable act of all CLs involved to 1. When the transfer of Ti is completed, the transfer manager accumulates the delay and power consumption of all active CLs to obtain the estimated total delay and power consumption: , , in, represents the total delay, Indicates the latency on the active CL involved, Indicates memory access latency. Represents the total power consumption, Indicates the power consumption on the active CL involved, Represents the power consumption of memory access delay. The power consumption and delay of each CL are obtained through a lookup table.

[0045] If there is insufficient space, the storage space needs to be evicted, and one or more low-priority tensors will be evicted. The priority of the tensor is maintained in the storage manager and is affected by the system design, size, and recent usage interval. Specifically, the storage manager checks whether the computing cores directly connected to the current storage block have completed all calculations involving the tensor. The more cores that have not completed, the higher the priority of the tensor. When the eviction operation needs to be performed, the storage manager retrieves all the lowest priority tensors and evicts the smallest tensors first to minimize the negative impact of tensor unloading, and then confirms again whether there is enough space. If all tensors are evicted and the remaining space still cannot accommodate the upcoming tensors, the storage manager sends a tensor refresh signal T_flush to the off-chip storage, requesting the re-division of the tensor transfer package to obtain a smaller set of tensors. Repeat the above steps until the tensor transfer is completed and the estimated delay and power consumption are obtained. The estimated total delay and power consumption at this time are: , , in, Indicates the delay on the active CL involved when there is insufficient space. Indicates the time required to send the tensor refresh signal and reorganize the tensor package. Indicates the power consumption on the active CL involved when there is insufficient space. Represents the power consumption required for sending tensor refresh signals and reorganizing tensor packets.

[0046] As a refinement of the above embodiment, Figure 5 The planning optimization module is a RISC-V-based planning optimization accelerator. It uses the highly modular and customizable features of RISC-V ISA to streamline the structure. This module generates task allocations that map SLs to computing cores, i.e. The SL granularity topology of the entire DNN and its mapping relationship with the computing core are maintained through a built-in hash lookup table, and the data dependency matrix Matrix_dependency between each SL is reflected. This module mainly deploys a mathematical programming optimizer. Gurobi is used in this solution to perform offline optimization on heterogeneous system resource deployment.

[0047] The objective functions of optimization are total power consumption and total delay. The total delay is the main optimization target. At the same time, the optimization task meets the following constraints: (1) , which indicates that one SL can only be assigned to one computing core; (2) The allocation of SLs on the core must satisfy the data dependencies between the layers and must comply with the results of the lookup table; (3) The total amount of weights stored on each core must not exceed the weight storage limit of the blocks directly connected to the core; (4) For special paths, the data link delay needs to meet the path delay requirements.

[0048] The planning optimization module performs task optimization tasks and generates optimized Collection. Based on the new collection Allocate and estimate the total latency and total power consumption of the optimized system. Repeat this step until the number of optimization iterations reaches the upper limit or meets the preset performance requirements. At this point, the optimization deployment plan of the entire DNN on the heterogeneous system is completed.

[0049] The above description and the accompanying drawings fully illustrate the embodiments of the present disclosure so that those skilled in the art can practice them. Other embodiments may include structural, logical, electrical, process and other changes. The embodiments represent only possible changes. Unless explicitly required, separate components and functions are optional, and the order of operations may vary. Parts and features of some embodiments may be included in or replace parts and features of other embodiments. Moreover, the words used in this application are only used to describe the embodiments and are not used to limit the claims. As used in the description of the embodiments and the claims, unless the context clearly indicates, the singular forms of "a", "an" and "the" are intended to include plural forms as well. Similarly, the term "and / or" as used in this application refers to any and all possible combinations of one or more associated listings. In addition, when used in this application, the term "comprise" and its variants "comprises" and / or comprising refer to the presence of stated features, wholes, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or groups thereof. In the absence of further restrictions, the elements defined by the sentence "including one..." do not exclude the presence of other identical elements in the process, method or device including the elements. In this article, each embodiment may focus on the differences from other embodiments, and the same and similar parts between the embodiments may refer to each other. For the methods, products, etc. disclosed in the embodiments, if they correspond to the method part disclosed in the embodiments, then the relevant parts can refer to the description of the method part.

[0050] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software may depend on the specific application and design constraints of the technical solution. The technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the embodiments of the present disclosure. The technicians may clearly understand that for the convenience and simplicity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here.

[0051] In the embodiments disclosed herein, the disclosed methods and products (including but not limited to devices, equipment, etc.) can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units can be only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between each other shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms. The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the units may be selected according to actual needs to implement this embodiment. In addition, each functional unit in the embodiment of the present disclosure may be integrated in a processing unit, or each unit may exist physically separately, or two or more units may be integrated in one unit.

[0052] The flowchart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to the embodiment of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of the code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. In some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, which can depend on the functions involved. In the description corresponding to the flowchart and the block diagram in the accompanying drawings, the operations or steps corresponding to different boxes can also occur in a different order from the order disclosed in the description, and sometimes there is no specific order between different operations or steps. For example, two consecutive operations or steps can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, which can depend on the functions involved. Each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented by a dedicated hardware-based system that performs the specified functions or actions, or may be implemented by a combination of dedicated hardware and computer instructions.

Claims

1. A general DNN deployment method based on heterogeneous systems, characterized in that: include: Fine-grained splitting of the deep neural network to be deployed into multiple sub-layers as the smallest unit for heterogeneous system scheduling, and maintaining the data dependency between sub-layers during the splitting process; Based on the hardware parameters of the heterogeneous system, the sublayer is mapped to the computing core of the heterogeneous system to perform preliminary mapping allocation; Modeling the communication links of the heterogeneous system, forming a preliminary allocation plan, and estimating the delay and power consumption of each communication link; According to the preliminary allocation plan, the communication delay and power consumption in the data transmission process are monitored in real time, and the allocation plan is optimized offline through mathematical programming optimization algorithm to generate a final optimized allocation plan; Based on the final optimized allocation scheme, control the computing core of the heterogeneous system to execute the sub-layer computing tasks and manage the tensor life cycle of the storage module; The specific method of the data transmission process is as follows: According to the allocation scheme, the tensors corresponding to the sub-layers are transferred to the target computing core; If the target storage module has sufficient space, the active variables of all CLs involved are set to 1. After the transfer is completed, the delay and power consumption of all active CLs are accumulated to obtain the estimated total delay and power consumption. If the target storage module has insufficient space, the existing tensors are evicted according to priority, and the priority is calculated based on the following factors: whether the computing core directly connected to the storage module has completed the computing task involving the tensor; the size of the tensor and the time interval between the last use; If there is still insufficient space after eviction, a refresh signal is sent to the off-chip storage, the tensor set is repartitioned and the transmission process is repeated until the tensor transmission is completed and the estimated delay and power consumption are obtained. At this time, the estimated total delay and power consumption are the accumulated delay and power consumption of all active CLs and the delay and power consumption of the tensor refresh signal.

2. The DNN general deployment method based on heterogeneous systems according to claim 1 is characterized in that: During the fine-grained splitting process of the deep neural network, the input-output dependency relationship between each sub-layer is maintained through the data dependency matrix.

3. The DNN general deployment method based on heterogeneous systems according to claim 1 is characterized in that: The delay of the communication link The calculation formula is as follows; , in, Indicates the size of the communication packet in the computing core array, Indicates the data bit width of the dual-end computing core interface, Indicates energy consumption.

4. The DNN general deployment method based on heterogeneous systems according to claim 1 is characterized in that: The mathematical programming optimization algorithm performs offline iterative optimization on the allocation scheme, taking minimization of total delay and total power consumption as the objective function, and optimizing the allocation variables; The following constraints are imposed during the optimization process: each sublayer is assigned to only one computing core; the allocation scheme must satisfy the hierarchical dependency of the data dependency matrix; the weight storage capacity of each computing core does not exceed the upper limit of the storage module directly connected to it; the communication link delay of a special path does not exceed the preset threshold; The optimization is repeated until the preset number of iterations is reached or the performance index is met.

5. The DNN general deployment method based on heterogeneous systems according to claim 1 is characterized in that: When the storage space is insufficient in the prioritization of existing tensors, the tensor with the lowest priority and the smallest volume is preferentially evicted.

6. A DNN universal deployment system based on heterogeneous systems using any of the methods described in claims 1-5, characterized in that: Includes the following modules: Allocation Manager: used to maintain allocation variables, which are used to indicate the allocation of sublayers on the target core. When the allocation variable set changes, it sends an enable signal and the label of the new target core to the storage board that stores the tensor set required by the sublayer, regenerates the packet header, and activates data transmission; Transmission manager: Maintains the power consumption and bit width of each CL based on CL, monitors the CL status through active variables and packet size, estimates the transmission delay and records it; Storage manager: tracks the complete life cycle of each tensor in memory, performs memory loading and eviction operations, and checks whether the tensor set is stored on the board directly connected to the target core before the target core performs operations. If not, it determines whether the storage space is sufficient. If there is insufficient space, it performs eviction operations. If there is still insufficient space after all tensors are evicted, it sends a tensor refresh signal to the off-chip storage; Planning optimization module: It is a RISCV-based planning optimization accelerator that generates task allocations that map sublayers to computing cores, that is, allocates variable sets, and uses a mathematical programming optimizer to perform offline optimization of heterogeneous system resource deployment. The optimization objective functions are total power consumption and total delay.

7. The DNN universal deployment system based on heterogeneous systems according to claim 6 is characterized in that: The planning optimization module has a built-in hash lookup table for maintaining the mapping relationship between the sub-layer topology structure and the computing core, and constraining the optimization process through a data dependency matrix.

8. The DNN universal deployment system based on heterogeneous systems according to claim 6, characterized in that: The transmission manager monitors the handshake feedback signals of each computing core through active variables and dynamically updates the active status of the communication link. When the handshake feedback signal is pulled high, the active variable signal of the corresponding CL is pulled high, and the packet size data is requested and recorded from the core that sends the handshake feedback signal, that is, the recipient of the current communication.

9. The DNN universal deployment system based on heterogeneous systems according to claim 6, characterized in that: The planning optimization module adopts Gurobi optimizer.

Images