Compilation methods and systems, electronic devices and storage media for near-memory computing architectures

Abstract

This disclosure relates to a compilation method and system, electronic device, and storage medium for near-memory computing architectures. The method includes: acquiring operator-level intermediate representations of a neural network model and hardware configuration information of the near-memory computing architecture; generating a tree-like abstract model of the near-memory computing architecture; constructing a search space for compilation strategies; determining multiple candidate compilation strategies and their compilation results by performing performance prediction on the compilation strategies in the search space; simulating the compilation results corresponding to each candidate compilation strategy to obtain simulation results for each candidate compilation strategy; and determining a target compilation strategy from the multiple candidate compilation strategies, and defining the compilation result of the target compilation strategy as the target compilation result. Therefore, it can provide an adapted target compilation strategy for deploying any neural network model on any near-memory computing architecture, thereby enabling the deployment of any neural network model on any near-memory computing architecture.

Description

Technical Field

[0001] This disclosure relates to the field of near-memory computing, and more particularly to a compilation method and system, electronic device and storage medium for near-memory computing architecture. Background Technology

[0002] Near-Memory Computing (NMC) architecture is a method of integrating memory and computing units to reduce data transfer latency and improve computational efficiency. This architecture maximizes the use of the high access bandwidth of internal memory by placing computing units near the memory and leveraging high-bandwidth circuit integration techniques, thereby achieving efficient data processing.

[0003] With the development of deep learning technology, how to compile and deploy neural network models on various hardware architectures is an important research direction. At present, the toolchain for compiling and deploying neural network models on near-memory computing architectures generally includes two parts: a compiler and a simulator. The compiler is responsible for converting the algorithm input into instructions that can be executed on near-memory computing architectures, while the simulator is responsible for comparing the advantages and disadvantages of different compilation strategies that may be involved in the compilation process in order to select the appropriate compilation strategy.

[0004] However, current toolchains for compiling and deploying neural network models on near-memory computing architectures are designed specifically for different near-memory computing architectures. This limits the compilation and deployment tools to a single type or a single level of near-memory computing architecture, making them unable to adapt to different types of near-memory computing architectures or provide optimal compilation strategies for different near-memory computing architectures. Summary of the Invention

[0005] In view of this, this disclosure proposes a compilation method and system, electronic device and storage medium for near-memory computing architectures, which can provide an adapted target compilation strategy for the deployment of arbitrary neural network models on arbitrary near-memory computing architectures, thereby realizing the deployment of arbitrary neural network models on arbitrary near-memory computing architectures and having high versatility.

[0006] According to one aspect of this disclosure, a compilation method is provided, comprising: obtaining an operator-level intermediate representation of a neural network model to be deployed to a near-memory computing architecture and hardware configuration information of the near-memory computing architecture, the hardware configuration information indicating the hardware configuration of the near-memory computing architecture; generating a tree-structured abstract model of the near-memory computing architecture based on the hardware configuration information of the near-memory computing architecture, the tree-structured abstract model describing the near-memory computing architecture using a tree structure; constructing a search space for compilation strategies based on the operator-level intermediate representation of the neural network model and the tree-structured abstract model of the near-memory computing architecture, the compilation strategies being used to compile the operator-level intermediate representation; and determining multiple candidate compilation strategies by performing performance prediction on the compilation strategies in the search space. The compilation results corresponding to each candidate compilation strategy include a complete instruction sequence obtained by compiling the operator-level intermediate representation according to the candidate compilation strategy; based on the tree-structured abstract model, the execution process of the compilation results corresponding to each candidate compilation strategy on the near-memory computing architecture is simulated to obtain the simulation results corresponding to each candidate compilation strategy, and the simulation results represent the total time required for the compilation results to be executed on the near-memory computing architecture; based on the simulation results corresponding to each candidate compilation strategy, a target compilation strategy is determined from the multiple candidate compilation strategies, and the compilation result of the target compilation strategy is determined as the target compilation result of the operator-level intermediate representation of the neural network model, so as to deploy the neural network model on the near-memory computing architecture based on the target compilation result.

[0007] In one possible implementation, the near-memory computing architecture includes a multi-layered hardware structure. Each layer of the hardware structure in the near-memory computing architecture corresponds to a node in the tree-like architecture abstract model. Each layer in the tree-like architecture abstract model contains at least one central node, and each central node connects to at least one of the following child nodes: the central node of the next lower layer, storage nodes, compute nodes, and cache nodes in the layer to which the central node belongs. The central node represents the data transmission relationship between two adjacent layers. The storage node represents a storage unit in the near-memory computing architecture and contains attribute information of the storage unit; the compute node represents a compute unit in the near-memory computing architecture and contains attribute information of the compute unit; and the cache node represents a cache unit in the near-memory computing architecture and contains attribute information of the cache unit.

[0008] In one possible implementation, a single compilation strategy includes a partitioning strategy and a corresponding mapping strategy. The partitioning strategy represents the partitioning result of dividing the operator-level intermediate representation along multiple dimensions at multiple levels. The mapping strategy represents the way to map the partitioning result of the operator-level intermediate representation to the memory in the near-memory computing architecture. The partitioning result represents multiple sub-intermediate representations divided by the operator-level intermediate representation. The construction of the search space for the compilation strategy based on the operator-level intermediate representation of the neural network model and the tree-like architecture abstract model of the near-memory computing architecture includes: based on the operator-level intermediate representation and the tree-like architecture abstract model, partitioning the operator-level intermediate representation along multiple dimensions level by level, according to the order from the highest to the lowest level of the memory in the near-memory computing architecture, under the condition of satisfying a first constraint, to obtain multiple partitioning strategies. The first constraint includes that the product of the number of partitions along multiple dimensions at each level is less than or equal to the number of hardware structures corresponding to each level in the memory. Different partitioning strategies have different partitioning methods under the first constraint. For each sub-intermediate representation partitioned by each partitioning strategy, each sub-intermediate representation is divided into multiple data slices under the second constraint, and each data slice is mapped to the memory of the near-memory computing architecture under the third constraint, resulting in multiple mapping strategies corresponding to each partitioning strategy. The second constraint includes that the product between the size of each data slice and the data bit width of each data slice is less than or equal to the product between the burst length of the memory and the column bit width of the memory. The third constraint includes that the number of data slices mapped to the same row in the memory is less than or equal to the ratio between the total number of columns in that row in the memory and the burst length.

[0009] In one possible implementation, the step of determining multiple candidate compilation strategies and their corresponding compilation results by performing performance prediction on the compilation strategies in the search space includes: determining the performance upper limit of each compilation strategy in the search space based on the hardware data corresponding to the near-memory computing architecture, wherein the performance upper limit represents the upper limit of the time required for the compilation result of the compilation strategy to be executed on the near-memory computing architecture, and the hardware data includes the memory bandwidth, line miss cost, number and performance of computing units in the near-memory computing architecture; determining multiple initial candidate compilation strategies whose performance upper limits are higher than a preset performance threshold based on the performance upper limits of each compilation strategy in the search space; and using a compiler to partially compile the operator-level intermediate representation based on the initial candidate compilation strategies and a specific instruction set to obtain a partial instruction sequence compiled by each initial candidate compilation strategy, wherein the specific instruction set covers the instructions executed on various near-memory computing architectures. The performance predictor is used to predict the performance of partial instruction sequences compiled by each initial candidate compilation strategy, obtaining the predicted performance of the partial instruction sequences compiled by each initial candidate compilation strategy. The predicted performance characterizes the time required for the partial instruction sequences to complete execution on the near-memory computing architecture. Based on the predicted performance of the partial instruction sequences compiled by each initial candidate compilation strategy and the number of loop executions corresponding to the partial instruction sequences, the target predicted performance corresponding to each initial candidate compilation strategy is determined. The target predicted performance characterizes the time required for the complete instruction sequences compiled based on the initial candidate compilation strategies to complete execution on the near-memory computing architecture. Based on the target predicted performance of each initial candidate compilation strategy, multiple candidate compilation strategies are selected from the multiple initial candidate compilation strategies. The compiler is then used to fully compile the operator-level intermediate representation based on the multiple candidate compilation strategies and the specific instruction set, obtaining the compilation results corresponding to each candidate compilation strategy.

[0010] In one possible implementation, determining the performance upper limit of each compilation strategy in the search space based on the hardware data corresponding to the near-memory computing architecture includes: analyzing each compilation strategy in the search space to obtain the analysis results corresponding to each compilation strategy, the analysis results including the total amount of data accessed to memory and the number of line misses under each compilation strategy, as well as the computational load of the computing unit; and determining the performance upper limit of each compilation strategy in the search space based on the analysis results corresponding to each compilation strategy and the hardware data.

[0011] In one possible implementation, the step of using a performance predictor to predict the performance of partial instruction sequences compiled by each initial candidate compilation strategy, and obtaining the predicted performance of the partial instruction sequences compiled by each initial candidate compilation strategy, includes: determining, based on the partial instruction sequences compiled by each initial candidate compilation strategy, the total number of memory accesses in the near-memory computing architecture, the number of row misses in consecutive memory accesses, and the amount of data accessed by computing units to cache units or registers in the near-memory computing architecture; and determining the predicted performance of the partial instruction sequences compiled by each initial candidate compilation strategy based on the total number of memory accesses and column-to-column latency, the number of row misses and row miss costs in consecutive memory accesses, and the amount of data accessed by computing units to cache units or registers and bus bandwidth in the near-memory computing architecture.

[0012] In one possible implementation, the method utilizes a simulator based on the tree-structured abstract model to simulate the execution process of the compilation results corresponding to each candidate compilation strategy in the near-memory computing architecture, thereby obtaining the simulation results corresponding to each candidate compilation strategy. The simulator includes a hardware state space, an instruction queue, and a virtual memory controller. The hardware state space is used to simulate and track the hardware state of each hardware component in the near-memory computing architecture based on the tree-structured abstract model. The hardware components include storage units, computing units, cache units, and transmission buses. The instruction queue stores the complete instruction sequence compiled based on the candidate compilation strategies and maintains the data dependencies within the complete instruction sequence. The instruction queue includes multiple instruction groups, with independent instructions executed by different computing units located in different instruction groups. Each instruction group includes multiple sequentially executed instructions, and instructions in instruction groups without data dependencies can be issued in parallel. The virtual memory controller determines the issuance time of instructions in the instruction queue and controls the update time of the hardware state of each hardware component in the hardware state space based on the hardware state space.

[0013] In one possible implementation, tracking the hardware state of each hardware component in the near-memory computing architecture includes: tracking the activation state of each memory cell in the near-memory computing architecture to issue an activation command before an inactive memory cell is accessed; tracking the currently active row address in the memory cell to indicate whether subsequent accesses trigger a row miss and, if a row miss is triggered, scheduling a precharge command and an activation command to activate the row to be accessed; wherein, multiple clock countdowns are used in the hardware state space to track the hardware state of each hardware component, wherein the hardware state of the memory cell includes the time when the next command can be issued for the memory cell, and the hardware state of other hardware components besides the memory cell includes an idle state or an occupied state; the clock countdown corresponding to each hardware component is updated according to the hardware occupancy duration of the hardware component involved in issuing the command.

[0014] In one possible implementation, the step of using a simulator based on the tree-structured abstract model to simulate the execution process of the near-memory computing architecture for the compilation results corresponding to each candidate compilation strategy, and obtaining the simulation results corresponding to each candidate compilation strategy, includes: for the compilation result corresponding to any candidate compilation strategy, repeatedly executing the following simulation operations: the instruction queue determines the set of instructions that can be issued simultaneously based on the data dependencies involved in each instruction group, and sends the instruction set to the virtual memory controller, the instruction set including the instruction currently ranked first in each instruction group without data dependencies; the virtual memory controller determines each instruction in the instruction set according to the hardware state space. The virtual content controller determines the target instruction to be issued by selecting the instruction with the earliest issuance time. Based on the issuance time of the target instruction, the virtual content controller updates the hardware state of each hardware component in the hardware state space to the hardware state corresponding to the issuance time of the target instruction, and deletes the target instruction from the instruction queue. The virtual content controller updates the hardware state of the hardware component occupied by the target instruction in the hardware state space according to the hardware occupancy time of the hardware component occupied by the target instruction. When all instructions in the instruction queue have been issued and the hardware state of each hardware component in the hardware state space becomes idle, the simulation result corresponding to the candidate compilation strategy is obtained.

[0015] In one possible implementation, determining the issuance time of each instruction in the instruction set based on the hardware state space includes: for the i-th instruction in the instruction set, if the operand of the i-th instruction comes from a memory unit, determining the activation state of the row in the memory unit containing the operand of the i-th instruction based on the hardware state space, and determining the arrival time of the operand of the i-th instruction to the computing unit based on the activation state of the row containing the operand of the i-th instruction, the clock countdown of the memory unit, and the read delay of reading the operand from the memory unit; and for the i-th instruction... When the operands come from a cache unit, the arrival time of the operands of the i-th instruction to the computing unit is determined based on the release time of the cache unit, the read latency of reading the operands from the cache unit, and the idle time of the transmission bus between the cache unit and the computing unit. Based on the arrival time of the operands of the i-th instruction to the computing unit and the idle time of the computing unit, the start processing time of the computing unit for processing the operands of the i-th instruction is determined. Based on the start processing time of the computing unit for processing the operands of the i-th instruction, the issuance time of the i-th instruction is determined.

[0016] In one possible implementation, selecting the instruction with the earliest issuance time as the target instruction to be issued now includes: selecting the instruction with the earliest start time as the target instruction to be issued now.

[0017] According to another aspect of this disclosure, a compilation system is provided, comprising: an acquisition module, configured to acquire operator-level intermediate representations of a neural network model to be deployed to a near-memory computing architecture and hardware configuration information of the near-memory computing architecture, the hardware configuration information indicating the hardware configuration of the near-memory computing architecture; an abstraction module, configured to generate a tree-structured abstract model of the near-memory computing architecture based on the hardware configuration information of the near-memory computing architecture, the tree-structured abstract model describing the near-memory computing architecture using a tree structure; a construction module, configured to construct a search space for compilation strategies based on the operator-level intermediate representations of the neural network model and the tree-structured abstract model of the near-memory computing architecture, the compilation strategies being used to compile the operator-level intermediate representations; and a prediction module, configured to determine multiple candidate compilers by performing performance prediction on the compilation strategies in the search space. The system includes: a compilation strategy selection module and compilation results corresponding to each candidate compilation strategy, wherein the compilation results include a complete instruction sequence obtained by compiling the operator-level intermediate representation according to the candidate compilation strategy; a simulation module, used to simulate the execution process of the compilation results corresponding to each candidate compilation strategy on the near-memory computing architecture based on the tree-structured abstract model, to obtain simulation results corresponding to each candidate compilation strategy, wherein the simulation results characterize the total time required for the compilation results to be executed on the near-memory computing architecture; and a determination module, used to determine the target compilation strategy from the multiple candidate compilation strategies according to the simulation results corresponding to each candidate compilation strategy, and to determine the compilation result of the target compilation strategy as the target compilation result of the operator-level intermediate representation of the neural network model, so as to deploy the neural network model on the near-memory computing architecture based on the target compilation result.

[0018] According to another aspect of this disclosure, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the above-described method.

[0019] According to another aspect of this disclosure, a non-volatile computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the steps of the above-described method.

[0020] According to another aspect of this disclosure, a computer program product is provided, including a computer program or a non-volatile computer-readable storage medium carrying the computer program, wherein the computer program, when executed by a processor, implements the steps of the above-described method.

[0021] According to various aspects of this disclosure, by using a tree-structured abstract model to describe near-memory computing architectures, arbitrary near-memory computing architectures can be uniformly described using a tree structure in a temporal manner. Furthermore, by combining the operator-level intermediate representation of the neural network model, a compilation strategy search space is constructed. Then, by performing performance prediction on the compilation strategies in the search space, multiple candidate compilation strategies are determined. Finally, by simulating the execution process of the compilation results on the near-memory computing architecture using a simulator based on the abstract model, the precise simulation results corresponding to each candidate compilation strategy are obtained, thus determining the final target compilation strategy and target compilation result. This can provide a better target compilation strategy for the deployment of arbitrary neural network models on arbitrary near-memory computing architectures, thereby realizing the deployment of arbitrary neural network models on arbitrary near-memory computing architectures and exhibiting high versatility.

[0022] Other features and aspects of this disclosure will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0023] The accompanying drawings, which are included in and form part of this specification, illustrate exemplary embodiments, features, and aspects of this disclosure together with the specification and serve to explain the principles of this disclosure.

[0024] Figure 1 A flowchart illustrating a compilation method according to an embodiment of the present disclosure is shown.

[0025] Figure 2a , Figure 2b and Figure 2c Schematic diagrams of three NDP architectures according to an embodiment of the present disclosure are shown respectively.

[0026] Figure 3a , Figure 3b and Figure 3c Schematic diagrams of three tree-structure abstract models according to an embodiment of the present disclosure are shown respectively.

[0027] Figure 4 A flowchart illustrating operator partitioning and operator mapping according to an embodiment of the present disclosure is shown.

[0028] Figure 5 A schematic diagram illustrating the search space pruning and performance prediction process according to an embodiment of the present disclosure is shown.

[0029] Figure 6 A schematic diagram of a simulator architecture according to an embodiment of the present disclosure is shown.

[0030] Figure 7a and Figure 7b Schematic diagrams of two DRAM access procedures according to an embodiment of the present disclosure are shown respectively.

[0031] Figure 8 A schematic diagram of the framework of a compilation method according to an embodiment of the present disclosure is shown.

[0032] Figure 9 A block diagram of a compilation system according to an embodiment of the present disclosure is shown.

[0033] Figure 10 A block diagram of an electronic device 1900 according to an embodiment of the present disclosure is shown. Detailed Implementation

[0034] Various exemplary embodiments, features, and aspects of this disclosure will now be described in detail with reference to the accompanying drawings. The same reference numerals in the drawings denote elements that have the same or similar functions. Although various aspects of the embodiments are shown in the drawings, they are not necessarily drawn to scale unless specifically indicated otherwise.

[0035] As used herein, the terms “comprising,” “including,” “having,” or variations thereof are open-ended and include one or more of the stated features, integrals, elements, steps, components, or functions, but do not exclude the presence or addition of one or more other features, integrals, elements, steps, components, functions, or groups thereof.

[0036] When an element is referred to as “connected,” “coupled,” “responding,” or a variation thereof relative to another element, it may be directly connected, coupled, or responding to another element, or there may be an intermediate element present.

[0037] Although the terms first, second, third, etc., may be used herein to describe various elements / operations, these elements / operations should not be limited by these terms. These terms are only used to distinguish one element / operation from another. Therefore, without departing from the teachings of the inventive concept, a first element / operation in some embodiments may be referred to as a second element / operation in other embodiments.

[0038] The term “exemplary” as used herein means “serving as an example, embodiment, or illustration.” Any embodiment illustrated herein as “exemplary” is not necessarily to be construed as superior to or better than other embodiments.

[0039] Furthermore, to better illustrate this disclosure, numerous specific details are set forth in the following detailed description. Those skilled in the art will understand that this disclosure can be practiced without certain specific details. In some instances, methods, means, components, and circuits well known to those skilled in the art have not been described in detail in order to highlight the main points of this disclosure.

[0040] Figure 1 A flowchart illustrating a compilation method according to an embodiment of this disclosure is shown. Figure 1 As shown, the method includes steps S11 to S16.

[0041] In step S11, the operator-level intermediate representation of the neural network model to be deployed to the near-memory computing architecture and the hardware configuration information of the near-memory computing architecture are obtained. The hardware configuration information is used to indicate the hardware configuration of the near-memory computing architecture.

[0042] In practical applications, for example, the ONNX (Open Neural Network Exchange, an open-source deep learning model exchange format) framework can be used to transform some or all of the operators in the neural network model into a series of operator-level intermediate representations (IRs). Of course, those skilled in the art can also use other known conversion methods in the art to convert them into corresponding operator-level intermediate representations, and this disclosure does not limit such conversion.

[0043] The hardware configuration information may include the overall architecture configuration of the near-memory computing architecture and the timing parameters of the memory in the near-memory computing architecture. Timing parameters are a series of parameters describing memory performance. These parameters, measured in clock cycles, specify the timing requirements for read and write operations on memory cells. For example, taking DRAM (Dynamic Random Access Memory) as an example in a near-memory computing architecture, the timing parameters of DRAM may include: tRCD (RAS to CASDelay), the delay from row address to column address (i.e., the time from row activation to column address issuance); tCL (CAS Latency), the column address strobe delay (i.e., the time from column address issuance to data output); tRP (RASPrecharge Time), the row activation time (i.e., the shortest time from row activation to precharge); tRAS (RAS Active Time), the row activation time (i.e., the shortest time from row activation to precharge); tRC (Row Cycle Time), the row cycle time (i.e., the shortest time between two row activations); tWR (Write Recovery Time), the write recovery time (i.e., the time from write operation completion to precharge); tRRD (RAS to RASDelay), the row-to-row delay (i.e., the shortest time between two row activations); tWTR (Write... tRead Delay (write-to-read delay), i.e., the time from the completion of a write operation to the start of a read operation; tCCD (CAS to CASDelay), i.e., the shortest time between two column operations; tRTP (Read to Precharge Delay), i.e., the time from the completion of a read operation to the start of precharge; tRTP (Read to Precharge Delay), i.e., the time from the completion of a read operation to the start of precharge; tPD (Precharge Delay), i.e., the time from issuing a precharge command to its execution; tDAL (Data Access Latency), i.e., the time from issuing a read command to data output; tDPL (Data Precharge Latency), i.e., the time from issuing a precharge command to the completion of data precharge, etc. It should be understood that if other types of memory are used in the near-memory computing architecture, the timing parameters corresponding to the memory can be obtained, and this disclosure does not limit this aspect.

[0044] It should be understood that different near-memory computing architectures have different architecture configurations. Generally speaking, the architecture configuration can include at least the following: the hierarchical structure information of the memory (such as DRAM) (such as the number of channels, the number of columns in each channel, the number of devices in each column, and the number of banks in each device), the number of rows and columns and the bit width of the memory cells, the hierarchy of the near-memory cells (i.e., containing both memory cells and computing cells), the number of computing cells, the degree of computational parallelism and the degree of computational precision, etc., and the design information of the cache cells (such as cache size, read / write latency, etc.).

[0045] In this embodiment of the disclosure, the storage unit can be the smallest storage unit in the memory corresponding to the computing unit. For example, in a bank-level NDP (Near DRAM Processing) architecture, the computing unit corresponds to a storage bank in DRAM, so one storage unit can represent one bank. If in a device-level NDP architecture, the computing unit corresponds to a chip device in DRAM, then one storage unit can represent one device. This depends on the structural layout of the computing unit and storage unit in the NDP architecture, which is not limited in this embodiment of the disclosure. The NDP structure is a near-memory computing architecture.

[0046] For example, Figure 2a A schematic diagram of a bank-level NDP architecture is shown, such as... Figure 2a As shown, it can be a certain device under a certain Rank of a certain Channel. The device includes multiple Bank-level local memory units. Each Bank-level local memory unit includes a Bank PU (i.e., a Bank-level computing unit), a local cache unit (i.e., an input cache and an output cache), and a corresponding Bank-level storage unit. Figure 2b A schematic diagram of a device-level NDP architecture is shown, such as... Figure 2b As shown, each Device can include multiple Device PUs (i.e., Device-level computing units), multiple Device PUs share the same global cache, and each Device PU has a corresponding Bank-level storage unit; Figure 2c A schematic diagram of another device-level NDP architecture is shown, such as... Figure 2c As shown, a Rank contains multiple Device-level local storage units. Each Device-level local storage unit includes a Device PU, a local cache unit (i.e., input cache and output cache), and a corresponding Device-level storage unit.

[0047] It should be understood that the hierarchical structure and layout of different near-memory computing architectures can be different. For example, in some near-memory computing architectures, each Bank may include multiple groups. In this case, the hierarchical structure information of the memory may also include the number of groups in each Bank, and the computing units may correspond to the group-level storage units in the Bank. Some near-memory computing architectures may not include a chip device level in their memory. Those skilled in the art can obtain the corresponding hardware configuration information based on the actual type, structure, and layout of the near-memory computing architecture. This disclosure does not limit the method of obtaining hardware configuration information.

[0048] In step S12, based on the hardware configuration information of the near-memory computing architecture, a tree-like architecture abstract model of the near-memory computing architecture is generated. The tree-like architecture abstract model is used to describe the near-memory computing architecture using a tree structure.

[0049] To ensure that the compilation and deployment of neural network models are universally applicable to various near-memory computing architectures, this disclosure presents a unified architectural abstraction method. Specifically, it uses a tree-structured abstract model to describe different near-memory computing architectures, essentially employing a scalable tree structure to abstract them. This method can be used to describe existing near-memory computing architectures and can also be easily extended to support new ones. As mentioned above, a near-memory computing architecture includes multiple layers of hardware structure. Each layer of hardware structure in a near-memory computing architecture can correspond to a node in the tree-structured abstract model. In other words, each layer of the tree-structured abstract model can represent a hierarchical structure of the near-memory computing architecture (i.e., a hierarchical structure of memory within the near-memory computing architecture). For example, if a near-memory computing architecture includes four hierarchical structures: channel, rank, device, and bank, then the tree-structured abstract model can contain four layers of nodes (specifically, four central nodes). If a new near-memory computing architecture contains a new hierarchical structure (such as a group), a new node hierarchy can be introduced into the abstract model for expansion. If a certain hierarchical structure does not exist in the near-memory computing architecture, the corresponding node can be removed. For example, HBM (High Bandwidth Memory) does not contain a chip device hierarchy, so the corresponding tree-structured abstract model does not contain nodes of that hierarchy.

[0050] In the tree-structured abstract model, each layer contains at least one central node. Each central node connects to at least one of the following child nodes: the central node of the next lower level (a non-bottom-level node), storage nodes (bottom-level nodes), compute nodes (bottom-level nodes), and cache nodes (bottom-level nodes) within the layer to which the central node belongs. The central node represents the data transmission relationship between two adjacent layers. Specifically, the central node can represent a data transmission interface (e.g., a data bus) from the next lower level to the previous level, and the data transmission interface describes the data transmission relationship. A non-bottom-level node indicates that the node is not the last-level node, while a bottom-level node indicates that the node is the last-level node.

[0051] In this context, a storage node represents a storage unit in a near-memory computing architecture and contains the attribute information of the storage unit. That is, a storage node is the smallest storage unit corresponding to a processing unit. For example, in a bank-level NDP architecture, each storage node can represent a bank. The attribute information contained in the storage node can include at least the read / write bit width, capacity, and timing parameters of the storage unit.

[0052] In this context, a compute node represents a compute unit within a near-storage computing architecture and contains its attribute information. Specifically, a compute node can represent a compute unit at a corresponding node level, and its attribute information can include at least the compute unit's computational parallelism, computational precision, and operating frequency. Furthermore, if a central node contains multiple storage nodes and compute nodes, this means that each compute node under the same central node can read / write data from multiple storage nodes under that central node. Users can configure the storage nodes accessible to compute nodes under the same central node according to their actual needs.

[0053] In this context, a cache node represents a cache unit in a near-storage computing architecture and contains the attribute information of that unit. This cache unit can be a global cache or a local cache; a local cache is a non-globally shared unit. The attribute information contained in a cache node can include at least: the cache size (width and depth) and read / write latency. It should be understood that a cache node can be configured as a local cache for compute nodes under the same central node, or as a global cache shared by multiple processing units under different central nodes. Therefore, compute nodes under the same central node can share other cache nodes (i.e., local caches) under the same central node, while compute units from different central nodes need to use the cache node at the next higher level (i.e., the global cache) for data sharing.

[0054] It should be understood that the computing nodes and cache nodes can be optional in the tree - structured abstract model, that is, some layers of the tree - structured abstract model may not contain computing nodes and / or cache nodes. By adjusting the configuration of nodes, the number of central nodes, the number of storage nodes, the number of computing nodes, and the number of cache nodes in each layer of the tree - structured abstract model, the generated tree - structured abstract model can cover various types of near - memory computing architectures.

[0055] Exemplarily, Figure 3a 、 Figure 3b and Figure 3c respectively show schematic diagrams of three tree - structured abstract models. Among them, Figure 3a is obtained by abstracting the bank - level NDP architecture shown in Figure 2a . Figure 3b is obtained by abstracting the device - level NDP architecture shown in Figure 2b . Figure 3c is obtained by abstracting the device - level NDP architecture shown in Figure 2c . Among them, the circle represents the central node, the triangle represents the storage node, the square represents the cache node, and the heptagon represents the computing node; as can be seen from Figure 3a 、 Figure 3b and Figure 3c , using the tree - structure can uniformly describe any near - memory computing architecture, which is beneficial to making the subsequent compilation and deployment processes adapt to various near - memory computing architectures.

[0056] In step S13, based on the operator - level intermediate representation of the neural network model and the tree - structured abstract model of the near - memory computing architecture, a search space for the compilation strategy is constructed, and the compilation strategy is used to compile the operator - level intermediate representation.

[0057] Among them, a single compilation strategy may include a partitioning strategy, a corresponding mapping strategy, and a scheduling strategy. The partitioning strategy represents the partitioning result of dividing the operator - level intermediate representation along multiple dimensions at multiple levels. The mapping strategy represents the way of mapping the partitioning result of the operator - level intermediate representation to the storage units in the near - memory computing architecture. The partitioning result represents multiple sub - intermediate representations obtained by dividing the operator - level intermediate representation; the scheduling strategy represents the order of traversing and scheduling the multiple sub - intermediate representations obtained by dividing the operator - level intermediate representation during the computing process. Since currently, usually only a fixed number of several scheduling strategies are adopted, such as horizontal traversal, vertical traversal, Z - shaped traversal, etc., the embodiments of the present disclosure mainly introduce in detail the generation of the partitioning strategy and the mapping strategy. The partitioning strategy and the mapping strategy in the same compilation strategy can be described as how to divide the operator - level intermediate representation at multiple levels of the memory and allocate it to the processing units and storage units, or how to split the operator - level intermediate representation and map it to different columns and rows of the storage units, that is, the data layout.​

[0058] To achieve high parallelism, existing NDP architectures typically partition data along the matrix dimensions that do not require reduction, with the number of partitions equal to the number of available processing units. Taking the common matrix multiplication operator O = W × A in neural network models as an example, the dimensions of the weight matrix W, input activation matrix A, and output matrix O are (M,K), (K,N), and (M,N), respectively. Existing partitioning strategies typically divide the operator-level intermediate representation of the above matrix multiplication operator along the M-dimensional axis. However, as the K and N dimensions increase, the number of updates to the corresponding input buffer also increases, leading to more input data accesses to memory, thus hindering the achievement of high computational parallelism.

[0059] Therefore, this disclosure proposes a novel partitioning method that allows data to be partitioned along multiple dimensions and processed across multiple levels. The proposed partitioning encoding method sequentially describes how operator-level intermediate representations (such as matrix multiplication operators and input / output) are partitioned at different memory (such as DRAM) levels, and the partitioning along different dimensions at each memory (such as DRAM) level, such as... Figure 4 As shown, operator-level IR can be segmented layer by layer from the Channel level to the Bank level through data slicing. Then, each sub-operator's data slice (i.e., sub-operator) can be mapped to the memory array (i.e., memory cell) through data mapping (i.e., address allocation). Specifically, in one possible implementation, step S13, based on the operator-level intermediate representation of the neural network model and the tree-like architecture abstract model of the near-memory computing architecture, constructs the search space for the compilation strategy, which may include:

[0060] Step S131: Based on the operator-level intermediate representation and the tree-structure abstract model, following the order from the highest to the lowest level of the memory in the near-memory computing architecture, under the condition of satisfying the first constraint, the operator-level intermediate representation is partitioned along multiple dimensions level by level to obtain multiple partitioning strategies; wherein, the first constraint includes that the product of the number of partitions along each dimension at each level is less than or equal to the number of hardware structures corresponding to each level in the memory, and different partitioning strategies have different partitioning methods under the first constraint.

[0061] Step S132: For each sub-intermediate representation divided by each partitioning strategy, under the condition of satisfying the second constraint, each sub-intermediate representation is divided into multiple data slices, and under the condition of satisfying the third constraint, each data slice is mapped to a storage unit in the near-memory computing architecture, thus obtaining multiple mapping strategies corresponding to each partitioning strategy; wherein, the second constraint includes: the product between the size of each data slice and the data bit width of each data slice is less than or equal to the product between the burst length of the memory and the column bit width of the memory; the third constraint includes: the number of data slices mapped to the same row in the memory is less than or equal to the ratio between the total number of columns in that row in the memory and the burst length; the burst length is the number of columns of the memory accessed consecutively by an access command.

[0062] For example, this disclosure describes the generation process of the above partitioning and mapping strategies by taking the compilation and deployment of matrix multiplication operators on a bank-level NDP architecture with parallelism of {#CH,#RA,#DE,#BA} as an example. Here, #CH, #RA, #DE, and #BA represent the number of channels, the number of ranks per channel, the number of devices per rank, and the number of banks per device, respectively. The partitioning strategy can then be expressed as:

[0063] Partitioning strategy = {Part CH (M CH ,K CH N CH Part RA (M RA ,K RA N RA ),

[0064] Part DE (M DE ,K DE N DE Part BK (M BK ,K BK N BK )};

[0065] The above partitioning strategy represents a four-level partitioning of four DRAM layers. The partitioning starts from the highest channel layer, and each layer is divided along three dimensions M, K, and N. Specifically, Part CH (M CH ,K CH N CH The operator level intermediate representation of a matrix multiplication operator is divided equally along the M, K, and N dimensions into M... CH K CH and N CHThis process yields multiple intermediate representations at the channel level. Each intermediate representation at the channel level will be assigned to a different channel, and should satisfy M... CH ×K CH ×N CH ≤Number of channels, to avoid allocating more channels than are available. Part RA (M RA ,K RA N RA The ) represents that each channel level is further divided into M equal parts along the M, K, and N dimensions. RA K RA and N RA This process divides the data into multiple rank-level intermediate representations. Each rank-level intermediate representation will be assigned to a different column (rank), and should satisfy M. RA ×K RA ×N RA ≤ The number of ranks in each channel, to avoid allocating more than the available ranks; Part DE (M DE ,K DE N DE The ) represents that each rank level is further divided equally along the M, K, and N dimensions into M. DE K DE and N DE This process divides the data into multiple device-level intermediate representations. Each device-level intermediate representation will be assigned to a different chip device, and should satisfy M... DE ×K DE ×N DE ≤ Number of devices in each rank to avoid allocating more than the available devices; Part BK (M BK ,K BK N BK The ) represents that each device level is further divided into M, K, and N equal dimensions. BK K BK and N BK This process divides the data into multiple bank-level intermediate representations. Each bank-level intermediate representation will be assigned to a different storage bank, and should satisfy M... BK ×K BK ×N BK The number of banks in each device should be less than or equal to the number of available banks to avoid allocating more than the number of available banks.

[0066] Therefore, from the channel level to the bank level, the operator-level intermediate representation of the matrix multiplication operator can be partitioned along multiple dimensions and allocated to multiple hierarchical structures in DRAM. Furthermore, the partitioning result under this strategy can be represented by a 12-tuple: {{M,N,K} {CH,RA,DE,BK} The first constraint condition may include the aforementioned M. CH ×K CH ×N CH ≤Number of channels, M RA ×K RA ×N RA ≤ The number of ranks in each channel, M DE ×K DE ×N DE ≤ Number of devices in each rank, M BK ×K BK ×N BK ≤ Number of banks in each device. It should be understood that, under the first constraint mentioned above, the number of partitions in each dimension of the operator-level intermediate representation of the matrix multiplication operator can be different at each level (i.e., the partitioning methods can be different). Based on the different number of partitions in each dimension of each level, it means that multiple partitioning results can be obtained for the same operator-level intermediate representation, that is, multiple partitioning strategies can be obtained; among them, the sub-intermediate representations partitioned by each partitioning strategy can be the intermediate representations partitioned by the smallest level, such as the bank-level intermediate representations mentioned above.

[0067] After partitioning the operator-level intermediate representation of the matrix multiplication operator, the next step is to encode the mapping of the partitioned sub-intermediate representations (equivalent to the weights, inputs, and outputs of the sub-matrix multiplication operators) to DRAM rows and columns to obtain the mapping strategy. In current DRAM, multiple columns can be accessed consecutively using a single read / write command. The number of columns accessed is called the burst length (BL), which is the number of columns accessed consecutively by a single access command. In this embodiment, the multiple data columns corresponding to the burst length are considered as the smallest data block read from DRAM, and this serves as the basis for data mapping. The mapping strategy can then be expressed as:

[0068]

[0069] Among them, Map {W,A,O} This represents the mapping strategy between the weight matrix, input activation matrix, and output matrix. (Using Map...) WFor example, the resulting sub-weight matrix W is first divided into multiple data slices (i.e., the bank-level intermediate representation is first divided into multiple data slices). Each data slice is mapped to a minimum data block in DRAM, and the size of each data slice is [size missing]. The second constraint condition may include: In other words, the size of each data switch should be less than or equal to the size of the smallest data block read from DRAM, so that each data slice can be stored in the space corresponding to the smallest data block; where, The size of the data slice (i.e., the product of the size of the data slice in the M dimension and the size in the K dimension), the matrix data bit width represents the data bit width for data switching, and the DRAM column bit width represents the column bit width of the storage cell in DRAM. The subweight matrix represents the subweight matrix in the M-dimensional dimension. A data slice, The subweight matrix represents the subweight matrix in the K-dimensional dimension. Data slices, and specify totals Each data slice will be mapped to the same row in DRAM, thus the third constraint can include: That is, the number of data slices mapped to the same row in the storage unit is less than or equal to the ratio between the total number of columns in each row of the storage unit and the burst length. Similarly, the size of each data slice in the multiple data switches into which each sub-input activation matrix A is divided can be expressed as: The second constraint condition may include: The sub-input activation matrix is ​​represented in the M-dimensional direction. A data slice, The sub-input activation matrix is ​​located in the K-dimensional space. Data slices, and If multiple data slices are mapped to the same row in DRAM, then the third constraint can include: The size of each data slice in the multiple data switching processes where each sub-output matrix is ​​divided is represented as follows: The second constraint condition may include: The sub-output matrix represents the M-dimensional... A data slice, The sub-output matrix represents the K-dimensional... Data slices, and If multiple data slices are mapped to the same row in DRAM, then the third constraint can include:

[0070]

[0071] It should be understood that for each sub-intermediate representation in the partitioning result of the matrix multiplication operator under any partitioning strategy, a corresponding mapping method can be obtained, and the mapping result of the operator-level intermediate representation of the matrix multiplication operator can be encoded using 12 tuples. Under the second constraint mentioned above, the size of the data slices divided by each sub-intermediate representation under each partitioning strategy can be different, and under the third constraint mentioned above, the number of data slices mapped to the same row can also be different. Therefore, there can be multiple mapping methods for the partitioning result under each partitioning strategy to the rows and columns in memory, and each mapping method can be encoded to obtain multiple mapping strategies corresponding to each partitioning strategy.

[0072] Optionally, to narrow the search space of the compilation strategy to some extent, some constraints can be added to limit the encoding range of the above mapping strategy. For example, for the matrix multiplication operator mentioned above, the second constraint can also include: the size of the data slice in the dimension (i.e., K dimensions) where the reduction calculation is required (i.e., the size K of the sub-weight matrix in K dimensions). B W The size K of the sub-input activation matrix in K dimensions B A The computational parallelism of the subweight matrix should be as close as possible to that of the computational unit (PU) to fully utilize the computational parallelism of the reduction modules (e.g., addition trees) within the PU and improve computational efficiency. Additionally, the number of data slices K-dimensionally for the subweight matrix should also be considered. R W The number of data slices K in the K-dimensional sum of the input activation matrix R A The values ​​are set to be the same to match the weights and the granularity of the input read from DRAM, thereby reducing the number of redundant reads. Of course, those skilled in the art can also set other necessary constraints to limit the encoding range of the mapping strategy according to actual needs, and this disclosure does not limit this.

[0073] It should be understood that multiple partitioning strategies can be generated based on the first constraint, and multiple mapping strategies under each partitioning strategy can be generated based on the second and third constraints. Thus, a search space for compilation strategies can be constructed. As mentioned above, this search space can also contain multiple selectable scheduling strategies so that the target compilation strategy determined subsequently can also include scheduling strategies.

[0074] In step S14, by performing performance prediction on the compilation strategies in the search space, multiple candidate compilation strategies and the compilation results corresponding to each candidate compilation strategy are determined. The compilation results include a complete instruction sequence obtained by compiling the operator-level intermediate representation according to the candidate compilation strategies.

[0075] In practical applications, a performance predictor can be used to predict the performance of various compilation strategies in the search space, obtaining the predicted performance of each strategy. The predicted performance characterizes the execution time required for the complete instruction sequence obtained by compiling the operator-level intermediate representation using the compilation strategy to complete on a near-memory computing architecture. Then, the predicted performance can be sorted, and the top K strategies (i.e., the K shortest execution times) can be selected as candidate compilation strategies. Simultaneously, the compilation results of each candidate strategy can be obtained. The performance predictor can employ known performance prediction algorithms in the art, or it can be a custom-designed algorithm. Alternatively, a machine learning model or deep learning model can be trained as the performance predictor; this disclosure does not limit the scope of the embodiments.

[0076] Considering the diversity of partitioning, mapping, and scheduling strategies, the search space for compilation strategies is enormous. This large search space imposes a significant computational burden on performance prediction. Therefore, as... Figure 5 As shown, the performance ceiling of the near-memory computing architecture required under each compilation strategy can be analyzed based on hardware data first, and the performance ceiling can be used as an indicator to dynamically reduce the search space (i.e., perform search space pruning). Then, a performance predictor can be used to predict the performance of the instruction fragments (i.e., partial instruction sequences) compiled by the compilation strategies in the pruned search space. Thus, in one possible implementation, in step S14, by performing performance prediction on the compilation strategies in the search space, multiple candidate compilation strategies and the compilation results corresponding to each candidate compilation strategy are determined, including:

[0077] Step S141: Based on the hardware data corresponding to the near-memory computing architecture, determine the performance upper limit corresponding to each compilation strategy in the search space. The performance upper limit represents the upper limit of the time required for the compilation result corresponding to the compilation strategy to be executed on the near-memory computing architecture. The hardware data includes the bandwidth of the memory, the line miss cost (i.e., the delay required for line break access when the access line address changes), the number of computing units and their performance (such as the utilization rate of computing units) in the near-memory computing architecture.

[0078] Step S142: Based on the performance upper limit of each compilation strategy in the search space, determine a number of initial candidate compilation strategies whose performance upper limit is higher than the preset performance threshold.

[0079] Step S143: The compiler performs partial compilation of the operator-level intermediate representation based on each initial candidate compilation strategy and a specific instruction set to obtain a partial instruction sequence compiled by each initial candidate compilation strategy. The specific instruction set covers instructions executed on various near-memory computing architectures.

[0080] Step S144: Use a performance predictor to predict the performance of the partial instruction sequences compiled by each initial candidate compilation strategy, and obtain the predicted performance of the partial instruction sequences compiled by each initial candidate compilation strategy, wherein the predicted performance characterizes the time required for the partial instruction sequences to be executed on a near-memory computing architecture.

[0081] Step S145: Based on the prediction performance of the partial instruction sequences compiled by each initial candidate compilation strategy and the number of loop executions corresponding to the partial instruction sequences, determine the target prediction performance corresponding to each initial candidate compilation strategy. The target prediction performance characterizes the time required for the complete instruction sequence compiled based on the initial candidate compilation strategy to be executed on a near-memory computing architecture.

[0082] Step S146: Select multiple candidate compilation strategies from multiple initial candidate compilation strategies based on the target prediction performance corresponding to each initial candidate compilation strategy;

[0083] Step S147: The compiler is used to fully compile the operator-level intermediate representation based on multiple candidate compilation strategies and a specific instruction set, so as to obtain the compilation results corresponding to each candidate compilation strategy.

[0084] In step S141, based on the hardware data corresponding to the near-memory computing architecture, the performance upper limit corresponding to each compilation strategy in the search space is determined, including: by analyzing each compilation strategy in the search space, obtaining the analysis results corresponding to each compilation strategy, the analysis results including the total amount of data accessed to the memory and the number of line misses, as well as the computational load of the computing unit; and determining the performance upper limit corresponding to each compilation strategy in the search space based on the analysis results corresponding to each compilation strategy and the hardware data.

[0085] It should be understood that different compilation strategies produce different compilation results. When different compilation results are executed on near-memory computing architectures, the amount of data accessed to memory and the number of line misses are different, as are the computational loads on the computing units executing the compilation results. Therefore, a mapping model can be pre-established between the compilation strategy and the total amount of data accessed, the number of line misses, and the computational load. The mapping model can characterize the mapping relationship between the compilation strategy and the total amount of data accessed, the number of line misses, and the computational load. Then, the partitioning strategy, mapping strategy, and scheduling strategy in the compilation strategy can be analyzed using the mapping model to obtain the total amount of data accessed, the number of line misses, and the computational load generated under each compilation strategy. This disclosure does not limit this aspect.

[0086] For example, for the NDP architecture, the following performance cap formula can be used to determine the performance cap Perf for each compilation strategy in the search space based on the analysis results and hardware data corresponding to each compilation strategy.max :

[0087]

[0088] The first term calculates the minimum time for the computation unit (PU) to access DRAM; the second term, using row misses and miss costs, calculates the total precharge time; and the third term estimates the ideal latency of the PU computation. Since there is some overlap between the minimum time for the PU to access DRAM calculated in the first term and the ideal latency of the PU computation calculated in the third term, optionally, the performance cap Perf... max It can also equal to This disclosure does not limit the scope of the embodiments.

[0089] It is understood that since the performance ceiling calculated using the above performance ceiling formula does not consider the latency caused by additional data movement, such as the latency caused by data movement between PUs, caches, and other PUs, if the performance ceiling corresponding to a certain compilation strategy is lower than the preset performance threshold, it means that the actual performance of the compilation result under that compilation strategy on the near-memory computing architecture is also lower than the preset performance threshold. Therefore, the compilation strategy is considered not to meet the minimum performance requirements, and further performance prediction for that compilation strategy can be skipped to narrow the search space, reduce the amount of computation required for subsequent performance predictors to perform performance prediction on compilation strategies in the search space, and improve computational efficiency. Therefore, in step S142, multiple initial candidate compilation strategies with performance ceilings higher than the preset performance threshold can be determined from the search space based on the preset performance threshold, and multiple candidate compilation strategies can be selected from the multiple initial candidate compilation strategies. It should be understood that those skilled in the art can customize the specific value of the above preset performance threshold, and this embodiment of the present disclosure does not limit this.

[0090] In step S143, those skilled in the art can design a compiler according to actual needs, or they can use existing compilers in the field, as long as they can compile the operator-level intermediate representation according to the compilation strategy. It is understood that the compiler can compile the operator-level intermediate representation into a sequence of instructions that can be executed on near-memory computing architectures. Therefore, it is necessary to provide the compiler with a corresponding instruction set. Existing instruction sets are mainly designed separately for different near-memory computing architectures and cannot be applied to different memory computing architectures. Therefore, this disclosure proposes a unified NDP instruction set (i.e., a specific instruction set) based on a tree-structured NDP abstract model, as shown in Table 1, to cover all instruction operations that may be used in various NDP architectures, meeting the generalization requirement of deploying any neural network model on any NDP architecture.

[0091] Table 1 Specific Instruction Sets

[0092]

[0093] As shown in Table 1, the specific instruction set mainly includes three types of instructions: computation instructions, data copy instructions, and host access instructions. LB (Local Buffer) represents the local cache, and GB (Global Buffer) represents the global cache. It should be noted that the instructions listed in Table 1 will be further broken down into a series of DRAM commands (such as the ACT command, READ command, WRITE command, etc.) by the memory controller at runtime.

[0094] The computation instructions allow the computation unit (PU) to perform computations using input operands A from memory (e.g., a bank of DRAM) and operands B from memory (e.g., another bank), a global cache, or the computation unit's local input cache. The MAC (Multiply-Accumulate) computation instruction is a vectorized instruction that uses a single instruction to access multiple data items in consecutive columns of a memory row, reducing command and address bus overhead. Furthermore, each MAC computation instruction requires specific hardware support. For example, bank-level MAC-DRAM requires a dedicated data bus between the computation unit and two different banks; MAC-LB requires the computation unit to have a local cache. Therefore, if certain hardware components (such as the absence of a global cache) are missing in the near-memory computation architecture described by the tree-like abstract model, the compiler will automatically exclude instructions related to the missing hardware components during compilation.

[0095] The data copy instructions are used to exchange data within the NDP architecture and mainly include three types of instructions: the first type copies data between the registers and local cache of the computing unit, which is usually used to store and update temporary calculation results; the second type copies data between the local cache and memory (such as DRAM), which reduces memory access by taking advantage of data locality; and the third type transfers data between the global cache and memory (such as DRAM), enabling the computing unit to obtain data from invisible addresses.

[0096] The host access instructions control data communication between the central processing unit (CPU) and the near-memory computing architecture. In addition to regular memory read / write instructions, the instruction set includes four additional classes to enable highly parallel communication between the host and all computing units within the near-memory computing architecture. It can be assumed that the host can write to the registers, local cache, and global cache of computing units in parallel, thereby reducing data transfer within the near-memory computing architecture and improving the efficiency of each computing unit in acquiring input data. For data reading, since the near-memory computing architecture only needs to return the final result to the host, instructions for transferring data from the result register to the host can be retained.

[0097] The aforementioned instruction set covers various data flows across different near-memory computing architectures. Here, we use UPMEM memory (a type of memory embedded in the processor core) and AiM memory (Accelerator in Memory, a technology that embeds computing units directly in memory) as examples to illustrate the changes in instructions across different architectures. In the UPMEM near-memory computing architecture, the WRITE-LB instruction is used to prepare the input matrix. However, in the AiM near-memory computing architecture, due to the lack of local caches in the computing units, this process is replaced by the WRITE-GB or WRITE-DRAM instructions. When the input data is ready, the MAC-LB, MAC-GB, or MAC-DRAM instructions are used to perform multiply-accumulate calculations. It's important to note that in the AiM near-memory computing architecture, the MAC-DRAM instruction reduces the number of activated computing units to half the number of memory banks in DRAM (i.e., two DRAM memory banks activate one computing unit), thus sacrificing half the parallelism compared to the MAC-GB instruction. When some intermediate calculation results are obtained, the REG2LB instruction can be used to store the intermediate calculation results from the computing unit's register to the output cache. After obtaining the complete calculation result, the LB2DRAM instruction can be used to write part of the calculation result back to DRAM for subsequent processing. In AiM without an output buffer, the LB2DRAM instruction can be replaced by READ-REG and WRITE-REG to return the complete calculation result to the host (such as the CPU).

[0098] As mentioned above, in addition to pruning the search space of the compilation strategy, a performance predictor is introduced to predict the performance of the compilation strategy. Although the search space of the compilation strategy has been pruned, the overhead of using periodic exact simulation to accurately predict the performance of each compilation strategy is still significant. Furthermore, while analysis based on the ideal performance upper limit is fast enough, its accuracy is insufficient due to the lack of evaluation of additional data movement latency. Therefore, a fast and accurate performance predictor is needed to predict more precise performance. However, without a compiled instruction sequence, the additional data movement latency is difficult to derive theoretically because it is affected by various factors, such as cache size, data flow, and possible instruction overlap during out-of-order execution. Therefore, a performance predictor can be embedded during the compiler's compilation of operator-level intermediate representations. After the compiler compiles a partial instruction sequence, the performance predictor can be used to predict the performance of this partial instruction sequence, and then extrapolate the performance of the complete instruction sequence. This approach can improve the efficiency of performance prediction.

[0099] Therefore, in step S144, the performance predictor is used to predict the performance of the partial instruction sequences compiled by each initial candidate compilation strategy, and the predicted performance of the partial instruction sequences compiled by each initial candidate compilation strategy is obtained, including:

[0100] Step S1441: Based on the partial instruction sequences compiled by each initial candidate compilation strategy, determine the total number of memory accesses in the near-memory computing architecture, the number of line misses in consecutive memory accesses, and the amount of data accessed by the computing unit to the cache unit or register in the near-memory computing architecture during the execution of the partial instruction sequences compiled by each initial candidate compilation strategy.

[0101] Step S1442: Based on the total number of memory accesses and column-to-column latency, the number of row misses and row miss costs of consecutive memory accesses in the near-memory computing architecture, and the amount of data and bus bandwidth of the computing units accessing cache units or registers in the near-memory computing architecture during the execution of the partial instruction sequences compiled by each initial candidate compilation strategy, determine the predicted performance of the partial instruction sequences compiled by each initial candidate compilation strategy.

[0102] Specifically, by analyzing the total number of memory read / write instructions in a subset of instruction sequences (e.g., the total number of DRAM read / write commands triggered), the total number of memory read / write operations (i.e., the total number of accesses) that may originate from compute instructions, data copy instructions, and / or host access instructions can be obtained. Furthermore, by analyzing the number of instructions in a subset of instruction sequences that trigger line feed access to memory, the number of line misses during continuous memory read / write operations can be determined. Additionally, by analyzing the number of instructions in a subset of instruction sequences that access the host cache or registers, the amount of data accessed by the host cache or registers can be determined. This, combined with the memory timing parameters such as tCDD (column-to-column latency), known line miss costs (e.g., precharge latency, activation latency, and row-to-row latency), and the bandwidth of the data transfer bus between the host and the near-memory computing architecture (which is known), can the predicted performance of the subset of instruction sequences compiled by each initial candidate compilation strategy can be predicted.

[0103] For example, the performance predictor can use the following performance prediction formula to determine the predicted performance of a subset of instruction sequences compiled by each initial candidate compilation strategy. partial :

[0104] Lat partial =t CCD ×(#READ+#WRITE)+#Row miss (READ)×t RowChange (READ)

[0105] +#Row miss (WRITE)×t RowChange (WRITE)+Lat Buffer / Reg (Host)

[0106] Among them, t CCD Representing column-to-column latency (tCDD), #READ and #WRITE represent the total number of DRAM reads and writes from compute instructions, data copy instructions, and host access instructions, respectively; #Row miss (READ) represents the number of line misses during a continuous read process, t RowChange (READ) represents the cost of a row miss during a sequential read process (i.e., the additional delay when a row is hit), #Row miss (WRITE) represents the number of line misses during a continuous write operation, t RowChange (WRITE) represents the cost of a row miss during a continuous write process; Lat Buffer / Reg (Host) takes into account the impact of host access commands on the data bus, Lat Buffer / Reg(Host) = Amount of data accessed by the host to the cache or registers / Bus bandwidth. Using the above performance prediction formula allows for the inclusion of data movement latency in the predicted performance, improving the accuracy of performance predictions for certain instruction sequences.

[0107] In step S145, the predicted performance of the complete instruction sequence (i.e., the total latency of executing the complete instruction sequence on a near-memory computing architecture) can be obtained based on the number of loop executions of the partial instruction sequences compiled by each initial candidate compilation strategy. For example, total latency = number of loop executions × Lat partial In other words, the predicted performance of a partial instruction sequence can be multiplied by the number of loop instructions in that partial instruction sequence to obtain the target predicted performance of the complete instruction sequence. In practical applications, those skilled in the art can use any known analysis method to estimate the number of loop executions of each compiled partial instruction sequence by analyzing the partitioning strategy, mapping strategy, and scheduling strategy in each initial candidate compilation strategy. This disclosure does not limit this.

[0108] In step S146, after obtaining the target prediction performance corresponding to each initial candidate compilation strategy, for example, the initial candidate compilation strategies can be sorted in descending order according to the target prediction performance (equivalent to sorting the initial candidate compilation strategies in ascending order according to the execution time of the instruction sequence; the shorter the execution time, the higher the target prediction performance), and the K initial candidate compilation strategies with the highest target prediction performance are selected as the K candidate compilation strategies; then, in step S147, the compiler can be used to fully compile the operator-level intermediate representation according to each candidate compilation strategy based on the above specific instruction set, to obtain the complete instruction sequence compiled by each candidate compilation strategy.

[0109] In step S15, based on the tree-structured abstract model, the execution process of the compilation results corresponding to each candidate compilation strategy on the near-memory computing architecture is simulated to obtain the simulation results corresponding to each candidate compilation strategy. The simulation results represent the total time required for the compilation results to be executed on the near-memory computing architecture.

[0110] It should be understood that although the performance predictor mentioned above can be used to predict the performance of a complete instruction sequence (i.e., the execution time of a complete instruction sequence on a near-memory computing architecture), the performance predicted by the performance predictor is still not accurate enough. Therefore, we can further simulate the execution process of the compilation results corresponding to each candidate compilation strategy on a near-memory computing architecture to obtain a more accurate total time required for the complete instruction sequence to be executed on a near-memory computing architecture. This will help to accurately determine the target compilation strategy that is best suited to the near-memory computing architecture.

[0111] In one possible implementation, a simulator based on a tree-structured abstract model can be used to simulate the execution process of the compilation results corresponding to each candidate compilation strategy in a near-memory computing architecture, obtaining the simulation results for each candidate compilation strategy. The simulator includes a hardware state space, an instruction queue, and a virtual memory controller (VMC). The hardware state space is used to simulate and track the hardware states of each hardware component in the near-memory computing architecture based on the tree-structured abstract model. These hardware components include storage units, computing units, cache units, and transmission buses (such as the bus for data transfer between computing units and cache units). The instruction queue stores the complete instruction sequence compiled based on the candidate compilation strategies and maintains the data dependencies within the complete instruction sequence (data dependencies describe the required execution order between instructions). The virtual memory controller determines the issuance time of instructions in the instruction queue and updates the hardware states of each hardware component in the hardware state space based on the hardware state space.

[0112] The hardware state space can be updated by memory commands and tracked and updated according to the memory's own timing constraint model. It should be understood that a known tree-like abstract model of the near-memory computing architecture can simulate the various hardware components within it. Therefore, the state of each hardware component can be tracked based on the memory's own timing constraint model. Specifically, tracking the hardware state of each component in the near-memory computing architecture includes: tracking the activation state of each memory cell in the near-memory computing architecture to issue an activation command before an inactive memory cell is accessed; and tracking the currently active row address in the memory cell to indicate whether subsequent accesses trigger a row miss and, if a row miss occurs, scheduling a precharge command and an activation command to activate the row to be accessed.

[0113] In this hardware state space, multiple clock countdowns can be used to track the hardware state of each hardware component. The hardware state of a memory unit includes the time when the next command can be issued for that memory unit (i.e., a clock countdown can be used to indicate the time when the next command (such as ACT, PRE, READ, and WRITE) can be issued for that memory unit). The hardware states of other hardware components besides memory units (i.e., computing units, cache units, transmission buses, etc.) include idle or occupied states (i.e., a clock countdown can be used to indicate the space state or occupied state of the hardware component). The idle state is also the unused state, and the occupied state is the state in use. The clock countdowns for each hardware component are updated according to the hardware occupied duration of the hardware component involved in the issued instruction. By establishing a hardware state space to track the hardware states of different hardware components, accurate simulation of the execution process of instruction sequences on near-memory computing architectures can be provided, and timing parameters in memory (such as DRAM) can be accurately modeled during the simulation process.

[0114] For example, for an NDP architecture, the activation status of each DRAM bank can be tracked. This is because DRAM read / write commands are not allowed when a bank is inactive; therefore, an ACT command should be issued before accessing an inactive row. Secondly, the address of the currently active DRAM row can be tracked to indicate whether subsequent memory accesses will trigger a row miss (i.e., a row address change). In the event of a row miss, a precharge command (PRE command) and a subsequent activation command (ACT command) are scheduled to activate another DRAM row. Furthermore, considering the timing constraints between different DRAM commands, four DRAM timing countdowns can be maintained to indicate when a bank can issue the next ACT, PRE, READ, and WRITE command. Simultaneously, clock countdowns are also used to track the idle and occupied states of individual processing units (PUs), transfer buses, and cache units.

[0115] In this embodiment, all clock countdowns in the hardware state space can be updated based on a global clock. When the virtual memory controller issues a new instruction, the clock countdowns of the hardware components involved in that instruction are reset and updated according to the DRAM command converted from the instruction or the hardware occupancy time required to execute the instruction, in order to maintain the hardware state of each hardware component in the hardware state space. Since the hardware occupancy time (i.e., the hardware execution time of the instruction) of the hardware components involved in the execution of different instructions is different, a mapping relationship between different instructions and hardware occupancy times can be pre-built so that the hardware occupancy time corresponding to each instruction can be obtained based on this mapping relationship during simulation, and then the clock countdown of the corresponding hardware component can be updated based on the hardware occupancy time; alternatively, since different instructions are converted into different DRAM commands, the clock countdown of the hardware components involved in the DRAM command converted from the currently issued instruction can be updated according to the hardware occupancy time of the hardware components involved in the DRAM command. This embodiment does not limit the scope of this disclosure.

[0116] It is known that data dependencies between instructions are the most noteworthy factor in instruction-driven simulation. Therefore, simulators should follow two basic rules when issuing and evaluating instruction sequences: (1) the operands of an instruction should be prepared before the instruction is issued; (2) the memory space of the data should not be reused before the instruction is completed. Existing memory-oriented DRAM simulators typically use memory barrier instructions to separate DRAM read / write commands to follow the data dependencies between instructions. However, barrier instruction design is less efficient for NDP architectures. This is because, on the one hand, in traditional memory, different DRAM devices in the same rank or different banks in the same device work together, but in NDP architectures, different computing units work independently, which leads to a significant increase in the number of instructions. Therefore, using memory barrier instructions in this case will impose a heavy burden on the size of the code file; on the other hand, in NDP architectures, data dependencies may only occur within the local scope of certain instructions. For example, in bank-level NDP, the data dependency between writing input data and computing using input data only exists between instructions in the same bank. However, using memory barrier instructions to force synchronization of all banks will lead to pipeline bubbles and inefficient utilization of processing units.

[0117] In view of this, the embodiments of this disclosure design an instruction queue to schedule the order of instructions, ensuring that instruction issuance and corresponding simulation behavior do not violate the data dependencies between instructions. The instruction queue includes multiple instruction groups. Independent instructions executed by different computing units are located in different instruction groups. Each instruction group contains multiple instructions to be executed sequentially. Instructions in instruction groups without data dependencies can be issued in parallel. Alternatively, the instruction queue consists of different instruction groups, each containing multiple instructions that need to be executed sequentially, such as "input-calculation-output" instructions. Instructions that are independent between different computing units are placed in different instruction groups; for example, in a bank-level NDP architecture, instructions for different banks can be placed in different instruction groups. Therefore, instructions from different instruction groups can be issued in parallel. Of course, if data communication and synchronization are required between two computing units or between a computing unit and other hardware components, data dependencies between different instruction groups can be configured. In other words, data dependencies can also be configured between instruction groups, allowing instructions in instruction groups without data dependencies to be issued in parallel. In each iteration of the simulation, the instruction queue first selects instruction groups without predecessors from the data dependency graph (which describes the data dependencies between instruction groups in the instruction queue). These instruction groups do not depend on executing a certain instruction group first, and are equivalent to instruction groups without data dependencies. The first instruction in each instruction group without a predecessor (i.e., the instruction at the beginning) is then sent to the virtual memory controller (VMC) to control the issuance of instructions based on the virtual memory controller.

[0118] The virtual memory controller supports both instruction reordering and out-of-order execution when issuing instructions. It can also schedule DRAM commands and coordinate other hardware components, such as processing units, cache units, and buses. The virtual memory controller primarily performs two functions: finding the earliest time a given instruction can be issued, and updating the clock countdowns of various hardware components in the hardware state space to reflect the impact of issuing the instruction on the hardware state of the relevant components.

[0119] Exemplary embodiments of this disclosure provide, as follows: Figure 6 The diagram shown illustrates a simulator architecture, as follows: Figure 6As shown, the instruction queue contains N instruction groups (inst.Group1, inst.Group2, ..., inst.GroupN) to store instruction sequences. The instruction queue checks the data dependencies of the instruction groups and sends the currently available instruction set {inst1.1, inst2.1, ..., instN.1} to the Virtual Memory Controller (VMC). The VMC can also have an instruction issue queue to store the instructions in the instruction set. The VMC can check the possible issue time of each instruction in the instruction set based on the hardware state space (i.e., calculate the instruction start time) and select the instruction with the earliest issue time as the next instruction to be issued. For example, if the issue times of three instructions in the instruction issue queue are Ts1, Ts2, and Ts3, and Ts2 < Ts1 and Ts2 < Ts3, then the instruction with the earliest issue time will be the next one to be issued. The instruction related to Ts is the instruction corresponding to Ts2. Since no other instructions can be issued before the earliest issuance time, the hardware state space can be directly updated to the hardware state at the earliest issuance time. Then, VMC deletes (i.e. issues) the instruction corresponding to Ts2 from the instruction queue and updates the hardware state of the hardware components occupied by the instruction corresponding to Ts2. The hardware state space can use a clock countdown to indicate the time when the next command can be issued for the DRAM Bank under the current instruction, and use a clock countdown to indicate the current hardware state of the Bus, the cache unit Buf, and the processing unit PU.

[0120] Based on the above simulator architecture, the above simulation of the execution process of the compilation results corresponding to each candidate compilation strategy in the near-memory computing architecture using the simulator based on the tree-structured abstract model can include:

[0121] Step S151: For the compilation result corresponding to any candidate compilation strategy, repeatedly execute the following simulation operations from S1511 to S1514:

[0122] Step S1511: The instruction queue determines the set of instructions that can be issued simultaneously based on the data dependencies involved in each instruction group, and sends the instruction set to the virtual memory controller. The instruction set includes the instruction that is currently the first instruction in each instruction group that has no data dependencies.

[0123] In step S1512, the virtual content controller determines the issuance time of each instruction in the instruction set based on the hardware state space, and selects the instruction with the earliest issuance time as the target instruction to be issued.

[0124] In step S1513, the virtual content controller updates the hardware state of each hardware component in the hardware state space to the hardware state corresponding to the issuance time of the target instruction based on the issuance time of the target instruction, and deletes the target instruction from the instruction queue.

[0125] Step S1514: The virtual content controller updates the hardware state of the hardware component occupied by the target instruction in the hardware state space according to the hardware occupancy duration of the hardware component occupied by the target instruction.

[0126] Step S152: When all instructions in the instruction queue have been issued and the hardware states of each hardware component in the hardware state space have become idle, the simulation results corresponding to the candidate compilation strategy are obtained.

[0127] In step S1511, if there is no data dependency between the instruction groups in the instruction queue, the instruction set consisting of the first instruction currently in the instruction group in the instruction queue can be sent to the virtual memory controller; if there is a data dependency between some instruction groups in the instruction queue, the instruction set consisting of the first instruction currently in the instruction group in the instruction queue that has no predecessor can be sent to the virtual memory controller.

[0128] In step S1512, the arrival time T of operands from different sources to the computation unit can be checked. A And the idle time T of the computing unit itself F (PU), determines the earliest time T that the computing unit can begin computing. S To obtain the issuance time T of each instruction in the instruction set. i Specifically, T can be... S The latest calculation start time is considered, and DRAM command scheduling is performed in ASAP (As Soon As Possible) mode, with the issuance time of the earliest issued DRAM command (such as PRE, ACT, READ, etc.) as the issuance time of the corresponding instruction.

[0129] Specifically, in step S1512 above, the virtual content controller determines the issuance time of each instruction in the instruction set based on the hardware state space, which may include:

[0130] Step S15121: For the i-th instruction in the instruction set, if the operand of the i-th instruction comes from the memory unit, based on the hardware state space, determine the activation state of the row in the memory unit where the operand of the i-th instruction is located, and determine the arrival time of the operand of the i-th instruction to the computing unit based on the activation state of the row where the operand of the i-th instruction is located, the clock countdown of the memory unit, and the read delay of reading the operand from the memory unit, where i is an integer greater than or equal to 1;

[0131] Step S15122: If the operand of the i-th instruction comes from the cache unit, determine the arrival time of the operand of the i-th instruction to the computing unit based on the release time of the cache unit, the read delay of reading the operand from the cache unit, and the idle time of the transmission bus between the cache unit and the computing unit.

[0132] Step S15123: Based on the arrival time of the operand of the i-th instruction to the computing unit and the idle time of the computing unit, determine the start processing time of the computing unit to start processing the operand of the i-th instruction.

[0133] Step S15124: Determine the issuance time of the i-th instruction based on the start processing time of the operand of the i-th instruction by the computing unit.

[0134] As described above, the computation instruction allows the computation unit to perform computations (such as matrix multiplication) on an input operand A from DRAM (e.g., a memory bank) and an operand B from DRAM (e.g., another memory bank), a global cache, or a local input cache of the computation unit. The operand can be understood as the data to be computed by the operator; for example, the operand of a matrix multiplication operator could be the input activation matrix. Therefore, for operands from the DRAM bank (i.e., the operands of the instruction come from memory cells), the activation state of the row containing the operand can be read to check whether an ACT command and / or a PRE command needs to be inserted before the access. If the activation state indicates that the row containing the operand is not active, the arrival time T of the operand at the computation unit can be obtained by adding the time indicated by the DRAM clock countdown (the time when the next command (e.g., ACT, PRE, etc.) can be issued), the read latency of reading the operand from the memory cell, and the latency caused by inserting the PRE and ACT commands. A (BK i If the active state indicates that the row containing the operand is already active, the time indicated by the DRAM clock countdown for issuing the next command (such as a READ command) and the read delay for reading the operand from the memory cell can be added together to obtain the arrival time T of the operand to the computing unit. A (BK i ).

[0135] For example, Figure 7a The diagram shown illustrates a DRAM access process, as follows: Figure 7aAs shown, Src1: Bank indicates that operand 1 comes from one bank of DRAM; Src2: Bank indicates that operand 2 comes from another bank of DRAM; [Idle] represents spatial time; [Row miss] represents a row miss that occurs when a processing unit attempts to access data that is not in the currently active row; PRE represents the Precharge command, RP represents the row precharge delay, and the time between the current time and the PRF command can be the time indicated by the bank's clock countdown to issue the next command; ACT represents the activation command, RCD represents the row to column delay, i.e., the time from row activation to the time when column data can be accessed, CCD represents the column to column delay, RD represents the read command, and RL represents the read delay of reading operands from memory. Latency refers to the time from issuing a read command to the operand reaching the processing unit. Proc represents the stage where the processing unit receives and processes the data, and Occu._proc represents the time occupied by the processing unit (Occupancy Processor), that is, the time the processing unit is currently processing two operands. For example... Figure 7a As shown, the arrival time T of operands Src1:Bank and Src2:Bank from the current time to the PRF command can be obtained by superimposing the time between the current time and the PRF command, the precharge delay RP, the row-to-column delay RCD, and the read delay RL. A (BK 0,1,… ).

[0136] For operands from cache units (such as the global cache), the time T for the operand to reach the computation unit is... A (GB) is composed of the release time of the cache unit and the read latency RL(GB) of reading operands from the cache unit, as well as the idle time T of the transmission bus between the computing unit and the cache unit. F (BUS) can be used to determine this; for example, a formula can be used: T A (GB) = max(T) F (GB)+RL(GB),T F (BUS) is used to determine the arrival time of operands from the cache unit to the compute unit. That is, the operand arrival time is the greater of the global cache release time plus the read latency and the bus idle time. Wherein, the global cache release time T... F(GB) typically refers to the time required before data in the global cache can be used by the next access request. Bus idle time can be understood as the time the bus can transmit data, or the time from the completion of the last transmission to the start of the next transmission; through max(T) F (GB)+RL(GB),T F The arrival time of operands from the global cache (BUS) is determined because data transmission is limited not only by the memory read speed but also by bus availability. If the bus is still busy after the global cache is released and ready for data, data transmission must wait for the bus to become idle. If the bus is idle during the time the global cache is released and ready for data, data can be transmitted immediately. If the bus is busy, data transmission must wait for the bus to become idle. Therefore, the greater of the global cache release time plus the read latency and the bus idle time (i.e., the time it takes for the bus to become idle) is used as the arrival time of operands from the global cache to reach the computation unit. It should be understood that the arrival time of operands from the local cache can be calculated in the same way as the arrival time of operands from the global cache, and will not be elaborated here.

[0137] For example, Figure 7b The diagram shows another DRAM access procedure, as shown below. Figure 7b As shown, Src1:Bank indicates that operand 1 comes from a DRAM bank; Src2:Buf indicates that operand 2 comes from the global cache; [Idle] also represents spatial time; [Row miss] also indicates a row miss; PRE represents a precharge command, RP represents row precharge latency, ACT represents an activation command, RCD represents row to column delay, RD represents a read command, RL represents read latency for reading operands from the bank, buf can represent read latency for reading operations from the global cache, Proc represents the stage where the processing unit receives and processes data, Occu.proc represents the time the processing unit occupies the processing unit (Occupancy Processor), that is, the processing unit is processing two operands. Com indicates the time when the bus between the processing unit and the cache unit starts transmitting data. Figure 7bIt is evident that the idle time of the transmission bus is greater than the sum of the release time of the global cache (i.e., the time from the current moment to sending the RD command to the global cache) and the read delay buf of reading operands from the global cache. Therefore, the arrival time from the current time to the arrival of Src1:Bank and Src2:Buf at the processing unit can be the spatial time of the transmission bus.

[0138] As described above, the operands of the i-th instruction may all come from the memory unit, or one may come from the memory unit and the cache unit. Therefore, in step S15123, the arrival time of the operands of the i-th instruction to the computing unit may include the arrival time of the operands from the memory unit, or the arrival time of the operands from the memory unit and the arrival time of the operands from the cache unit. Thus, the start time for the computing unit to begin processing the operands of the i-th instruction can be the arrival time of the operands of the i-th instruction to the computing unit plus the idle time T of the computing unit. F The maximum value in (PU), that is, the start processing time can be equal to T. S =max(T) A (GB),T A (BK 0,1,… ),T F (PU)), or, the start processing time can also be equal to T. S =max(T) A (BK 0,1,… ),T F (PU)); where the idle time of the computing unit is the time when the computing unit becomes idle (i.e., the moment).

[0139] In step S15124, the start processing time T of the operand of the i-th instruction can be set. S The latest computation start time is considered, and DRAM command scheduling is performed according to ASAP based on this latest computation start time. The issuance time of the earliest issued DRAM command (such as PRE, ACT, READ, etc.) is taken as the issuance time of the i-th instruction. This is equivalent to calculating the issuance time of the earliest DRAM command based on the start time of the computation unit's processing of the operands of the i-th instruction. Then, the instruction with the earliest issuance time can be selected as the target instruction to be issued, or the instruction with the earliest start time can be selected as the target instruction to be issued.

[0140] In step S1513, after determining the target instruction to be issued, the virtual memory space controller can, based on the issuance time of the target instruction, send an update request to the hardware state space to directly update the hardware state of each hardware component in the hardware state space to the hardware state corresponding to the issuance time of the target instruction. This is equivalent to directly adjusting the clock countdown of each hardware component in the hardware state space to the issuance time of the target instruction. This allows the hardware state space in the simulator to operate in an instruction-driven update mode, meaning that the hardware state space is not updated periodically, but rather updated when a new instruction is issued. Therefore, the number of simulation iterations is reduced from the number of DRAM cycles required to compute the instruction sequence to the length of the instruction sequence. This enables the simulator to achieve a 10 to 300-fold reduction in simulation time on different near-memory computing architectures, which is beneficial for improving the efficiency of the compilation process.

[0141] In this process, the virtual memory controller deleting the target instruction from the instruction queue signifies that the target instruction has been issued. Then, in step S1514, after the target instruction has been issued, the virtual memory controller can update the hardware state of the hardware component occupied by the target instruction in the hardware state space based on the hardware occupancy duration of the hardware component occupied by the target instruction, essentially simulating the execution process of the target instruction. Specifically, the occupancy duration of the DRAM occupied by the target instruction can be determined based on the issuance time of the target instruction, the DRAM's automatic precharge strategy, and the number of consecutive DRAM columns read by the instruction. Then, the DRAM clock countdown can be updated based on the DRAM occupancy duration, i.e., updating the hardware state indicated by the clock countdown. The clock countdowns (i.e., hardware states) of other hardware components (such as computing units, cache units, and transmission buses) will also be updated according to their respective occupied durations.

[0142] It should be understood that the above steps S1511 to S1514 can be executed cyclically until all instructions in the instruction queue are issued (i.e., the instruction queue is empty) and the hardware state of each hardware component in the hardware state space becomes idle (i.e., the hardware component has executed all instructions in the instruction queue). In this case, the delay obtained from the start of the simulation to the end of the simulation is the total time required for the complete instruction sequence to be executed on the near-memory computing architecture.

[0143] In practical applications, the simulator can also be extended to support workloads of non-neural network models (i.e., operator-level intermediate representations) by modifying the performance model of the computing unit (e.g., using other existing simulators in the art, such as Gem5, to simulate near-memory CPUs or other general-purpose computing units), adding corresponding IRs, and reusing memory access modeling. This disclosure does not limit the scope of the embodiments.

[0144] In step S16, based on the simulation results corresponding to each candidate compilation strategy, the target compilation strategy is determined from multiple candidate compilation strategies, and the compilation result of the target compilation strategy is determined as the target compilation result of the operator-level intermediate representation of the neural network model, so as to deploy the neural network model on the near-memory computing architecture based on the target compilation result.

[0145] In practical applications, the candidate compilation strategy with the best simulation results (i.e., the shortest total time) can be selected from multiple candidate compilation strategies as the target compilation strategy. Alternatively, the candidate compilation strategy with the second-best simulation results can also be selected, and this disclosure does not impose any limitations on this. Furthermore, after obtaining the target compilation result of the operator-level intermediate representation of the neural network model, the neural network model can be deployed on a near-memory computing architecture based on the target compilation result. This disclosure does not impose any limitations on the deployment process of the neural network model on a near-memory computing architecture.

[0146] The compilation method according to the embodiments of this disclosure utilizes a tree-structured abstract model to describe the near-memory computing architecture. This allows for the sequential and unified description of any near-memory computing architecture using a tree structure. Furthermore, by combining the operator-level intermediate representation of the neural network model, a compilation strategy search space is constructed. Multiple candidate compilation strategies are then determined through performance prediction of the compilation strategies in the search space. Finally, a simulator is used to simulate the execution process of the compilation results on the near-memory computing architecture based on the abstract model. The simulation results obtained correspond to the precise simulation results of each candidate compilation strategy, thus determining the final target compilation strategy and target compilation result. This method provides a superior target compilation strategy for deploying any neural network model on any near-memory computing architecture, thereby enabling the deployment of any neural network model on any near-memory computing architecture and exhibiting high versatility.

[0147] Based on the compilation method provided in the above embodiments of this disclosure, the embodiments of this disclosure also provide Figure 8 The diagram shows a framework of a compilation method, such as... Figure 8As shown, the input consists of the neural network model to be compiled and deployed, and the hardware configuration information of the near-memory computing architecture (including the architecture configuration and DRAM timing parameters). First, the neural network model is analyzed using the ONNX framework to obtain a series of operator-level instruction recursions (IRs) transformed by the ONNX framework. Then, a tree-structured abstract model of the near-memory computing architecture is generated based on a general near-memory architecture abstraction method (i.e., a tree-structured abstract model). Next, a search space for compilation strategies is created for the operator-level IRs based on the operator-level IRs and the tree-structured abstract model. The compilation strategies include operator partitioning strategies, mapping strategies, and scheduling strategies. A hardware data-driven design space pruning strategy can be used to reduce the search space size. Hardware data includes, for example, the number of rows and columns read from DRAM and the number of DRAM row precharges. To further improve search efficiency, a performance predictor based on DRAM timing parameters is designed to find candidate compilation strategies corresponding to instruction files (i.e., files formed by partial instruction sequences) with the top K predicted performance. Next, the compiler can generate K different compilation results based on these K candidate compilation strategies, and a simulator can be invoked to evaluate the accuracy of these compilation results. The simulator's input is the compilation result (i.e., the complete instruction sequence), and its output, the simulation result, represents the total time required for the complete instruction sequence to execute. Finally, based on the simulation results, the optimal compilation strategy (i.e., the target compilation strategy) and the compilation result can be output.

[0148] It should be understood that the compilation method proposed in this disclosure uses a near-memory computing architecture configuration based on the real world. These configurations can also be flexibly changed to adapt to other hardware architectures. Furthermore, the compilation strategy for the search space proposed in this disclosure employs the ideas of pruning and hierarchical simulation. Some more efficient search algorithms, such as simulated annealing or genetic algorithms, can also replace the search method in this disclosure to provide higher search efficiency.

[0149] According to the compilation method of this disclosure, the near-memory architecture abstraction method can support various types and multi-level near-memory computing architectures, thereby enabling the simulator and compiler to adapt to different types of near-memory computing architectures; the proposed simulator uses instruction-driven updates to the hardware state space, which is about 1.7 times faster than existing simulators, while ensuring that the simulation error is within 10%.

[0150] Furthermore, by experimentally verifying the compilation method of the present disclosure embodiments on various near-memory computing architectures for single operators and complete models (such as convolutional neural network models and large language models), compared with existing compilation methods, the compilation method of the present disclosure embodiments can achieve an acceleration of 1.09 to 1.56 times on single operators and an acceleration of 1.05 to 3.43 times on complete models.

[0151] This paper analyzes the differences in compilation optimization strategies caused by the variations among near-memory computing architectures, and proposes a unified architectural abstraction and a general compilation method based on this abstraction. Furthermore, it delves into the operator partitioning and mapping strategy space in near-memory computing architectures and designs a series of effective methods for rapid strategy space exploration. As a result, a compilation method applicable to any near-memory computing architecture is proposed, and a general compilation and deployment toolchain for near-memory computing architectures is implemented, demonstrating high versatility.

[0152] Figure 9 A block diagram of a compilation system according to an embodiment of the present disclosure is shown, such as Figure 9 As shown, the device includes:

[0153] The acquisition module 901 is used to acquire the operator-level intermediate representation of the neural network model to be deployed to the near-memory computing architecture and the hardware configuration information of the near-memory computing architecture, wherein the hardware configuration information is used to indicate the hardware configuration of the near-memory computing architecture.

[0154] Abstract module 902 is used to generate a tree-like architecture abstract model of the near-memory computing architecture based on the hardware configuration information of the near-memory computing architecture. The tree-like architecture abstract model is used to describe the near-memory computing architecture using a tree structure.

[0155] The construction module 903 is used to construct a search space for a compilation strategy based on the operator-level intermediate representation of the neural network model and the tree-like architecture abstract model of the near-memory computing architecture. The compilation strategy is used to compile the operator-level intermediate representation.

[0156] The prediction module 904 is used to determine multiple candidate compilation strategies and the compilation results corresponding to each candidate compilation strategy by performing performance prediction on the compilation strategies in the search space. The compilation results include a complete instruction sequence obtained by compiling the operator-level intermediate representation according to the candidate compilation strategies.

[0157] The simulation module 905 is used to simulate the execution process of the compilation results corresponding to each candidate compilation strategy on the near-memory computing architecture based on the tree-structured abstract model, and to obtain the simulation results corresponding to each candidate compilation strategy. The simulation results represent the total time required for the compilation results to be executed on the near-memory computing architecture.

[0158] The determination module 906 is used to determine the target compilation strategy from the multiple candidate compilation strategies based on the simulation results corresponding to each candidate compilation strategy, and to determine the compilation result of the target compilation strategy as the target compilation result of the operator-level intermediate representation of the neural network model, so as to deploy the neural network model on the near-memory computing architecture based on the target compilation result.

[0159] In one possible implementation, the near-memory computing architecture includes a multi-layered hardware structure. Each layer of the hardware structure in the near-memory computing architecture corresponds to a node in the tree-like architecture abstract model. Each layer in the tree-like architecture abstract model contains at least one central node, and each central node connects to at least one of the following child nodes: the central node of the next lower layer, storage nodes, compute nodes, and cache nodes in the layer to which the central node belongs. The central node represents the data transmission relationship between two adjacent layers. The storage node represents a storage unit in the near-memory computing architecture and contains attribute information of the storage unit; the compute node represents a compute unit in the near-memory computing architecture and contains attribute information of the compute unit; and the cache node represents a cache unit in the near-memory computing architecture and contains attribute information of the cache unit.

[0160] In one possible implementation, a single compilation strategy includes a partitioning strategy and a corresponding mapping strategy. The partitioning strategy represents the partitioning result of dividing the operator-level intermediate representation along multiple dimensions at multiple levels. The mapping strategy represents the way to map the partitioning result of the operator-level intermediate representation to the memory in the near-memory computing architecture. The partitioning result represents multiple sub-intermediate representations divided by the operator-level intermediate representation. The construction of the search space for the compilation strategy based on the operator-level intermediate representation of the neural network model and the tree-like architecture abstract model of the near-memory computing architecture includes: based on the operator-level intermediate representation and the tree-like architecture abstract model, partitioning the operator-level intermediate representation along multiple dimensions level by level, according to the order from the highest to the lowest level of the memory in the near-memory computing architecture, under the condition of satisfying a first constraint, to obtain multiple partitioning strategies. The first constraint includes that the product of the number of partitions along multiple dimensions at each level is less than or equal to the number of hardware structures corresponding to each level in the memory. Different partitioning strategies have different partitioning methods under the first constraint. For each sub-intermediate representation partitioned by each partitioning strategy, each sub-intermediate representation is divided into multiple data slices under the second constraint, and each data slice is mapped to the memory of the near-memory computing architecture under the third constraint, resulting in multiple mapping strategies corresponding to each partitioning strategy. The second constraint includes that the product between the size of each data slice and the data bit width of each data slice is less than or equal to the product between the burst length of the memory and the column bit width of the memory. The third constraint includes that the number of data slices mapped to the same row in the memory is less than or equal to the ratio between the total number of columns in that row in the memory and the burst length.

[0161] In one possible implementation, the step of determining multiple candidate compilation strategies and their corresponding compilation results by performing performance prediction on the compilation strategies in the search space includes: determining the performance upper limit of each compilation strategy in the search space based on the hardware data corresponding to the near-memory computing architecture, wherein the performance upper limit represents the upper limit of the time required for the compilation result of the compilation strategy to be executed on the near-memory computing architecture, and the hardware data includes the memory bandwidth, line miss cost, number and performance of computing units in the near-memory computing architecture; determining multiple initial candidate compilation strategies whose performance upper limits are higher than a preset performance threshold based on the performance upper limits of each compilation strategy in the search space; and using a compiler to partially compile the operator-level intermediate representation based on the initial candidate compilation strategies and a specific instruction set to obtain a partial instruction sequence compiled by each initial candidate compilation strategy, wherein the specific instruction set covers the instructions executed on various near-memory computing architectures. The performance predictor is used to predict the performance of partial instruction sequences compiled by each initial candidate compilation strategy, obtaining the predicted performance of the partial instruction sequences compiled by each initial candidate compilation strategy. The predicted performance characterizes the time required for the partial instruction sequences to complete execution on the near-memory computing architecture. Based on the predicted performance of the partial instruction sequences compiled by each initial candidate compilation strategy and the number of loop executions corresponding to the partial instruction sequences, the target predicted performance corresponding to each initial candidate compilation strategy is determined. The target predicted performance characterizes the time required for the complete instruction sequences compiled based on the initial candidate compilation strategies to complete execution on the near-memory computing architecture. Based on the target predicted performance of each initial candidate compilation strategy, multiple candidate compilation strategies are selected from the multiple initial candidate compilation strategies. The compiler is then used to fully compile the operator-level intermediate representation based on the multiple candidate compilation strategies and the specific instruction set, obtaining the compilation results corresponding to each candidate compilation strategy.

[0162] In one possible implementation, determining the performance upper limit of each compilation strategy in the search space based on the hardware data corresponding to the near-memory computing architecture includes: analyzing each compilation strategy in the search space to obtain the analysis results corresponding to each compilation strategy, the analysis results including the total amount of data accessed to memory and the number of line misses under each compilation strategy, as well as the computational load of the computing unit; and determining the performance upper limit of each compilation strategy in the search space based on the analysis results corresponding to each compilation strategy and the hardware data.

[0163] In one possible implementation, the step of using a performance predictor to predict the performance of partial instruction sequences compiled by each initial candidate compilation strategy, and obtaining the predicted performance of the partial instruction sequences compiled by each initial candidate compilation strategy, includes: determining, based on the partial instruction sequences compiled by each initial candidate compilation strategy, the total number of memory accesses in the near-memory computing architecture, the number of row misses in consecutive memory accesses, and the amount of data accessed by computing units to cache units or registers in the near-memory computing architecture; and determining the predicted performance of the partial instruction sequences compiled by each initial candidate compilation strategy based on the total number of memory accesses and column-to-column latency, the number of row misses and row miss costs in consecutive memory accesses, and the amount of data accessed by computing units to cache units or registers and bus bandwidth in the near-memory computing architecture.

[0164] In one possible implementation, the simulation module utilizes a simulator based on the tree-structured abstract model to simulate the execution process of the near-memory computing architecture for the compilation results corresponding to each candidate compilation strategy, thereby obtaining the simulation results corresponding to each candidate compilation strategy. The simulator includes a hardware state space, an instruction queue, and a virtual memory controller. The hardware state space is used to simulate and track the hardware state of each hardware component in the near-memory computing architecture based on the tree-structured abstract model. The hardware components include storage units, computing units, cache units, and transmission buses. The instruction queue stores the complete instruction sequence compiled based on the candidate compilation strategies and maintains the data dependencies within the complete instruction sequence. The instruction queue includes multiple instruction groups, with independent instructions executed by different computing units located in different instruction groups. Each instruction group includes multiple sequentially executed instructions, and instructions in instruction groups without data dependencies can be issued in parallel. The virtual memory controller determines the issuance time of instructions in the instruction queue and controls the update time of the hardware state of each hardware component in the hardware state space based on the hardware state space.

[0165] In one possible implementation, tracking the hardware state of each hardware component in the near-memory computing architecture includes: tracking the activation state of each memory cell in the near-memory computing architecture to issue an activation command before an inactive memory cell is accessed; tracking the currently active row address in the memory cell to indicate whether subsequent accesses trigger a row miss and, if a row miss is triggered, scheduling a precharge command and an activation command to activate the row to be accessed; wherein, multiple clock countdowns are used in the hardware state space to track the hardware state of each hardware component, wherein the hardware state of the memory cell includes the time when the next command can be issued for the memory cell, and the hardware state of other hardware components besides the memory cell includes an idle state or an occupied state; the clock countdown corresponding to each hardware component is updated according to the hardware occupancy duration of the hardware component involved in issuing the command.

[0166] In one possible implementation, the step of using a simulator based on the tree-structured abstract model to simulate the execution process of the near-memory computing architecture for the compilation results corresponding to each candidate compilation strategy, and obtaining the simulation results corresponding to each candidate compilation strategy, includes: for the compilation result corresponding to any candidate compilation strategy, repeatedly executing the following simulation operations: the instruction queue determines the set of instructions that can be issued simultaneously based on the data dependencies involved in each instruction group, and sends the instruction set to the virtual memory controller, the instruction set including the instruction currently ranked first in each instruction group without data dependencies; the virtual memory controller determines each instruction in the instruction set according to the hardware state space. The virtual content controller determines the target instruction to be issued by selecting the instruction with the earliest issuance time. Based on the issuance time of the target instruction, the virtual content controller updates the hardware state of each hardware component in the hardware state space to the hardware state corresponding to the issuance time of the target instruction, and deletes the target instruction from the instruction queue. The virtual content controller updates the hardware state of the hardware component occupied by the target instruction in the hardware state space according to the hardware occupancy time of the hardware component occupied by the target instruction. When all instructions in the instruction queue have been issued and the hardware state of each hardware component in the hardware state space becomes idle, the simulation result corresponding to the candidate compilation strategy is obtained.

[0167] In one possible implementation, determining the issuance time of each instruction in the instruction set based on the hardware state space includes: for the i-th instruction in the instruction set, if the operand of the i-th instruction comes from a memory unit, determining the activation state of the row in the memory unit containing the operand of the i-th instruction based on the hardware state space, and determining the arrival time of the operand of the i-th instruction to the computing unit based on the activation state of the row containing the operand of the i-th instruction, the clock countdown of the memory unit, and the read delay of reading the operand from the memory unit; and for the i-th instruction... When the operands come from a cache unit, the arrival time of the operands of the i-th instruction to the computing unit is determined based on the release time of the cache unit, the read latency of reading the operands from the cache unit, and the idle time of the transmission bus between the cache unit and the computing unit. Based on the arrival time of the operands of the i-th instruction to the computing unit and the idle time of the computing unit, the start processing time of the computing unit for processing the operands of the i-th instruction is determined. Based on the start processing time of the computing unit for processing the operands of the i-th instruction, the issuance time of the i-th instruction is determined.

[0168] In one possible implementation, selecting the instruction with the earliest issuance time as the target instruction to be issued now includes: selecting the instruction with the earliest start time as the target instruction to be issued now.

[0169] The compilation system according to embodiments of this disclosure utilizes a tree-structured abstract model to describe near-memory computing architectures. This allows for the sequential and unified description of any near-memory computing architecture using a tree structure. Furthermore, by combining the operator-level intermediate representations of neural network models, a compilation strategy search space is constructed. Performance prediction of the compilation strategies in the search space determines multiple candidate compilation strategies. Then, a simulator simulates the execution process of the compilation results on the near-memory computing architecture based on the abstract model, obtaining precise simulation results corresponding to each candidate compilation strategy. This determines the final target compilation strategy and target compilation result. This system provides a superior target compilation strategy for deploying any neural network model on any near-memory computing architecture, thereby enabling the deployment of any neural network model on any near-memory computing architecture and exhibiting high versatility.

[0170] In some embodiments, the functions or modules of the apparatus provided in this disclosure can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0171] This disclosure also provides an electronic device, including a memory employing a near-memory computing architecture, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the above method.

[0172] This disclosure also provides a non-volatile computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the above-described method.

[0173] This disclosure also provides a computer program product, including a computer program or a non-volatile computer-readable storage medium carrying the computer program, wherein the computer program, when executed by a processor, implements the steps of the above method.

[0174] Figure 10 A block diagram of an electronic device 1900 according to an embodiment of the present disclosure is shown. For example, the electronic device 1900 may be provided as a server or a terminal device. (Refer to...) Figure 10 The electronic device 1900 includes a processing component 1922, which further includes one or more processors, and memory resources represented by a storage unit 1932 for storing instructions, such as application programs, that can be executed by the processing component 1922. The storage unit 1932 and the processing component 1922 may employ a near-memory computing architecture, or they may be separate memory and processors; this is not limited in the embodiments disclosed herein. The application programs stored in the storage unit 1932 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processing component 1922 is configured to execute instructions to perform the compilation method described above.

[0175] Electronic device 1900 may also include a power supply component 1926 configured to perform power management of electronic device 1900, a wired or wireless network interface 1950 configured to connect electronic device 1900 to a network, and an input / output interface 1958 (I / O interface). Electronic device 1900 can operate on an operating system, such as Windows Server, stored in storage unit 1932. TM Mac OS X TM Unix TM Linux TM FreeBSD TM Or similar.

[0176] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a storage unit 1932 including computer program instructions, which can be executed by a processing component 1922 of an electronic device 1900 to perform the above-described method.

[0177] Computer-readable storage media can be tangible devices capable of holding and storing programs / instructions used by instruction execution devices. Computer-readable storage media can be, for example—but not limited to—electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, dynamic random access memory (DRAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination of the foregoing. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0178] The computer program (or computer-readable program instructions) described herein can be downloaded from a computer-readable storage medium to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage medium in the respective computing / processing device.

[0179] The computer program (or computer program instructions) used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuits, such as programmable logic circuits, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), are personalized by utilizing state information of computer-readable program instructions. These electronic circuits can execute computer-readable program instructions to implement various aspects of this disclosure.

[0180] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0181] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0182] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0183] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0184] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or technical improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A compilation method, characterized in that, include: Obtain the operator-level intermediate representation of the neural network model to be deployed to the near-memory computing architecture and the hardware configuration information of the near-memory computing architecture, wherein the hardware configuration information is used to indicate the hardware configuration of the near-memory computing architecture; Based on the hardware configuration information of the near-memory computing architecture, a tree-like architecture abstract model of the near-memory computing architecture is generated. The tree-like architecture abstract model is used to describe the near-memory computing architecture using a tree structure. Based on the operator-level intermediate representation of the neural network model and the tree-structured abstract model of the near-memory computing architecture, a search space for a compilation strategy is constructed, and the compilation strategy is used to compile the operator-level intermediate representation. By performing performance prediction on the compilation strategies in the search space, multiple candidate compilation strategies and the compilation results corresponding to each candidate compilation strategy are determined. The compilation results include a complete instruction sequence obtained by compiling the operator-level intermediate representation according to the candidate compilation strategies. Based on the tree-structured abstract model, the execution process of the compilation results corresponding to each candidate compilation strategy on the near-memory computing architecture is simulated to obtain the simulation results corresponding to each candidate compilation strategy. The simulation results represent the total time required for the compilation results to be executed on the near-memory computing architecture. Based on the simulation results corresponding to each candidate compilation strategy, a target compilation strategy is determined from the multiple candidate compilation strategies, and the compilation result of the target compilation strategy is determined as the target compilation result of the operator-level intermediate representation of the neural network model, so as to deploy the neural network model on the near-memory computing architecture based on the target compilation result; The construction of the search space for the compilation strategy based on the operator-level intermediate representation of the neural network model and the tree-like abstract model of the near-memory computing architecture includes: Based on the operator-level intermediate representation and the tree-structured abstract model, following the order from the highest to the lowest level of the memory in the near-memory computing architecture, and under the condition of satisfying the first constraint, the operator-level intermediate representation is partitioned along multiple dimensions level by level to obtain multiple partitioning strategies; wherein, the first constraint includes that the product of the number of partitions along each dimension at each level is less than or equal to the number of hardware structures corresponding to each level in the memory, and different partitioning strategies have different partitioning methods under the first constraint. For each sub-intermediate representation resulting from each partitioning strategy, each sub-intermediate representation is divided into multiple data slices under the condition of satisfying the second constraint, and each data slice is mapped to the memory of the near-memory computing architecture under the condition of satisfying the third constraint, thus obtaining multiple mapping strategies corresponding to each partitioning strategy; wherein, the second constraint includes: the product between the size of each data slice and the data bit width of each data slice is less than or equal to the product between the burst length of the memory and the column bit width of the memory; the third constraint includes: the number of data slices mapped to the same row in the memory is less than or equal to the ratio between the total number of columns in that row in the memory and the burst length.

2. The method according to claim 1, characterized in that, The near-memory computing architecture includes a multi-layered hardware structure. Each layer of the hardware structure in the near-memory computing architecture corresponds to a node in the tree-like architecture abstract model. Each layer in the tree-like architecture abstract model contains at least one central node. Each central node connects to at least one of the following child nodes: the central node of the next level, the storage node, the computing node, and the cache node in the layer to which the central node belongs. The central node represents the data transmission relationship between two adjacent layers. Wherein, the storage node represents a storage unit of the memory in the near-memory computing architecture and contains the attribute information of the storage unit; the computing node represents a computing unit in the near-memory computing architecture and contains the attribute information of the computing unit; and the cache node represents a cache unit in the near-memory computing architecture and contains the attribute information of the cache unit.

3. The method according to claim 1, characterized in that, The step of determining multiple candidate compilation strategies and their corresponding compilation results by performing performance prediction on compilation strategies in the search space includes: Based on the hardware data corresponding to the near-memory computing architecture, the performance upper limit corresponding to each compilation strategy in the search space is determined. The performance upper limit represents the upper limit of the time required for the compilation result corresponding to the compilation strategy to be executed on the near-memory computing architecture. The hardware data includes the bandwidth of the memory, the line miss cost, the number of computing units and the performance in the near-memory computing architecture. Based on the performance upper limit of each compilation strategy in the search space, multiple initial candidate compilation strategies with performance upper limits higher than a preset performance threshold are determined. The compiler performs partial compilation of the operator-level intermediate representation based on the initial candidate compilation strategies and a specific instruction set, to obtain a partial instruction sequence compiled by each initial candidate compilation strategy. The specific instruction set covers instructions executed on various near-memory computing architectures. The performance of a partial instruction sequence compiled by each initial candidate compilation strategy is predicted using a performance predictor, and the predicted performance of the partial instruction sequence compiled by each initial candidate compilation strategy is obtained. The predicted performance characterizes the time required for the partial instruction sequence to be executed on the near-memory computing architecture. Based on the predicted performance of partial instruction sequences compiled by each initial candidate compilation strategy and the number of loop executions corresponding to the partial instruction sequences, the target predicted performance corresponding to each initial candidate compilation strategy is determined. The target predicted performance characterizes the time required for the complete instruction sequence compiled based on the initial candidate compilation strategy to be executed on the near-memory computing architecture. Based on the target prediction performance corresponding to each initial candidate compilation strategy, select multiple candidate compilation strategies from the multiple initial candidate compilation strategies; The compiler is used to fully compile the operator-level intermediate representation based on the multiple candidate compilation strategies and the specific instruction set, so as to obtain the compilation results corresponding to each candidate compilation strategy.

4. The method according to claim 3, characterized in that, The step of determining the performance ceiling for each compilation strategy in the search space based on the hardware data corresponding to the near-memory computing architecture includes: By analyzing each compilation strategy in the search space, the analysis results corresponding to each compilation strategy are obtained. The analysis results include the total amount of data accessed to the memory and the number of line misses, as well as the computational load of the computing unit. Based on the analysis results corresponding to each compilation strategy and the hardware data, the performance upper limit corresponding to each compilation strategy in the search space is determined.

5. The method according to claim 3, characterized in that, The process of using a performance predictor to predict the performance of partial instruction sequences compiled by each initial candidate compilation strategy, and obtaining the predicted performance of partial instruction sequences compiled by each initial candidate compilation strategy, includes: Based on the partial instruction sequences compiled by each initial candidate compilation strategy, determine the total number of memory accesses in the near-memory computing architecture, the number of consecutive line misses in memory accesses, and the amount of data accessed by computing units in the near-memory computing architecture to cache units or registers in the partial instruction sequences compiled by each initial candidate compilation strategy. Based on the total number of memory accesses and column-to-column latency, the number of row misses and row miss costs for consecutive memory accesses in the partial instruction sequences compiled by each initial candidate compilation strategy, as well as the amount of data accessed by the computing units to cache units or registers and the bus bandwidth in the near-memory computing architecture, the predicted performance of the partial instruction sequences compiled by each initial candidate compilation strategy is determined.

6. The method according to claim 1, characterized in that, The method utilizes a simulator based on the tree-structured abstract model to simulate the compilation results corresponding to each candidate compilation strategy during the execution process of the near-memory computing architecture, thereby obtaining the simulation results corresponding to each candidate compilation strategy. The simulator includes a hardware state space, an instruction queue, and a virtual memory controller. The hardware state space is used to simulate and track the hardware state of each hardware component in the near-memory computing architecture based on the tree-structured abstract model. The hardware components include storage units, computing units, cache units, and transmission buses. The instruction queue is used to store the complete instruction sequence compiled based on the candidate compilation strategy and maintain the data dependency relationship in the complete instruction sequence. The instruction queue includes multiple instruction groups. The independent instructions executed by different computing units are located in different instruction groups. The same instruction group includes multiple instructions that are executed sequentially. Instructions in instruction groups without data dependency relationships can be issued in parallel. The virtual memory controller is used to determine the issuance time of instructions in the instruction queue and update the hardware state of each hardware component in the hardware state space based on the hardware state space.

7. The method according to claim 6, characterized in that, The tracking of the hardware status of each hardware component in the near-memory computing architecture includes: The activation status of each memory cell in the near-memory computing architecture is tracked so that an activation command is issued before an inactive memory cell is accessed. Track the currently active row address in the storage unit to indicate whether subsequent access will trigger a row miss, and schedule precharge and activation commands to activate the row to be accessed when a row miss is triggered; The hardware state space employs multiple clock countdowns to track the hardware state of each hardware component. The hardware state of the storage unit includes the time it takes for the next command to be issued to the storage unit. The hardware states of other hardware components besides the storage unit include idle or occupied states. The clock countdowns corresponding to each hardware component are updated based on the hardware occupation duration of the hardware component involved in issuing the command.

8. The method according to claim 6 or 7, characterized in that, The simulation, based on the tree-structured abstract model, uses a simulator to simulate the compilation results corresponding to each candidate compilation strategy during the execution process of the near-memory computing architecture, obtaining simulation results for each candidate compilation strategy, including: For the compilation results corresponding to any candidate compilation strategy, the following simulation operation is executed repeatedly: The instruction queue determines the set of instructions that can be issued simultaneously based on the data dependencies involved in each instruction group, and sends the instruction set to the virtual memory controller. The instruction set includes the instruction that is currently the first instruction in each instruction group that has no data dependencies. The virtual memory controller determines the issuance time of each instruction in the instruction set based on the hardware state space, and selects the instruction with the earliest issuance time as the target instruction to be issued. The virtual memory controller, based on the issuance time of the target instruction, controls the hardware state of each hardware component in the hardware state space to be updated to the hardware state corresponding to the issuance time of the target instruction, and deletes the target instruction from the instruction queue. The virtual memory controller updates the hardware state of the hardware component occupied by the target instruction in the hardware state space according to the hardware occupancy duration of the hardware component occupied by the target instruction. When all instructions in the instruction queue have been issued and the hardware states of each hardware component in the hardware state space have become idle, the simulation results corresponding to the candidate compilation strategy are obtained.

9. The method according to claim 8, characterized in that, Determining the issuance time of each instruction in the instruction set based on the hardware state space includes: For the i-th instruction in the instruction set, when the operand of the i-th instruction comes from the memory unit, based on the hardware state space, the activation state of the row in the memory unit where the operand of the i-th instruction is located is determined, and based on the activation state of the row where the operand of the i-th instruction is located, the clock countdown of the memory unit, and the read delay of reading the operand from the memory unit, the arrival time of the operand of the i-th instruction to the computing unit is determined. When the operand of the i-th instruction comes from the cache unit, the arrival time of the operand of the i-th instruction to the computing unit is determined based on the release time of the cache unit, the read delay of reading the operand from the cache unit, and the idle time of the transmission bus between the cache unit and the computing unit. Based on the arrival time of the operand of the i-th instruction to the computing unit and the idle time of the computing unit, the start processing time of the computing unit for the operand of the i-th instruction is determined. The issuance time of the i-th instruction is determined based on the start processing time of the operand of the i-th instruction by the computing unit.

10. The method according to claim 9, characterized in that, The step of selecting the instruction with the earliest issuance time as the target instruction to be issued includes: selecting the instruction with the earliest start time as the target instruction to be issued.

11. A compilation system, characterized in that, include: The acquisition module is used to acquire the operator-level intermediate representation of the neural network model to be deployed to the near-memory computing architecture and the hardware configuration information of the near-memory computing architecture. The hardware configuration information is used to indicate the hardware configuration of the near-memory computing architecture. An abstract module is used to generate a tree-like abstract model of the near-memory computing architecture based on the hardware configuration information of the near-memory computing architecture. The tree-like abstract model is used to describe the near-memory computing architecture using a tree structure. A construction module is used to construct a search space for a compilation strategy based on the operator-level intermediate representation of the neural network model and the tree-like architecture abstract model of the near-memory computing architecture. The compilation strategy is used to compile the operator-level intermediate representation. The prediction module is used to determine multiple candidate compilation strategies and the compilation results corresponding to each candidate compilation strategy by performing performance prediction on the compilation strategies in the search space. The compilation results include a complete instruction sequence obtained by compiling the operator-level intermediate representation according to the candidate compilation strategies. The simulation module is used to simulate the execution process of the compilation results corresponding to each candidate compilation strategy on the near-memory computing architecture based on the tree-structured abstract model, and to obtain the simulation results corresponding to each candidate compilation strategy. The simulation results represent the total time required for the compilation results to be executed on the near-memory computing architecture. The determination module is used to determine the target compilation strategy from the multiple candidate compilation strategies based on the simulation results corresponding to each candidate compilation strategy, and to determine the compilation result of the target compilation strategy as the target compilation result of the operator-level intermediate representation of the neural network model, so as to deploy the neural network model on the near-memory computing architecture based on the target compilation result; The construction of the search space for the compilation strategy based on the operator-level intermediate representation of the neural network model and the tree-like abstract model of the near-memory computing architecture includes: Based on the operator-level intermediate representation and the tree-structured abstract model, following the order from the highest to the lowest level of the memory in the near-memory computing architecture, and under the condition of satisfying the first constraint, the operator-level intermediate representation is partitioned along multiple dimensions level by level to obtain multiple partitioning strategies; wherein, the first constraint includes that the product of the number of partitions along each dimension at each level is less than or equal to the number of hardware structures corresponding to each level in the memory, and different partitioning strategies have different partitioning methods under the first constraint. For each sub-intermediate representation resulting from each partitioning strategy, each sub-intermediate representation is divided into multiple data slices under the condition of satisfying the second constraint, and each data slice is mapped to the memory of the near-memory computing architecture under the condition of satisfying the third constraint, thus obtaining multiple mapping strategies corresponding to each partitioning strategy; wherein, the second constraint includes: the product between the size of each data slice and the data bit width of each data slice is less than or equal to the product between the burst length of the memory and the column bit width of the memory; the third constraint includes: the number of data slices mapped to the same row in the memory is less than or equal to the ratio between the total number of columns in that row in the memory and the burst length.

12. An electronic device comprising a memory and a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the steps of the method according to any one of claims 1 to 10.

13. A non-volatile computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 10.

Images