Memory address cache for neural networks

Abstract

A request is received to provide an application with direct memory access to data stored at an external memory address of an external memory. In response to determining that the external memory address is not registered in a cache, the data is copied from the external memory address to a first internal memory address within an internal memory. A first cache line within the cache associated with the external memory address is updated to include a reference to the first internal memory address. The data is provided from the internal memory to the application.

Description

Technical Field

[0001] This disclosure generally relates to memory systems, and more particularly to memory address caches for neural networks. Background Technology

[0002] A computing system may include one or more caches for storing data. For example, a computing system may include cache memory divided into three levels: Level 1 (L1) cache, Level 2 (L2) cache, and Level 3 (L3) cache. Generally, processors within a computing system (e.g., neural network (NN) accelerators) can utilize caches to temporarily store data retrieved from internal memory (e.g., SRAM) and external memory (e.g., DRAM). A processor can read data from and write data to a cache faster than it can read data from and write data to both internal and external memory. Similarly, a processor can read data from and write data to internal memory faster than it can read data from and write data to external memory. Furthermore, by reading / writing to / from the cache, the bandwidth to external memory can be reduced. Summary of the Invention

[0003] A controller circuitry operatively coupled to the processor receives a request for direct memory access to data stored in external memory at an external memory address, which is provided to an application. In response to determining that the external memory address is not registered in a cache, the controller circuitry copies the data from the external memory address to a first internal memory address within internal memory. The controller circuitry updates a first cache line associated with the external memory address in the cache to include a reference to the first internal memory address and provides the data from the internal memory to the application. In some embodiments, the first cache line includes a first portion corresponding to the first internal memory address, a second portion corresponding to a tag associated with the external memory address, and a third portion corresponding to an expiration number. In some embodiments, the controller circuitry invalidates the first cache line using the expiration number after a specified amount of time. In some embodiments, the first cache line is runtime configurable. In some embodiments, the processor is a neural network (NN) accelerator, and the data includes a data structure of input data associated with the workload of an NN application. In some embodiments, the processing device is a direct memory access (DMA) controller.

[0004] In some embodiments, the controller circuitry may determine that the external memory address is registered in the cache. In response to determining that the external memory address is registered in the cache, the controller circuitry identifies a second cache line associated with the external memory address within the cache. In some embodiments, the second cache line contains a reference to a second internal memory address. The controller circuitry then copies the data from the second internal memory address to a third internal memory address within the internal memory. Attached Figure Description

[0005] This disclosure will be more fully understood from the detailed description given below and from the accompanying drawings illustrating embodiments of this disclosure. The drawings are provided to give knowledge and understanding of embodiments of this disclosure and are not intended to limit the scope of this disclosure to these specific embodiments. Furthermore, the drawings are not necessarily drawn to scale.

[0006] Figure 1 The illustrations depict an example computing system including a memory system according to some embodiments of the present disclosure.

[0007] Figure 2 This is a diagram illustrating a memory address cache system according to some embodiments of the present disclosure.

[0008] Figure 3A This is a diagram illustrating an example of external memory address and cache according to some embodiments of the present disclosure.

[0009] Figure 3B This is a diagram illustrating an example of external memory address and cache according to some embodiments of the present disclosure.

[0010] Figure 4 A flowchart depicts an example method for a memory address cache for a neural network according to some embodiments of the present disclosure.

[0011] Figure 5 A diagram depicting an example computer system in which embodiments of the present disclosure may operate. Detailed Implementation

[0012] Various aspects of this disclosure relate to memory caches for neural networks.

[0013] Using neural network (NN) accelerators can significantly improve the speed and efficiency of artificial intelligence (AI) and machine learning (ML) applications, thereby reducing the time, energy, and processing resources required to train and deploy neural networks. NN accelerators are specialized hardware components optimized to perform the computations involved in training and inference of deep neural networks. For example, NN accelerators are designed to perform computationally intensive operations such as matrix multiplication, convolution, pooling, and activation functions at high speeds and high efficiency. With the ever-growing demand for AI applications, the development of more efficient and specialized NN accelerators is becoming increasingly important.

[0014] Because neural networks (NNs) consume significant processing resources and bandwidth, processors (e.g., NN accelerators) can retrieve data from external (i.e., off-chip) memory (e.g., dynamic random access memory (DRAM)) and temporarily store that data in a cache. A cache provides faster access to data that is frequently used or requested by the processor, reducing latency and improving overall system performance. Additionally, using a cache reduces traffic that passes through the interface to external memory. However, because cache memory space is typically limited, caching large amounts of data (e.g., tens or hundreds of gigabytes) processed by a few NN applications is increasingly challenging. For example, some NN applications (e.g., sparse submanifold convolution, deformable convolution, deformable attention, etc.) can process large data structures (e.g., 1,024-byte vectors) across several layers of the NN application. Caching such large vectors processed by a few NNs can lead to numerous cache misses and cache thrashing, thus degrading overall system performance. Therefore, large input and output vectors can be stored in external memory.

[0015] Given the above, when increasing the capacity of on-chip cache may not be reasonable, the latency associated with external memory access can be reduced. Some techniques utilize Direct Memory Access (DMA) to prefetch data from external memory before the workload of a neural network (NN) application runs. DMA is a feature of computer systems that allows hardware devices (e.g., DMA controllers) to access the system's memory without requiring the involvement of the system's processor (e.g., NN accelerator). DMA controllers can transfer large blocks of data from external memory, which reduces the amount of time the NN accelerator spends waiting for data, allowing it to perform other computations. However, such techniques relying on DMA cannot reduce latency in NN applications with unpredictable access patterns. Unpredictable access patterns refer to situations where memory access requests are made in ways that may be difficult to anticipate by the system's memory hierarchy. For example, deformable convolution is a type of NN model with unpredictable access patterns. Deformable convolution has unpredictable access patterns because the offsets used to shift corresponding fields of the model are determined during training and can take any value within a specific range. Therefore, the exact location and shape of the corresponding field in each output unit can vary depending on the offset and input characteristics. Unpredictable memory access patterns can disrupt the performance of the memory hierarchy and render such DMA techniques unsuitable for neural network applications with unpredictable memory access patterns.

[0016] This disclosure provides a system and method for caching the memory addresses of large (e.g., 1 kilobyte (kB)) data structures processed by a neural network (NN) application workload with unpredictable access patterns. A DMA controller may receive a request for direct memory access to data stored at an external memory address in external memory for an application (e.g., an NN application). The external memory may include off-chip memory (e.g., DRAM) associated with a processor (e.g., an NN accelerator) executing the NN application workload. The DMA controller may search a cache located on the processor to determine whether the external memory address is registered in the cache. If an index and a tag associated with the external memory address are currently present in the cache, then the external memory address is registered in the cache, resulting in a cache hit. In some embodiments, the cache is configured to store the memory address of a reference memory location where the data is stored. For example, a cache line may contain a reference to an internal memory address where the requested data is stored. The DMA controller may identify the internal memory address as an addressable location within internal memory.

[0017] In some embodiments, the DMA controller may determine that an external memory address is not registered in the cache, resulting in a cache miss. In response to determining that the external memory address is not registered in the cache, the DMA controller may perform a read operation at the external memory address to retrieve the requested data from the external memory. The DMA controller may copy the data from the location in memory corresponding to the external memory address to the location in memory corresponding to the internal memory address. The DMA controller may update the cache line associated with the external memory address to include a reference to the internal memory address within the cache. The DMA controller may further provide the requested data from internal memory to the NN application.

[0018] In some embodiments, the DMA controller may determine that an external memory address is registered in a cache, thereby achieving a cache hit. In response to determining that the external memory address is registered in a cache, the DMA controller may identify a cache line in the cache associated with data requested by the processor. The cache line may contain a reference to a first internal memory address. The DMA controller may copy the data from a location in internal memory corresponding to the first internal memory address to another location in internal memory corresponding to a second internal memory address. The DMA controller may further provide the requested data from internal memory to the NN application.

[0019] By leveraging the flexibility of storing references to data in a cache, one or more embodiments of this disclosure may store references to internal memory addresses in a cache, reducing traffic between the processor and external memory. The technical advantages of this disclosure include, but are not limited to, storing data in internal memory and references to internal memory in a cache, enabling the retrieval of requested data from internal memory. This significantly reduces external memory accesses, thereby reducing traffic via interfaces connected to external memory. Therefore, the systems and methods of this invention reduce off-chip data access and associated power consumption, thereby improving the overall energy efficiency and latency of the system, and enabling the system to increase overall utilization, thereby improving performance.

[0020] It should be noted that the various aspects of the methods and systems referenced above are described in detail below by way of example rather than by way of limitation. For the purposes of simplicity and brevity only, the embodiments and examples provided below are applicable to neural network applications and neural network accelerators. However, the embodiments and examples of this disclosure can be applied to applications with repetitive tasks of processing large amounts of data that occur at the programming level. For example, aspects and embodiments of this disclosure can be applied to high-performance computing applications.

[0021] Figure 1The illustration depicts an example computing system 100 including a memory system according to some embodiments of the present disclosure. The memory system may include a direct memory access (DMA) controller 115, internal memory 120, and external memory 130. The computing system 100 may be a computing device, such as a desktop computer, laptop computer, web server, mobile device, vehicle (e.g., an airplane, drone, train, car, or other means of transportation), an Internet of Things (IoT) capable device, an embedded computer (e.g., an embedded computer included in a vehicle, industrial equipment, or networked business device), or such a computing device including memory and processing means. In at least one embodiment, one or more components of the computing system 100 may be packaged on the same die.

[0022] The computing system 100 may include a processor 110, such as a host processor, operatively coupled to a DMA controller 115. In some embodiments, the processor 110 is operatively coupled to a DMA controller 115, which may be part of a different type of memory system. Figure 1 The diagram illustrates an example of a processor 110 operatively coupled to a DMA controller 115. As used herein, “coupled to” or “coupled with” generally refers to a connection between components, which can be a wired or wireless indirect communication connection or a direct communication connection (e.g., without an intermediary component), including connections such as electrical connections, optical connections, magnetic connections, etc. The processor 110 may include a processor chipset and a software stack executed by the processor chipset. The processor chipset may include one or more cores, one or more caches, a memory controller (e.g., a non-volatile dual inline memory module (NVDIMM) controller), and a memory protocol controller (e.g., a high-speed peripheral component interconnect (PCIe) controller, a Serial Advanced Technology Attachment (SATA) controller). In some embodiments, the processor 110 is a neural network (NN) accelerator. The NN accelerator may be in the form of a dedicated hardware component, such as a graphics processing unit (GPU), a field-programmable gate array (FPGA), a tensor processing unit (TPU), an application-specific integrated circuit (ASIC), etc. In some embodiments, the NN accelerator may be implemented as one or more software components running on a general-purpose CPU or GPU.

[0023] Processor 110 may include cache 112 (e.g., processor cache). In some embodiments, cache 112 may be an on-chip cache located on the same chip as processor 110 and may include multiple cache levels (L1, L2, L3, and L4). In some embodiments, cache 112 may be shared between processor 110 and DMA controller 115, thereby enabling both to access cache 112 simultaneously. Processor 110 may use cache coherence protocols (e.g., Modify-Exclusive-Shared-Invalid (MESI) coherence protocol, Modify-Own-Exclusive-Shared-Invalid (MOESI) coherence protocol, etc.) to allow processor 110 and DMA controller 115 to perform consistent access.

[0024] Processor 110 may include internal memory 120. Internal memory 120 may include any combination of different types of non-volatile memory and / or volatile memory. Internal memory 120 (also referred to herein as on-chip memory) may be directly integrated into the same chip as the processing units of processor 110. Volatile memory may be, but is not limited to, static random access memory (SRAM), embedded dynamic random access memory (eDRAM), and ferroelectric random access memory (FeRAM). Non-volatile memory may be, but is not limited to, flash memory, phase-change memory (PCM), resistive random access memory (RRAM), and magnetic random access memory (MRAM). Internal memory 120 may include one or more memory arrays composed of memory cells. Internal memory 120 may also include additional circuitry or components not illustrated. In some embodiments, internal memory 120 may include address circuitry (e.g., row decoder and column decoder) capable of receiving addresses from DMA controller 115 and decoding the addresses to access internal memory 120.

[0025] External memory 130 (also referred to herein as off-chip memory) may be auxiliary memory not located directly on processor 110. External memory 130 may comprise any combination of different types of non-volatile memory. For example, external memory 130 may include, but is not limited to, dynamic random access memory (DRAM), flash memory, hard disk drive (HDD), solid-state drive (SSD), universal serial bus (USB) flash drive, memory card, external hard drive, etc. External memory 130 may also include additional circuitry or components not illustrated. In some embodiments, external memory 130 may be separate from system 100 and may be accessed by system 100 via a wireless network, Ethernet port, USB port, or other connection.

[0026] The DMA controller 115 can communicate with internal memory 120, external memory 130, and cache 112 to perform operations such as reading data, writing data, transferring data, or erasing data, and other such operations, without the intervention of processor 110. The DMA controller 115 may include hardware, such as one or more integrated circuits and / or discrete components, buffer memories, or combinations thereof. The hardware may include a digital circuit system with dedicated (i.e., hard-decoded) logic to perform the operations described herein. The DMA controller 115 may be a microcontroller, a dedicated logic circuit system (e.g., a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), etc.), or another suitable processor.

[0027] Generally speaking, the DMA controller 115 can receive commands or operations from the processor 110 and can translate those commands or operations into instructions or appropriate commands to perform the desired memory operation. The processor 110 can perform other operations while the memory operation is in progress and can receive an interrupt from the DMA controller 115 when the memory operation is completed. For example, the processor 110 can initiate a data transfer from external memory 130 to internal memory 120. The DMA controller can transfer data from external memory 130 to internal memory 120 while the processor 110 performs other operations. In response to the completion of the data transfer from external memory to internal memory, the DMA controller 115 can send an interrupt to the processor 110.

[0028] The DMA controller 115 may include multiple hardware registers that can be written to and read from by the processor 110. The DMA controller 115 may include registers such as: a counter register for holding the number of bytes to be transferred; an address register for holding the source and destination memory addresses of the data transfer; one or more control registers for controlling the operation of the DMA controller 115; a status register for indicating the current status of the DMA transfer, including completion and error conditions; and so on. The DMA controller 115 may include a DMA interface bus (not illustrated) that connects the DMA controller 115 to the system bus.

[0029] In some embodiments, the DMA controller 115 may perform the operations described herein. The DMA controller 115 may store and retrieve references to the internal memory 120 within cache 112 to perform the operations described herein, as follows: Figure 2Detailed description. In some embodiments, the DMA controller 115 may perform the operations described herein using software (e.g., instructions that run or execute on or on the processor 110). In some embodiments, the DMA controller 115 may perform the operations described herein using processing logic, which may include hardware (e.g., processing means, circuitry, dedicated logic, microcode, device hardware, integrated circuits, etc.). In some embodiments, the DMA controller 115 may perform the operations described herein using a combination of processing logic and software.

[0030] Figure 2 This is a diagram illustrating a memory address cache system according to some embodiments of the present disclosure. Figure 200 may include information related to... Figure 1 The described computing system 100 illustrates similar components. It should be noted that... Figure 1 The components can be used in this article to help describe Figure 2 .about Figure 2 The operations described are shown to be performed sequentially for illustrative purposes and not for limitation. Although shown in a specific sequence or order, the order of operations may be modified unless otherwise specified. Therefore, the illustrated embodiments should be understood as merely examples, and the described operations may be performed in different orders, and some operations may be performed in parallel. In addition, one or more operations may be omitted in some embodiments. Therefore, not all described operations are required in every embodiment, and other process flows are possible. In some embodiments, the same, different, fewer, or more operations may be performed. Figure 200 illustrates a memory address caching technique that utilizes a DMA controller 215 to cache references (e.g., memory addresses) to data processed by workloads of NN applications executing on NN accelerators.

[0031] The neural network (NN) described herein may comprise multiple layers connected by nodes, which can be trained using input data to solve complex problems. For example, images can be used as input data to construct, train, and deploy an NN model for image classification or object detection. Once trained, the NN can be deployed and used to identify and classify objects or patterns during inference, through which the NN application processes information from a given input to infer results. During the inference process for a given workload, the NN application may perform memory accesses to retrieve various types of data from external memory. For example, the NN application may read input data supplied to the NN application from external memory for inference. In another instance, the NN application may read weights or biases from external memory for use within the NN application. Weights and biases are learned during the training of the NN and used by the NN application to transform input data into output data. Weights and biases may be retrieved from external memory by the layers of the NN application. In some examples, the data processed by the NN application (e.g., input data, weights, biases, etc.) may be read from external memory multiple times.

[0032] DMA controller 215 may be configured to receive a request for direct memory access (DMA) of data 202 stored at external memory address 201 in external memory 230 for an application (e.g., a neural network application). In some embodiments, the request may be received from a processor (e.g., processor 110). In some embodiments, the processor may be a neural network (NN) accelerator, and the requested data may be data associated with the workload of an NN application executing on the NN accelerator, as described above regarding... Figure 1 As described, the DMA controller 215 can determine whether the external memory address 201 is registered in the cache 240.

[0033] Cache 240 may be configured to store references to internal memory addresses of internal memory 210. In some embodiments, cache 240 may correspond to... Figure 1 The high-speed cache 112 and the internal memory 220 can correspond to Figure 1 The internal memory 120, and the external memory 230 may correspond to Figure 1 External memory 130.

[0034] Given a received external memory address 201, the DMA controller 215 can determine whether the external memory address is registered in cache 240. In some embodiments, a portion of the external memory address may contain an index corresponding to cache 240. The index may indicate a set within cache 240. A set is a group of cache lines within cache 240 that share the same index bit. In some embodiments, a portion of the external memory address 201 may contain a tag corresponding to cache 240. For example, external memory address 201 may contain a tag 224 corresponding to cache line 228 within cache 240. For each cache line in the identified set, the DMA controller 215 can compare the tag associated with the corresponding cache line with the tag of external memory address 201. If a match is found based on the comparison, then external memory address 201 is registered in cache 240. If no match is found, then external memory address 201 is not registered in cache 240.

[0035] In response to determining that external memory address 201 is not registered in cache 240, DMA controller 215 may copy the requested data from the location in external memory 230 corresponding to external memory address 201 to the location in internal memory 220 corresponding to internal memory address 226. For example, external memory address 201 may refer to data 202 stored in external memory 230, as illustrated in the figure. DMA controller 215 may copy data 202 from external memory 230 to internal memory 220. To copy data 202, DMA controller 215 may read data 202 from external memory 230 and store data 202 at the internal memory address in internal memory 220, as illustrated in the figure. In some embodiments, DMA controller 215 may update the cache line in cache 240 associated with external memory address 201 to include a reference to internal memory address 226. For example, DMA controller 215 may update cache line 228 of cache 240 to include internal memory address 226. Additionally, the DMA controller 215 can update tag 224 of cache line 228 with the tag corresponding to external memory address 201. The DMA controller 215 can provide data 202 stored at internal memory address 226 to the NN accelerator.

[0036] In response to determining that external memory address 201 is registered in cache 240, DMA controller 215 can copy data from an internal memory address referenced in a cache line to another internal memory address. For example, DMA controller 215 can determine that external memory address 201 is registered at cache line 228 in cache 240. DMA controller 215 can identify internal memory address 226 within cache line 228. DMA controller 215 can identify data 202 in internal memory 220 corresponding to internal memory address 226. DMA controller 215 can read data 202 from internal memory 220 corresponding to internal memory address 226 and copy data 202 to another location within internal memory 220.

[0037] In some embodiments, when the DMA controller 215 copies data 202 from external memory 230 to internal memory 220, or when the DMA controller 215 copies data 202 from internal memory 220 to another location within internal memory 220, the NN accelerator (e.g., processor 110) may request the DMA controller 215 to read data 202. The NN accelerator may have specific requests regarding the format and organization of the data stored in internal memory 220 to enable the NN accelerator to operate on data 202 efficiently. Therefore, the DMA controller 215 may transform data 202 into the appropriate data structure and format requested by the NN accelerator. For example, the NN accelerator may request that data 202 be organized into a specific array or tensor, or that data elements have a specific format (e.g., fixed-point numbers, floating-point numbers, etc.). When the DMA controller 215 creates a copy of data 202 within internal memory 220, the DMA controller 215 may reorganize the data 202 according to the requested format. In some embodiments, the NN accelerator may support multiple data formats and enable data to be stored directly in internal memory without any additional transformation.

[0038] In some embodiments, each cache line within cache 240 may contain an expiration number. The expiration number associated with a cache line indicates whether the cache line has expired. An expired cache line may be considered invalid, meaning that the internal memory address associated with the expired cache line is not usable by the processor. When an internal memory address is added to cache 240, an expiration number may be assigned to the associated cache line. The cache line can then be invalidated by changing a register indicating which expiration numbers are valid. For example, DMA controller 215 may add cache line 228 to cache 240 and set expiration number 222. A processor (e.g., an NN accelerator) may invalidate a cache line by updating a register (e.g., located on the NN accelerator) that compares the cache line with the expiration number 222 in the cache line. If the register value is greater than the expiration number 222, then cache line 228 is invalidated. In some embodiments, the register value may be a timestamp (e.g., time-based cache invalidation) that is updated (e.g., incremented) at periodic time intervals associated with cache line 228. In some embodiments, the expiration number 222 may be updated based on the occurrence of an event (e.g., an event-based cache line invalidation). For example, the value in the register may be updated in response to the completion of an NN layer associated with an internal memory address (e.g., incremented, set to a defined value, etc.).

[0039] During memory access to cache line 228, DMA controller 215 may check the expiration number 222 to determine if cache line 228 is valid. In response to determining that cache line 228 is valid, cache 228 may provide internal memory address 226 to DMA controller 215. In response to determining that cache line 228 is invalid, cache 240 may provide DMA controller 215 with an indication that cache line 228 is invalid, which can be considered a cache miss. It should be understood that the cache line expiration mechanism described above may be implemented by another cache invalidation mechanism (e.g., a cache refresh mechanism, a probabilistic cache expiration mechanism, etc.). In some embodiments, cache 240 may implement a cache replacement policy to determine which cache lines will be replaced when cache 240 is full. For example, cache 240 may implement a Least Recently Used (LRU) algorithm to ensure that the most recently accessed cache line remains in the cache.

[0040] In some embodiments, external memory address 201 and internal memory address 226 may refer to a data structure associated with the workload of an NN application currently executing on the NN accelerator. For example, the data structure may be a 1,024-byte vector of input data associated with the first layer of the NN application, as shown in the figure. Figure 3BA detailed description is provided. A 1,024-byte vector of input data can be 1,024-byte aligned in external memory 230. Input data can also be 1,024-byte aligned in internal memory 220. In another example, the data structure can be a 256-byte vector of input data associated with the second layer of the NN application, as described in [the following text is missing from the original] Figure 3A Detailed description. The 256-byte vector of input data can be aligned to 256 bytes in external memory 230. Input data can also be aligned to 256 bytes in internal memory 220.

[0041] In some embodiments, cache lines in cache 240 may be runtime-configurable to accommodate different sizes of input data. Cache 240 being runtime-configurable means that cache 240 can be configured or modified prior to the runtime of a given workload of the application. That is, the number of tag bits used to tag cache lines can be adjusted to account for the size of the input data to be processed by a given layer of the application. For example, the first layer of an NN application executing on an NN accelerator (e.g., processor 110) may process a 256-byte alignment vector of data stored in external memory 230. Prior to the runtime of the first layer, each cache line of cache 240 may be configured to include a 16-bit tag to identify the 16 most significant bits (MSB) of the external memory address associated with a particular cache line. The subsequent 8 bits of the external memory address may be associated with an index of cache 240, as described below regarding... Figure 3A As described. In another instance, the second layer of an NN application executing on an NN accelerator can process a 1024-byte alignment vector of data stored in external memory 230. Before the first workload runs, each cache line of cache 240 can be updated to include a 14-bit tag identifying the 14 most significant bits (MSB) of the external memory address associated with a particular cache line. The subsequent 8 bits of the external memory address can be associated with an index in cache 240, as described below regarding... Figure 3B As described.

[0042] Figure 3AThis is a diagram 300 illustrating an example external memory address 301 and a cache 310 according to some embodiments of the present disclosure. External memory address 301 is an example configuration of an external memory address that references data in external memory (e.g., external memory 230) aligned to 256 bytes. External memory address 301 is a 32-bit address comprising a first portion corresponding to tag 302, a second portion corresponding to index 304, and a third portion corresponding to field 306. Tag 302 is a 14-bit tag that references a location in external memory corresponding to 1024 bytes of data. In some embodiments, tag 302 corresponds to a tag within cache 310. Index 304 is an 8-bit reference to an index within cache 310.

[0043] In some embodiments, field 306 may be a portion of external memory address 301 that is not used under the illustrated addressing scheme. In some embodiments, the bit corresponding to field 306 may be reserved as zero, may be written as 1, or may be written as either 1 or any combination of zero. In some embodiments, field 306 may be used for purposes other than memory addressing. For example, field 306 may be used to indicate other attributes of the memory block referenced by external memory address 301 or access permission.

[0044] In some embodiments, external memory address 301 can be reformatted for memory addressing under an alternative configuration of external memory address space and cache 310. For example, a 256-byte vector of data processed by a given workload of an application (e.g., an NN application) can be determined. Prior to the execution of a given workload, the external memory corresponding to external memory address 301 can be reconfigured to a 256-byte alignment. Tag 302 of external memory address 301 can be updated from the most significant 14 bits of external memory address 301 to the most significant 16 bits of external memory address 321, as described below. Figure 3B The diagram illustrates this. Therefore, index 304 and field 306 can be shifted by 2 bits to account for the increased size of tag 322, thereby reducing tag 326 from 10 bits to 8 bits. Cache 310 can be updated to contain the 16-bit tag of the updated tag 322 corresponding to external memory address 321, such as... Figure 3B As shown in the image.

[0045] In some embodiments, cache 310 may correspond to Figure 1 Cache 112. In some embodiments, cache 310 may correspond to... Figure 2 The cache 240, and the external memory address 301 can correspond to Figure 2 The external memory address is 201.

[0046] Figure 3B This is diagram 320 illustrating an example external memory address 321 and cache 330 according to some embodiments of the present disclosure. External memory address 321 is an example configuration of an external memory address that references data in external memory (e.g., external memory 230) aligned to 256 bytes. External memory address 321 is a 32-bit address comprising a first portion corresponding to tag 322, a second portion corresponding to index 324, and a third portion corresponding to field 326. Tag 322 is a 16-bit tag that references the location in external memory corresponding to 256 bytes of data. In some embodiments, tag 322 corresponds to a tag within cache 330. Index 324 is an 8-bit reference to an index within cache 330.

[0047] In some embodiments, field 326 may be a portion of external memory address 321 that is not used under the illustrated addressing scheme. In some embodiments, the bit corresponding to field 326 may be reserved as zero, may be written as 1, or may be written as any combination of 1 and zero. In some embodiments, field 326 may be used for purposes other than memory addressing. For example, field 326 may be used to indicate other attributes of the memory block referenced by external memory address 321 or access permission.

[0048] In some embodiments, field 326 may be a portion of external memory address 321 that is not used in the described configuration. Note that in alternative configurations, a portion of field 326 may be available for memory addressing. For example, a 1024-byte vector of data that an application (e.g., an NN application) can store and retrieve for a given workload may be determined. Prior to the execution of a given workload, the external memory corresponding to external memory address 321 may be reconfigured to a 1024-byte alignment. Therefore, tag 322 of external memory address 321 may be updated from the most significant 16 bits of external memory address 321 to the most significant 14 bits of external memory address 301, as per [reference to external memory address 301]. Figure 3A The diagram illustrates this. Index 324 and field 326 can be shifted to account for the reduction, and cache 330 can be updated to include a 14-bit tag.

[0049] In some embodiments, cache 330 may correspond to Figure 1 Cache 112. In some embodiments, cache 330 may correspond to... Figure 2 The cache 240, and the external memory address 321 can correspond to Figure 2 The external memory address is 201.

[0050] Figure 4A flowchart depicts an example method for a memory address cache for a neural network according to some embodiments of the present disclosure. Method 400 may be executed using a controller circuitry system, which may include hardware (e.g., a processing device, a processing circuitry system, special-purpose logic, microcode, device hardware, integrated circuits, etc.), software (e.g., instructions that run or execute on the processing device), firmware, or a combination thereof. In at least one embodiment, method 400 may be executed by a non-transitory computer-readable medium having instructions that, in response to execution by a processor, cause a processor of a computing system to perform the operations described herein. In one embodiment, method 400 may be executed by a system (e.g., a processor that executes the instructions). Figure 1 The method 400 is executed by one or more components of the system 100. In some embodiments, one or more operations of the method 400 may be performed by... Figure 1 The DMA controller 115 performs the operation. In some embodiments, one or more operations of method 400 may be performed by... Figure 2 The DMA controller 215 performs the operation. In some embodiments, one or more operations of method 400 may be performed by one or more other machines not depicted in the figure.

[0051] For simplicity of explanation, the method is depicted and described as a series of actions. However, the actions according to this disclosure can occur in various orders and / or simultaneously, and with other actions not presented or described herein. Furthermore, not all illustrated actions can be performed to implement the method according to the disclosed object. Additionally, those skilled in the art will understand and appreciate that the method may alternatively be represented by a state diagram or events as a series of related states. Furthermore, it should be understood that the methods disclosed in this specification can be stored on an article of manufacture for transporting and transmitting such methods to a computing device. As used herein, the term "article of manufacture" is intended to encompass a computer program accessible from any computer-readable device or storage medium.

[0052] At 402 of method 400, a controller circuitry operatively coupled to a processor (e.g., processor 110) receives a request for direct memory access to data stored at an external memory address in external memory (e.g., external memory 130) for an application running on the processor. In some embodiments, the controller circuitry is a direct memory access controller, such as DMA controller 115. In some embodiments, the processor is a neural network (NN) accelerator, and the data comprises a data structure containing input data associated with the NN application.

[0053] At 404, the controller circuitry determines whether the external memory address is registered in a cache (e.g., cache 112 of processor 110). For example, the control circuitry may check the cache to find the index and tag corresponding to the external memory address. If the tag in the cache matches the tag corresponding to the external memory address (i.e., a cache hit), then the external memory address is registered in the cache. If no tag associated with the external memory address exists in the cache (i.e., a cache miss), then the external memory address is not registered in the cache. In response to determining that the external memory address is not registered in the cache, method 400 continues to 406. In response to determining that the external memory address is registered in the cache, method 400 continues to 410.

[0054] At 406, the controller circuitry copies data from an external memory address to a first internal memory address within internal memory (e.g., internal memory 120 of processor 110). The internal memory address corresponds to an addressable location within internal memory. In some embodiments, the mapping between the external memory address and the first internal memory address may be managed by a DMA controller (e.g., DMA controller 115 of system 100). For example, this may involve mapping the memory space of the external memory to the address space of the processor. Thus, the DMA controller can access the external memory as if it were part of the internal memory, enabling the DMA controller to perform data transfers between external and internal memory without processor intervention.

[0055] At 408, the controller circuitry updates a first cache line associated with an external memory address within the cache to include a reference to a first internal memory address. In some embodiments, the first cache line includes a first portion corresponding to the first internal memory address, a second portion corresponding to a tag associated with the external memory address, and a third portion corresponding to an expiration number. In some embodiments, the controller circuitry invalidates the first cache line using the expiration number after a specified amount of time. In some embodiments, the first cache line is runtime configurable.

[0056] At 410, the controller circuitry identifies a second cache line within the cache that is associated with an external memory address. This second cache line contains a reference to a second internal memory address.

[0057] At address 412, the controller circuitry copies data from a second internal memory address to a third internal memory address within the internal memory. In some embodiments, the controller circuitry may read data while copying it from the second internal memory address to the third internal memory address. A processor coupled to the controller circuitry may request a specific format or organization of the data to enable the processor to operate on the data efficiently. For example, the processor may request that the data be organized into a specific array or tensor, or that the data elements have a specific format (e.g., fixed-point numbers, floating-point numbers, etc.). Therefore, the controller circuitry may copy data from the second internal memory address, transform the copied data into the appropriate data structure and format requested by the processor, and store the transformed data at the third internal memory address.

[0058] At 414, the controller circuitry provides data from internal memory to the application.

[0059] Figure 5 The illustration depicts an example machine of computer system 500, within which a set of instructions can be executed to cause the machine to perform any or more of the methods discussed herein. In alternative embodiments, the machine may be connected (e.g., networked) to other machines in a LAN, intranet, extranet, and / or the Internet. The machine may operate as a server or client machine in a client-server network environment, as a peer-to-peer (or distributed) network environment, or as a server or client machine in a cloud computing infrastructure or environment.

[0060] The machine may be a personal computer (PC), tablet PC, set-top box (STB), personal digital assistant (PDA), cellular phone, web appliance, server, network router, switch, or bridge, or any machine capable of executing a set of instructions (sequential or otherwise) specifying actions to be taken by the machine. Furthermore, while the illustrations depict a single machine, the term "machine" should also be considered as encompassing any collection of machines that individually or jointly execute a set of instructions (or multiple sets of instructions) to perform any or more of the methods discussed herein.

[0061] Example computer system 500 includes a processing device 502, a main memory 504 (e.g., read-only memory (ROM), flash memory, dynamic random access memory (DRAM) (e.g., synchronous DRAM (SDRAM)), a static memory 506 (e.g., flash memory, static random access memory (SRAM) and so on) and a data storage device 518, which communicate with each other via a bus 530.

[0062] Processing device 502 represents one or more processors (e.g., microprocessors, central processing units, etc.). More specifically, the processing device may be a Complex Instruction Set Computing (CISC) microprocessor, a Reduced Instruction Set Computing (RISC) microprocessor, a Very Long Instruction Word (VLIW) microprocessor, or a processor implementing other instruction sets or combinations of instruction sets. Processing device 502 may also be one or more special-purpose processing devices (e.g., application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), network processors, etc.). Processing device 502 may be configured to execute instructions 526 for performing the operations and steps described herein.

[0063] The computer system 500 may further include a network interface device 508 for communication via a network 520. The computer system 500 may also include a video display unit 510 (e.g., a liquid crystal display (LCD) or a cathode ray tube (CRT)), an alphanumeric input device 512 (e.g., a keyboard), a cursor control device 514 (e.g., a mouse), a graphics processing unit 522, a signal generation device 516 (e.g., a speaker), a graphics processing unit 522, a video processing unit 528, and an audio processing unit 532.

[0064] The data storage device 518 may include a machine-readable storage medium 524 (also referred to as a non-transitory computer-readable medium) storing one or more sets of instructions 526 or software embodying any or more of the methods or functions described herein. The instructions 526 may also reside wholly or partially in main memory 504 and / or processing device 502 during execution by computer system 500, which also constitute machine-readable storage media.

[0065] In some embodiments, instruction 526 includes instructions for implementing functionality corresponding to this disclosure. Although machine-readable storage medium 524 is shown as a single medium in example embodiments, the term "machine-readable storage medium" should be understood as including a single medium or multiple media (e.g., a centralized or distributed database and / or associated caches and servers) storing one or more sets of instructions. The term "machine-readable storage medium" should also be understood as including any medium capable of storing or encoding sets of instructions for execution by a machine, thereby enabling the machine and processing device 502 to perform any or more of the methods of this disclosure. Therefore, the term "machine-readable storage medium" should be understood as including, but not limited to, solid-state memory, optical media, and magnetic media.

[0066] Some of the foregoing detailed descriptions have been presented regarding the algorithms and symbolic representations for manipulating data bits within computer memory. These algorithmic descriptions and representations are methods used by those skilled in the art of data processing to most effectively communicate the essence of their work to those skilled in other related fields. An algorithm can be a sequence of operations that leads to a desired result. These operations are those that require physical manipulation of physical quantities. These quantities can take the form of electrical or magnetic signals that can be stored, combined, compared, and otherwise manipulated. These signals can be referred to as bits, values, elements, symbols, characters, items, numbers, etc.

[0067] However, it should be remembered that all these terms and similar terms are associated with appropriate physical quantities and are merely convenient notations for application to those quantities. Unless otherwise specifically stated in this disclosure, it should be understood that throughout this description, specific terms refer to the actions and processes of a computer system or similar electronic computing device that manipulate and transform data represented as physical (electronic) quantities in the registers and memories of the computer system into other data similarly represented as physical quantities in the computer system's memory or registers or other such information storage devices.

[0068] This disclosure also relates to an apparatus for performing the operations described herein. This apparatus may be specifically constructed for a given purpose, or it may comprise a computer selectively activated or reconfigured by a computer program stored therein. This computer program may be stored in a computer-readable storage medium, such as, but not limited to, any type of disk (including floppy disks, optical disks, CD-ROMs, and magneto-optical disks), read-only memory (ROM), random access memory (RAM), EPROM, EEPROM, magnetic cards, or optical cards, or any type of medium suitable for storing electronic instructions, each coupled to a computer system bus.

[0069] The algorithms and displays presented herein are not inherently related to any particular computer or other device. Various other systems may be used with the programs taught herein, or it may prove convenient to construct more specialized devices to perform similar sequences of process steps. Furthermore, this disclosure is not described with reference to any particular programming language, and any language used in such computer systems may be used to implement the teachings of this disclosure described herein.

[0070] This disclosure may be provided as a computer program product or software that may include a non-transitory computer-readable storage medium (CRM) having instructions stored thereon, which can be used to program a computer system (or other electronic device) to perform processes according to this disclosure. The CRM includes any mechanism for storing information in a machine-readable (e.g., computer-readable) form. For example, the CRM includes machine-readable (e.g., computer-readable) storage media such as read-only memory (“ROM”), random access memory (“RAM”), disk storage media, optical storage media, flash memory devices, and the like.

[0071] In the foregoing disclosure, embodiments of the present disclosure have been described with reference to characteristic examples. It will be apparent that various modifications may be made to the present disclosure without departing from the broader spirit and scope of the embodiments set forth in the appended claims. Where elements are referred to in the singular tense in this disclosure, more than one element may be depicted in the figures, and similar elements may be labeled with similar numbers. Therefore, this disclosure and the figures should be considered illustrative rather than restrictive.

Claims

1. A system comprising: A processor, which includes cache and internal memory; as well as A controller circuitry operatively coupled to the processor, the controller circuitry being configured to perform operations including: Receives a request for the application to access data stored at an external memory address, the external memory being coupled to the controller circuitry. Copy the data from the external memory address to the first internal memory address within the internal memory; as well as The first cache line within the cache is updated to include a reference to the address of the first internal memory.

2. The system of claim 1, wherein copying the data from the external memory address to the first internal memory address within the internal memory is performed in response to determining that the external memory address is not registered in the cache.

3. The system of claim 1, further comprising providing the data from the internal memory to the application.

4. The system according to claim 1, further comprising: In response to determining that the external memory address is registered in the cache: Identify a second cache line within the cache that is associated with the external memory address, the second cache line containing a reference to a second internal memory address; as well as The data is copied from the second internal memory address to a third internal memory address within the internal memory.

5. The system of claim 1, wherein the first cache line comprises a first portion corresponding to the first internal memory address, a second portion corresponding to a tag associated with the external memory address, and a third portion corresponding to an expiration number.

6. The system of claim 5, further comprising invalidating the first cache line using the expiration number.

7. The system of claim 6, wherein the first cache line is runtime configurable.

8. The system of claim 1, wherein the processor includes a neural network (NN) accelerator, and the data includes a data structure of input data associated with the workload of the NN application.

9. The system of claim 1, wherein the controller circuitry includes a direct memory access (DMA) controller.

10. An apparatus comprising: A direct memory access (DMA) controller, coupled to the processor and configured to perform operations including: Receives a request for the application to access data stored at an external memory address, the external memory being coupled to the DMA controller; Copy the data from the external memory address to a first internal memory address within the processor's internal memory; as well as The first cache line associated with the external memory address within the processor's cache is updated to include a reference to the first internal memory address.

11. The device of claim 10, wherein copying the data from the external memory address to the first internal memory address within the processor's internal memory is performed in response to determining that the external memory address is not registered in the processor's cache.

12. The device of claim 10, further comprising providing the data from the internal memory to the application.

13. The device of claim 10, wherein the DMA controller is configured to perform operations further including: In response to determining that the external memory address is registered in the cache: Identify a second cache line within the cache associated with the external memory address, the second cache line containing a reference to a second internal memory address; and The data is copied from the second internal memory address to a third internal memory address within the internal memory.

14. The device of claim 10, wherein the first cache line comprises a first portion corresponding to the first internal memory address, a second portion corresponding to a tag associated with the external memory address, and a third portion corresponding to an expiration number.

15. The device of claim 14, wherein the DMA controller is configured to perform operations further including: Invalidate the first cache line using the expiration number.

16. The device of claim 15, wherein the first cache line is runtime configurable.

17. The device of claim 10, wherein the processor includes a neural network (NN) accelerator, and the data includes a data structure of input data associated with the workload of the NN application.

18. A method for managing a cache located on a processor, the method comprising: The controller circuitry coupled to the processor receives a request from the application to provide direct memory access to data stored at an external memory address in external memory coupled to the controller circuitry. The controller circuitry copies the data from the external memory address to a first internal memory address within the processor's internal memory. as well as The controller circuitry updates the first cache line located in the cache and associated with the external memory address to include a reference to the first internal memory address. as well as The controller circuitry provides the data from the internal memory to the application.

19. The method of claim 18, wherein copying the data from the external memory address to a first internal memory address within the processor's internal memory by the controller circuitry is performed in response to determining that the external memory address is not registered in the cache.

20. The method of claim 19, further comprising: In response to determining that the external memory address is registered in the cache: Identify a second cache line within the cache that is associated with the external memory address, the second cache line containing a reference to a second internal memory address; as well as The data is copied from the second internal memory address to a third internal memory address within the internal memory.

Images