Memory access

Abstract

A method for managing memory accesses for implementing at least one layer of a convolutional neural network is provided. The method includes predicting, based on characteristics of the convolutional neural network, an access process related to a portion of memory. In response to the prediction, the method includes performing, prior to the predicted access process, an operation for obtaining a memory address translation corresponding to the portion of memory and storing it in a storage device. An apparatus is provided that includes at least one processor and a storage device. The apparatus is configured to predict an access process related to a portion of memory external to the processor. In response to the prediction, the apparatus is configured to obtain, prior to the predicted access process, a memory address translation corresponding to the portion of memory and store it in the storage device.

Description

Technical Field

[0001] This disclosure relates to methods and apparatus for managing data processing. Background Technology

[0002] Processors used to implement convolutional neural networks (such as neural processing units (NPUs), central processing units (CPUs), graphics processing units (GPUs), digital signal processors (DSPs), and coprocessors) have on-board memory, typically in the form of static random access memory (SRAM). When implementing convolutional neural networks using such processors, it may not be possible to keep all data (e.g., including input data, output data, and data corresponding to the operations involved in the convolutional neural network) in processor memory. At least some data can be stored in memory outside the processor, which can include volatile and non-volatile memory. The processor can access external memory when executing a convolutional neural network. The processor may have associated storage devices (e.g., caches, also known as translation back buffers) that store memory address translations corresponding to locations in memory. Memory address translations are used to translate memory addresses used by the processor into memory addresses that identify the portion of memory storing the associated data. Using virtual memory can make dispersed physical memory locations appear contiguous. They can also allow processes running on the same virtual address to access different versions of data stored in different portions of physical memory. Due to size limitations, the storage device may not be able to store memory address translations for all memory address locations. Size limitations may be imposed by desired attributes such as required access speed, area, or cost. When a memory location needs to be accessed and the associated memory address translation is not stored in the storage device, retrieving the memory address translation makes access to the memory location possible. Retrieval may include accessing a larger, auxiliary storage device, such as a large cache, or may involve accessing a main page table that includes memory address translations for a larger system.

[0003] The goal is to improve the efficiency of memory access when implementing convolutional neural networks. Summary of the Invention

[0004] According to a first aspect of this disclosure, a method for managing memory access is provided, the method comprising implementing at least one layer of a convolutional neural network on input data using a processor to generate output data, the method comprising at least: predicting an access process associated with a portion of memory outside the processor based on at least one or more features of the convolutional neural network; and, in response to the prediction, performing an operation to obtain a memory address translation corresponding to the portion of memory and storing the memory address translation in a storage device prior to the predicted access process.

[0005] According to a second aspect of this disclosure, a non-transitory computer-readable storage medium is provided, the non-transitory computer-readable storage medium including instructions that, when executed by a processor, cause the processor to at least: predict an access process associated with a portion of memory outside the processor based on at least one or more features of a convolutional neural network, for implementing at least one layer of the convolutional neural network on input data to generate output data; and, in response to the prediction, perform operations for obtaining a memory address translation corresponding to the portion of memory and storing the memory address translation in a storage device prior to the predicted access process.

[0006] According to a third aspect of this disclosure, an apparatus is provided, comprising: at least one processor for implementing at least one layer of a convolutional neural network on input data to generate output data; and a storage means for storing a plurality of memory address translations, wherein the apparatus is configured to: predict an access process associated with a portion of memory outside the processor based on at least one or more features of the convolutional neural network; and, in response to the prediction, perform operations for obtaining memory address translations corresponding to the portions of memory and storing the memory address translations in the storage means prior to the predicted access process. Attached Figure Description

[0007] Other features and advantages will become apparent from the following description of preferred embodiments (given by way of example only) with reference to the accompanying drawings, in which the same reference numerals are used to denote the same features.

[0008] Figure 1 This is a schematic diagram illustrating the apparatus according to the example;

[0009] Figure 2 This is a flowchart illustrating the method based on the example;

[0010] Figure 3 This is a schematic diagram of an image to be processed using a convolutional neural network, based on an example.

[0011] Figure 4A This is a schematic diagram illustrating a simplified implementation of a convolutional neural network based on an example;

[0012] Figure 4B This is a schematic diagram illustrating convolution processing based on an example;

[0013] Figure 5 This is a schematic diagram illustrating how a convolutional neural network is implemented based on an example;

[0014] Figure 6 It is a schematic diagram of a system including features, based on an example;

[0015] Figure 7It is a schematic diagram of a system including features, based on an example; and

[0016] Figure 8 This is a schematic diagram of a non-transitory computer-readable storage medium based on an example. Specific Implementation

[0017] The details of the systems and methods described below with reference to the accompanying drawings will become apparent. In this description, for illustrative purposes, numerous specific details of certain examples are set forth. References to "example" or similar language in the specification mean that a feature, structure, or characteristic described in connection with that example is included in at least one example, but not necessarily in other examples. It should be further noted that some examples are described schematically, in which certain features are omitted and / or necessaryly simplified to facilitate explanation and understanding of the concepts behind the examples.

[0018] Certain examples described herein provide a method for storing memory address translations corresponding to portions of memory that are subsequently accessed in the storage device when at least one layer of a convolutional neural network is implemented, after which the processor accesses the portions of memory. Memory access patterns can be determined based on one or more features of the convolutional neural network implemented by the processor. In some cases, portions of memory to be accessed sequentially may be scattered throughout the memory, so determining the addresses of these portions of memory before they can be accessed can be time-consuming. Predicting the portions of memory to be accessed and obtaining the memory address translations corresponding to those portions of memory before accessing them can prevent read or write operations from being delayed or postponed in determining the addresses of the relevant memory portions. Delaying or postponing read or write operations is undesirable because it reduces processing throughput, thereby reducing processing efficiency and wasting power. Prediction can also refer to deterministic processing, such as deterministic processing based on at least one feature of the convolutional neural network. Implementing at least one layer of a convolutional neural network may include a training phase (training the convolutional neural network on sample data) and an inference phase (using the convolutional neural network to process data and perform inference). The examples described herein are particularly useful when the convolutional neural network reads or writes data out of order compared to how the data is stored. In this scenario, frequent distributed memory accesses will be performed.

[0019] When implementing at least one layer of a convolutional neural network, memory accesses can be performed on various types of data. The examples described herein are applicable to accessing portions of memory when reading and / or writing input data, output data, data related to the convolutional neural network (e.g., data representing the weights of the kernel in at least one layer of the convolutional neural network), and / or bias data. Input data can refer to data input to the first layer of the convolutional neural network and data input to each subsequent layer of the convolutional neural network. Input data can include sensor data derived from one or more sensors (e.g., image sensors, sound sensors), and other suitable forms of sensor data as described below. Input data can also include input feature maps generated by performing operations on sensor data. In some examples, the data input to the first layer of the convolutional neural network can be sensor data, and the data input to subsequent layers of the convolutional neural network can be referred to as input feature maps. Output data can refer to data output from the last layer of the convolutional neural network and data output when performing convolutions at each intermediate layer. Data output when convolutional layers are applied to input data or input feature maps from sensors can be referred to as one or more output feature maps. Data can be compressed or uncompressed.

[0020] Some of the examples described herein determine when it is appropriate to fetch and store memory address translations that will subsequently be accessed, based on the size of the storage device used to store them. Furthermore, memory address translations can be removed from the storage device. In this case, determining when it is appropriate to fetch and store memory address translations in the storage device can be based on a retention and / or eviction policy for memory address translations within the storage device. In some implementations, storing only the most relevant memory address translations in the storage device may have a high level of precision. This can allow for a reduction in the size of the storage device, and consequently, a reduction in the time spent searching for memory address translations within the storage device. Memory can refer to both volatile and non-volatile memory types. Memory address translations can also be referred to as virtual memory address translations or simply virtual address translations.

[0021] Figure 1This is a schematic diagram of device 100 according to an example. Device 100 can be included in a computing device. For example, device 100 can be part of a handheld computing device such as a smartphone or tablet. In these examples, device 100 can be used to implement a convolutional neural network or at least one layer of a convolutional neural network on data acquired by the smartphone or tablet. The data that the smartphone or tablet can acquire depends on the device's capabilities and the sensors included in the device. This data can include image data, audio data, telemetry data, accelerometer data, GPS data, magnetometer data, light sensor data, fingerprint reader data, any other data that can be collected by sensors included in or local to the computing device, or any data that can be received in a local area network or wide area network via a wired or wireless communication interface included in the computing device. Alternatively, device 100 can be used in another type of computing device, such as a computing device used in an autonomous vehicle, or in robotic applications where convolutional neural networks are used to process control data, sensor feedback data, or provide any other suitable information processing.

[0022] Apparatus 100 includes at least one processor 110 and a storage device 120 for storing multiple memory address translations. The memory address translations can be between virtual memory addresses and physical memory addresses. For example, in the case of using virtual memory. The memory address translations can correspond to volatile memory addresses or non-volatile memory addresses. Alternatively, the memory associated with the memory translations can be independent of at least one non-transitory computer-readable storage medium 130. In other examples, the memory address translations can be between different virtual memory addresses. This could be a case where a convolutional neural network is implemented using a virtual machine. At least one processor 110 can include a neural processing unit (NPU), which may also be referred to as an AI accelerator or neural network processing unit. In other examples, at least one processor 110 can include another form of processor configured to be implemented at least at the layers of a convolutional neural network. For example, at least one processor 110 can include a CPU, GPU, DSP, coprocessor, or any other suitable processor. The storage device 120 may also be referred to as a cache or translation back buffer. The storage device 120 can be part of a memory management unit (MMU) that handles memory address translations. In other examples, the storage device 120 can be independent of the MMU but can communicate with it. In some examples, storage device 120 is a dedicated translation backup buffer serving at least processor 110, providing memory address translation used by processor 110 to perform operations. Device 100 may include, or may be part of, a device that includes further processors and storage devices, wherein storage device 120 serves at least one processor 110 but not additional processors.

[0023] The apparatus is configured to predict an access process associated with a portion of at least one external memory based on at least one feature of a convolutional neural network; and, in response to the prediction, to perform operations for obtaining a memory address translation corresponding to the portion of memory and storing the memory address translation in storage device 120 prior to the predicted access process. The portion of at least one external memory can be any suitable form of memory, including volatile memory (e.g., RAM) and non-volatile memory (e.g., ROM). In some examples, device 100 includes a portion of this memory. For example, device 100 may include memory for storing at least some of input data, output data, and data representing at least a portion of the convolutional neural network. In other examples, device 100 may be communicatively coupled to memory for storing at least some of the input data, output data, and data representing at least a portion of the convolutional neural network. The type of memory used can be determined by several factors, including: the location where the input data is stored, the size of the input data that can be determined whether it can be stored in the volatile memory of device 100, the location where the output data is to be written, and the location where the data corresponding to the convolutional neural network is stored. Memory accesses performed by processor 110 while implementing at least one layer of a convolutional neural network can be deterministic, but these accesses are scattered throughout memory. The time spent performing predicted access procedures can be reduced by storing the memory address translations of the subsequently accessed portion of memory in storage device 120 and then performing the predicted access procedure associated with that portion of memory. When at least one processor 110 attempts to access a portion of memory, an access translation back buffer or storage device 120 determines the address of that portion of memory. If the memory address translation for that portion of memory is not in the translation back buffer 120, processor 110 cannot directly access that portion of memory. Instead, the memory address translation is fetched, thereby increasing the time spent by the processor reading data from and / or writing data to that portion of memory. Fetching memory address translations may include accessing another storage device that includes the memory address translations, or performing a page traversal of a home table that includes memory address translations on the system including device 100. Prefetching memory address translations can improve the efficiency of device 100 when implementing at least one layer of a convolutional neural network. When implementing convolutional neural networks, a large number of memory accesses may be performed, and these memory accesses are scattered throughout the memory. Therefore, prefetching memory address translation can improve the efficiency of the system.

[0024] Figure 2This is a flowchart of a method 200 for managing memory access, which uses a processor to implement at least one layer of a convolutional neural network on input data to generate output data. At block 210, the method includes predicting an access process associated with a portion of memory outside the processor based on at least one or more features of the convolutional neural network. At block 220, the method includes, in response to the prediction, performing an operation of obtaining a memory address translation corresponding to the portion of memory and storing the memory address translation in a storage device prior to the predicted access process. Obtaining the memory address translation and storing it in the storage device prior to the predicted access process can reduce the time spent on read and / or write operations (which use that portion of memory). Because memory access is deterministic when implementing a convolutional neural network, the access process associated with a portion of memory can be predicted based on at least one or more features of the convolutional neural network. In examples where portions of memory to be accessed when implementing at least one layer of a convolutional neural network are scattered throughout the entire memory, prefetching memory address translations for these portions of memory may be particularly useful.

[0025] Now refer to Figure 3 This discussion covers specific examples of the application of this disclosure. Figure 3 An image 300 represented by image data is schematically shown. The image data may include multiple pixel intensity values ​​representing multiple pixel locations. The pixel intensity values ​​representing image 300 may be stored in memory in contiguous blocks. For example, the pixel intensity value for each pixel location may be stored contiguously in memory along with pixel intensity values ​​representing horizontally adjacent pixel locations. The memory may include any suitable memory, including random access memory (RAM), read-only memory (ROM), video random access memory (VRAM), synchronous dynamic random access memory (SDRAM), or any other suitable type of memory.

[0026] exist Figure 3 In the example shown, image 300 was partially processed using a neural network to detect and classify objects in the image. The results of object detection and object classification are as follows: Figure 3As shown, the dashed boxes mark the regions in the image where people are detected. These marked regions in the image can then be processed using a convolutional neural network to identify people in the image. In this example, a convolutional neural network is implemented on the image data representing the marked regions to identify people in these regions. When at least one layer of the convolutional neural network is applied to the image data representing region 310, the first portion of the data representing pixel position 310a can be accessed first, followed by the portions of the data representing pixel positions along the same row. The portions of the data representing horizontally adjacent pixel positions can be stored contiguously, resulting in very little processing involved when processing those portions of the data representing pixel positions between 310a and 310b. However, the portion of the data representing the next row, starting from pixel position 310c, may not be stored contiguously in physical memory as the portion representing pixel position 310b. Therefore, an extraction operation can be performed to extract the memory address translation corresponding to the portion of memory containing the data representing pixel position 310c. Predicting the access process associated with a portion of memory, including image data representing pixel position 310c, and obtaining and storing the memory address translation corresponding to that portion of memory in the storage device before the predicted access process, prevents the need to wait to determine the memory address translation before accessing the associated portion of memory. In some examples, the prediction of the access process associated with a portion of memory outside the processor is based on at least one feature of the input data. Convolutional neural networks can be used to process various types of input data. Features of the input data can also influence access patterns and can be used to predict the access process associated with a portion of memory. In some examples, at least one feature of the input data includes the storage configuration of the input data in memory outside the processor. That is, the features of the input data can include how the input data is stored, such as whether the input data is stored contiguously, whether the input data is stored in a specific pattern, whether the input data is stored in blocks that are not contiguous to each other, and other possible arrangements. Other features of the input data that can be used to predict the access process can include what type of data the input data is, and other possible features that can be used to predict the portion of memory to be accessed. The method can also be used for cases where image data is stored according to different rotations (in which the image data will be used by the convolutional neural network). For example, the image data is stored row-majorly but read column-majorly for processing. This also applies to output data, for example, where the output data is image data, and the output data is generated in column order but stored in row order.

[0027] Figure 4A This is a schematic diagram illustrating the implementation of a simple convolutional neural network layer to generate output data. This simple convolutional neural network has a single feature map associated with the input data. Image data will be discussed in the context of this diagram. Figure 4AThe examples shown are for illustrative purposes only; however, it should be understood that the features described herein can also be applied to other data types (such as audio data).

[0028] Figure 4A Input data 410 is shown, comprising multiple portions 410a, 410b, and 410c. For example, input data 410 may be image data including multiple pixel intensity values ​​representing multiple pixel locations. Each portion 410a-410c may represent a different color channel. Input data 410a may include pixel intensity values ​​corresponding to the green channel, input data 410b may include pixel intensity values ​​corresponding to the red channel, and input data 410c may include pixel intensity values ​​corresponding to the blue channel. The convolutional neural network may include multiple filters 420 representing a core. The core is used to identify features of a given type in the input data 410. Each filter 420a, 420b, and 420c may correspond to its respective color channel and may be represented by a matrix. Filters 420 may be used to process the input data 410 to detect features in the image. When implementing layers of a convolutional neural network, a convolution is performed between a first portion 41a of the input data 410 and a filter 420a. This convolution is a process of adding each element in the input data 410a that overlaps with an element in the filter 420a to its local neighbor, with this addition weighted by the individual elements of the filter 420a. The filter is then shifted to a second portion of the input data 410a, and this process is repeated. The increment of the shift of the filter 420a can be called the stride. This process is performed until the filter 420a is applied to all of the input data 410a according to its size and stride, resulting in output data 430a. This process is performed for each color channel, resulting in output data 430 comprising multiple portions 430a, 430b, and 430c corresponding to each color channel. In practice, filters 420a, 420b, and 420c can be considered as a single core, with a single convolution performed between the core and all layers of the input data. In this case, the results across layers can be combined to generate a single output feature map, i.e., an output feature map with a single component. The output data 430 can then be further processed or modified.

[0029] Figure 4B The convolution process is illustrated in more detail. Figure 4B A filter 420a applied to input data 410a is shown. Figure 4BFilter 420a is applied to input data 410a with a stride of 1, resulting in output data 430a having a smaller dimension than input data 410a. In other examples, the dimension between input data 410a and output data 43a can be maintained by applying padding to input data 410a. The size of output data 430a can also be reduced by pooling. At each position of filter 420a, the values ​​in input data 410a that overlap with the filter are summed according to the weights of filter 420a. For example, at position 440, the output data generated by the convolution includes the value -3 shown at position 435 in output data 430a. This value is determined by performing the following operations:

[0030] [(-1)*1]+[(0)*1]+[(1)*1]+[(0)*5]+[(1)*1]+[(0)*2]+[(-1)*6]+[(0)*2]+[(1)*2].

[0031] refer to Figure 4B The example discussed only shows a single channel of input data and the core used. As mentioned above, in practice, filters 420a, 420b, and 420c can be used as a single core. In this case, the results of the above calculations will be accumulated together with similar calculations for the other channels.

[0032] When implementing at least one layer of a convolutional neural network, the operations performed on the input data may affect the order in which different parts of the memory are accessed. In the example, implementing at least one layer of a convolutional neural network includes performing multiple operations on the input data, each operation being performed on at least a portion of a plurality of parts of the input data. Compare this to... Figure 4A and Figure 4BAs in the example, a portion of the input data may be associated with portion 410a. Multiple operations may involve each convolution of filter 420a with a portion of the input data 410a. Memory accesses performed in this case for implementing at least one layer of the convolutional neural network can be determined by the amount and / or order of generating output data and the amount and / or order of accessing input data. Memory accesses can also be determined by the amount and / or order of accessing the core representing the convolutional neural network. One or more features of the convolutional neural network may include the size of at least one of a plurality of portions. The size of the portion of input data operated on by each operation may depend on the size of the filter 420a being used. In some examples, the portion of input data operated on by an operation may include multiple channels. In this case, the size of the data portion operated on by each operation may depend on the size of the core, where the core includes a corresponding plurality of filters. Therefore, one or more features of the convolutional neural network may include the size or dimension of at least one core used in the convolutional neural network. The size of core 420 may affect the size of output data 430. One or more features of the convolutional neural network may also include the stride of core 420, or, in examples where filters are implemented separately, at least one filter of these filters used for implementing at least one layer of the convolutional neural network. One or more core features can include which parts of the input data are included in each part. This can be determined by several factors, including: overlap between successive operations, whether padding is added, whether bias is used, activation function, whether pooling is used, and other factors such as the size of the included parts.

[0033] In some examples, one or more features of a convolutional neural network include the order in which portions of the input data are accessed to implement at least one layer of the convolutional neural network. The order in which the portions of the input data are accessed may relate to the order in which operations for generating output data are performed on the input data to implement at least one layer of the convolutional neural network. When the input data is too large to be stored in the processor's on-board storage, the input data can be streamed when implementing at least one layer of the convolutional neural network. That is, the input data can be segmented, and at least one layer of the convolutional neural network can be implemented on each segment separately, with the final results stored sequentially or in combination. The order in which the portions of the input data are accessed may depend on how the input data is segmented. In one implementation, the input data can be streamed across different channels; for example, if the input data is image data, the image data can be divided into multiple channels. In other implementations, the input data can be streamed across groups of channels. Figure 4AThe example shown illustrates input data 410 divided into three channels, but in some applications, the data can be divided into more parts, such as sixty-four parts. In this example, streaming the input data could involve operating on the first thirty-two parts first, and then on the last thirty-two parts. In other implementations, the input data can be divided into tiles. For example, in the case where the input data represents an image, the image can be divided into multiple parts that at least partially overlap. The convolutional neural network can be implemented separately on each of the multiple parts and then combined at the end. In some examples, the convolutional neural network is implemented on input data stored as parts of a larger dataset.

[0034] In the example, one or more features of a convolutional neural network include the order in which the data corresponding to the convolutional neural network is accessed.

[0035] In the example, one or more features of the convolutional neural network include the order in which the portions of the output data are generated when at least one layer of the convolutional neural network is implemented. When generating output data or partial output data according to the convolutional neural network, the output data can be generated based on convolutions performed between the filter or kernel and the input data. The output data can be stored according to a predetermined storage scheme, wherein different portions of the output data will be stored together with other predetermined portions of the output data. Predictions for the portion of memory used to store the output data that will be subsequently accessed can be based on the order in which the output data was generated.

[0036] Figure 5 An implementation of a convolutional neural network 500 is schematically illustrated, which includes multiple feature maps in the first layer of the convolutional neural network. As mentioned above, when implementing at least one layer of a convolutional neural network, it may be impossible to store all input data and the data corresponding to the cores in the processor's on-board memory. This can be overcome in several ways, including input streaming, weight streaming, or a combination of both. In the example where the input data cannot be entirely stored in the processor's on-board memory, the input data can be divided into multiple planes and extracted one by one, which can be referred to as input streaming. Figure 5Input data 510 is shown, which is divided into multiple parts 510a, 510b, 510c, 510d to be sent sequentially to the processor. If it is not possible to store the data corresponding to all the cores of the convolutional neural network in the processor's memory, weight stream transfer can be used. Weight stream transfer is the process of sending data corresponding to the cores (e.g., the weights that make up the cores) to the processor in parts. In this example, one or more features of the convolutional neural network may include portions of data representing at least a portion of the convolutional neural network that will be accessed in order to implement at least one layer of the convolutional neural network. The data representing at least a portion of the convolutional neural network may include weights, bias data, and other data representing the convolutional neural network. Core 520 is divided into multiple parts, and these parts are extracted one by one. Figure 5 A core set 520 representing feature maps is shown, included in the first layer of a convolutional neural network. Each feature map consists of a core representation that can be divided into multiple parts, such as features... Figure 1 It is represented by a core K1 that is divided into multiple parts 522a, 522b, 522c, and 522d. Each feature map can be represented as a single core; however, in... Figure 5 In the example shown, the core is divided into parts corresponding to the dimensions of the input data. Part 522a of core K1 can be applied to input data 500a, part 522b of core K2 can be applied to input data 500b, part 522c of core K3 can be applied to input data 500c, and part 522d of core K4 can be applied to input data 500d. Figure 5 In the example shown, the portions of cores K1 to K4 in region 530, indicated by dashed lines, can be the first set of core portions extracted and used by the processor during weight stream propagation. Subsequently, the next set of extracted core portions can be portions 522b, 524b, 524c, and 524d. Alternatively, the core portions can be extracted in a different order; for example, portions 522a, 522b, 522c, and 522d can be the first set of core portions extracted during weight stream propagation. Subsequent sets of core portions can include 524a, 524b, 524c, and 524d. Weight stream propagation can be performed in this manner until all convolutions are completed.

[0037] The output data (representing the corresponding feature maps) generated by performing convolutions between cores K1 to K4, along with the input data 510, can be used as input to another layer of the convolutional neural network. If weight stream propagation is used to implement at least one layer of the convolutional neural network, then for the next layer of the convolutional neural network, the order in which the data is written may be unordered compared to the order in which the data is read. This can be achieved through... Figure 5As seen in the diagram, output data 540, generated by performing convolutions between portions 522a, 524a, 526a, and 528a and the input data, is shown below the input data 510 and the core set 520. When using the output data 540 as input to the next convolutional layer, the output data 540 can be streamed into an input stream according to the plane indicated by the dashed line at 540. Therefore, it is desirable to store the data included in each plane consecutively. When the first portion of data 542a is generated, it can be written to the first portion of memory. The first portion of memory can be the first portion of a memory block used to store data corresponding to horizontal plane OFM 1. An access process associated with the memory portion used to store the second portion 544a of data is predicted, and in response to this prediction, at the beginning of the next memory block used to store data corresponding to the next horizontal plane OFM 2, an operation of obtaining and storing the memory address translation corresponding to the portion of memory can be performed. This process is repeated for the next two portions of output data 546a and 548a. Then, this process is repeated for the next set K1 to K4 of the core portion of the weight stream. As a result of this operation, the data portions 542a, 544a, 546a, and 548a are not stored contiguously. Instead, they are stored in memory in an order that corresponds to the order in which they are read when used as input to the next layer of the convolutional neural network.

[0038] In an alternative arrangement, the output data generated when convolution is performed between the core and the input data can be stored in memory in the same order as it was generated. To efficiently transmit this data as an input stream to another layer of the convolutional neural network, the memory access management scheme described above can be used for this input. A first portion of the memory can be accessed to read the first portion of the input data. The access process associated with a second portion of memory that is not adjacent to the first portion can be predicted. In response to this prediction, an operation to retrieve the second portion of memory and store it in the storage device can be performed before the predicted access process associated with the second portion of memory. This allows for fast access to the second portion of memory and reading the data therein without adding latency while waiting for the memory address translation.

[0039] In the example, the operation of obtaining the memory address translation corresponding to a portion of memory and storing it in the storage device is a write operation, wherein the write operation is a null write operation, such that no data is written to that portion of memory. A null write operation can also be referred to as a pseudo-write operation. A pseudo-write operation can be a write transfer in which all byte channel strobes are low, such that no data is written to memory. A null write operation can be performed by the processor, for example by a dedicated thread within the processor to perform null write operations to fill the storage device with memory address translations. Null write operations can be used because they are interoperable with the current memory management unit. Using null write operations prevents the need to reconfigure other hardware and / or software to be compatible with the memory access management scheme described herein. Null write operations can be used regardless of the type of predicted access procedure. For example, when implementing at least one layer of a convolutional neural network, a null write operation can be used if the predicted access procedure is a read procedure for reading data stored in the corresponding portion of memory, or if the predicted access procedure is a write procedure for writing data to a portion of memory.

[0040] In other examples, the operation of obtaining the memory address translation corresponding to a portion of memory and storing it in the storage device is a read operation of reading that portion of memory. The read operation can be performed by a processor. The processor can implement threads that perform read operations on portions of memory subsequently accessed during the implementation of at least one layer of the convolutional neural network. Where the predicted access process is a write process that writes data to the corresponding portion of memory, the read operation for obtaining the memory address translation and storing it in the storage device can be performed at any appropriate time before that portion of memory is to be accessed. Where the portion of memory stores data (e.g., input data) that the convolutional neural network will use, the read operation can be timed so that each time a command is sent, the data stored in that portion of memory is received by the processor when it is to be used to implement at least one layer of the convolutional neural network.

[0041] In some implementations, the operation of obtaining the memory address translation corresponding to a portion of the memory and storing it in the storage device is a custom operation for obtaining the memory address translation and storing it in the storage device. This custom operation allows the operation to discard redundant communication protocols and thus improves operational efficiency. Examples of custom operations used to obtain memory address translations and store them in the memory device include the "StashTranslation" transaction provided in the Advanced Microcontroller Bus Architecture (AMBA) 5 specification, "AMBAAXI and ACE Protocol Specification (AXI3, AXI4, AXI5, ACE and ACE 5) - ARM IHI 0022F.b (ID122117)," released in 2017. As discussed in the aforementioned references, the StashTranslation transaction can be used to instruct the memory management unit to obtain a translation of the address provided with the transaction. The StashTranslation transaction has no associated data transfer. Address and control information are provided on the Address Write (AW) channel. A single response is provided on the Write Response (BRESP) channel. A response is provided after the address is accepted. In other cases, different custom operations can be used. In some examples, custom operations are also capable of performing other functions.

[0042] In some implementations, the method includes removing memory address translations from the memory device based on at least one of a prediction of a further access process associated with a portion of the memory and a prediction of the duration of performing a further operation to obtain a memory address translation and store it in the memory device. In some cases, memory address translations can be removed from the memory device to free up space for further memory address translations and to prevent the memory device from becoming too large. If it is determined that a portion of the corresponding memory will be subsequently accessed, it may not be possible to remove the memory address translations from the memory device. For example, if the time taken to access the portion of memory and perform the operation to obtain the corresponding memory address translation and store it in the memory device is longer than the time before the next access process, it may not be possible to remove the memory address translations from the memory device. However, if it is determined that a portion of the memory has not been accessed for a considerable period of time, its corresponding memory address translation can be temporarily removed from the memory.

[0043] Figure 6 This is a simplified schematic diagram illustrating an implementation of the system 600 that demonstrates the features of this disclosure. Figure 6 A processor 610 for implementing at least one layer of a convolutional neural network is shown. The processor 610 is communicatively coupled to a storage device 620 for storing memory address translations. Figure 6 The storage device 620 shown is a micro-translation back buffer that stores memory address translations for use by the processor 610 at least when implementing at least one layer of a convolutional neural network. A system memory management unit 630 is shown, which includes a further storage device 640, such as a translation back buffer. Storage device 640 may be larger than storage device 620 and may store a larger number of memory address translations. For example, storage device 640 may store memory address translations corresponding to memory address translations used by multiple processors and / or processes. That is, the system memory management unit may serve multiple processors. Using a first, smaller storage device 620 for a processor allows for faster determination of memory address translations because the number of entries in storage device 620 that can be scanned to find memory address translations is less than the number of entries in the larger storage device 640. A memory access controller 650 is provided connected to memory 660. In the example, the processor may attempt to access a portion of memory to implement at least one layer of a convolutional neural network. The micro-translation back buffer 620 is accessed to find memory address translations corresponding to that portion of memory. If the micro-translation back buffer 620 does not contain the relevant memory address translations, then the translation back buffer 640 can be scanned to determine the relevant memory address translations. If the translation back buffer 640 does not contain the relevant memory address translations, then, for example at memory 660, a page table traversal is performed on the main page table of system 600, which includes the memory address translations. This process increases the time spent determining the relevant memory address translations. It is desirable to ensure that the relevant memory address translations are stored in the micro-translation back buffer 620 before the processor 610 needs them to implement at least one layer of the convolutional neural network.

[0044] Figure 7 The diagram schematically illustrates a processor 710, a storage device 720 including a first plurality of memory address translations 725, a memory management unit 730 including a second plurality of memory address translations 735, and a main page table 740 including a third plurality of memory address translations 745. In the example, a portion of the memory is a first portion of the memory, memory address translation 725a is a translation of a first memory address, and the method includes performing a predicted access process associated with the first portion of the memory. The method also includes predicting a further access process associated with a second portion of memory outside the processor 710, based at least on one or more features of a convolutional neural network. In response to performing the predicted access process with respect to the first portion of the memory, the method includes: prior to the predicted further access process associated with the second portion of the memory, performing an operation of obtaining a second memory address translation 725b corresponding to the second portion of the memory and storing the second memory address translation 725b in the storage device. Figure 7As shown in the example, the second memory address translation can be retrieved from any of the second plurality of memory address translations 735 and the third plurality of memory addresses 745. Retrieving and storing the second memory address translation 725b in response to a predicted access process associated with the first portion of the memory can prevent the memory address translation 725b from being stored for an extended period before it is needed, and can also allow the second memory address translation 725b to be stored in the storage device 720 before it is required.

[0045] Storage device 720 may be of fixed size, thus entries in storage device 720 can be removed to free up space for future memory address translations to be used. In the example, after performing a predicted further access process associated with the second portion of memory, the method includes removing the first memory address translation from storage device 720. Since the second portion of memory is accessed after the first portion of memory, accessing the second portion of memory can serve as an acceptable indicator for removing the first memory address translation. In this way, space in the storage device can be freed up for further operations to retrieve and store memory address translations therein. Instead of individually deleting data and writing it to storage device 720, entries can be overwritten as new entries are generated. Removing the first memory address translation from the storage device may include overwriting the first memory address translation with a third memory address translation corresponding to a third portion of memory outside the processor and subsequently accessed.

[0046] In implementations of this disclosure, the method may include determining, at least based on the size of the storage device, when to perform operations for retrieving and storing memory address translations in the storage device. The size of the storage device may determine the number of memory address translations that can be stored therein. The number of entries that can be accommodated in the storage device may depend on the size of the pages used to store the memory address translations and the size of the storage device. In some examples, the number of memory address translations (which may be referred to as entries) that can be stored in the storage device may be known. In other examples, the maximum number of entries in the storage device may be determined by performing a batch of accesses that utilize the memory address translations stored in the storage device. The total number of entries in the storage device may be determined by timing the access time for each access in this batch of accesses. Memory address translations should not be removed or overwritten in the storage device before they are used to access their respective portions of memory. Operations for retrieving and storing memory address translations may be performed based on the determination of when to perform operations for retrieving and storing memory address translations in the storage device. This may allow memory address translations to be stored in the storage device at the appropriate time, enabling them to be stored in the storage device without evicting another memory address translation that was used earlier. Determining when to perform the operation of retrieving and storing memory address translations in the storage device can also be based on the rate at which memory address translations stored in the storage device are removed. Memory address translations can be removed from the storage device at appropriate times. This method may involve predicting when there will be available space to store memory address translations in the storage device and performing the operation at that time. In some examples, memory address translations may also be maintained in the storage device after the corresponding memory portion has been subsequently accessed, if it is expected that the corresponding memory portion will be accessed again soon.

[0047] The examples described herein can also be used when the input and / or output data are compressed. Some data compression methods are deterministic, so prediction of the access process can be considered. In this case, the memory address translation for acquisition and storage can correspond to the starting address of the compressed block to be read or written.

[0048] Figure 8A non-transitory computer-readable storage medium 800 according to an example is illustrated. The non-transitory computer-readable storage medium 800 includes computer-readable instructions shown in blocks 810 and 820. The instructions at block 810, when executed by processor 830, cause processor 830 to at least: predict an access process associated with a portion of memory outside the processor based on at least one or more features of a convolutional neural network, for implementing at least one layer of the convolutional neural network on input data to generate output data. The processor implementing at least one layer of the convolutional neural network may be processor 800. Alternatively, the processor implementing at least one layer of the convolutional neural network may be a different processor than processor 830. For example, processor 830 may be a processor for managing memory address translation. The instructions at block 830, when executed by processor 830, cause the processor to at least: in response to the prediction, perform an operation for obtaining a memory address translation 840 corresponding to a portion of memory and storing the memory address translation in storage device 850 prior to the predicted access process. This may be the case, for example, if the processor is any of a CPU, GPU, or DSP. Alternatively, in the case of an NPU, predicting the portion of memory to be accessed can be performed by a state machine.

[0049] The examples above are to be understood as illustrative examples of this disclosure. Further examples are contemplated. For example, memory address translation can be an address translation between virtual memory addresses and other virtual memory addresses. This may be the case, for example, when implementing a convolutional neural network on one or more virtual machines. It should be understood that any feature described with respect to any example can be used alone or in combination with other features described, and can also be used in combination with one or more features of any other example, or in any combination with any other example. Furthermore, equivalents and modifications not described above may be employed without departing from the scope of this disclosure as defined by the appended claims.

Claims

1. A method for managing memory access, the method comprising implementing at least one convolutional layer of a convolutional neural network on input feature map data using a processor to generate output feature map data, the method comprising: Based on at least one or more features of the convolutional neural network, predict the access process that is partially related to the memory outside the processor; as well as In response to the prediction, prior to the predicted access process, operations are performed to obtain a memory address translation corresponding to a portion of the memory and store the memory address translation in the storage device. The one or more features of the convolutional neural network include the order in which portions of the input feature map data are accessed to implement the at least one convolutional layer of the convolutional neural network.

2. The method according to claim 1, wherein, Implementing the at least one convolutional layer of the convolutional neural network includes: performing a plurality of operations on the input feature map data, and each operation is performed on at least a portion of a plurality of parts of the input feature map data, and the one or more features of the convolutional neural network include any one of the following: The size of at least one of the plurality of parts; and Which parts of the input feature map data are included in each part of the input feature map data? 3. The method according to claim 1, wherein, The prediction of the access process related to a portion of the memory outside the processor is based on at least one feature of the input feature map data, wherein the at least one feature of the input feature map data includes the storage configuration of the input feature map data in the memory outside the processor.

4. The method according to claim 1, wherein, The operation of obtaining the memory address translation corresponding to a portion of the memory and storing the memory address translation in the storage device is any one of the following: A write operation, wherein the write operation is a null write operation, such that no data is written to a portion of the memory; A read operation is used to read a portion of the memory; as well as A custom operation is used to obtain the memory address translation and store the memory address translation in the storage device.

5. The method according to claim 1, wherein, The method includes removing the memory address translation from the storage device according to at least one of the following: Predictions related to further access processes to portions of the memory; as well as Prediction of the duration of further operations performed to obtain the memory address translation and store the memory address translation in the storage device.

6. The method according to claim 1, wherein, The portion of the memory is a first portion of the memory, the memory address translation is a first memory address translation, and the method includes: Execute the predicted access procedure associated with the first portion of the memory; Based on at least one or more features of the convolutional neural network, predict further access processes related to a second portion of memory external to the processor; and In response to the execution of the predicted access process associated with the first portion of the memory, prior to the predicted further access process associated with the second portion of the memory, an operation is performed to obtain a second memory address translation corresponding to the second portion of the memory and store the second memory address translation in the storage device.

7. The method according to claim 6, wherein, After performing the predicted further access process associated with the second portion of the memory, the method includes removing the first memory address translation from the memory device, wherein removing the first memory address translation from the memory device includes overwriting the first memory address translation with a third memory address translation corresponding to a third portion of memory outside the processor and subsequently accessed.

8. The method according to claim 1, comprising: The timing of the operation to acquire and store the memory address translations in the storage device is determined at least based on the size of the storage device, wherein the operation to acquire and store the memory address translations is performed based on the determination of when to acquire and store the memory address translations in the storage device, and wherein the determination of when to acquire and store the memory address translations in the storage device is based on the size of the storage device and the rate at which memory address translations stored in the storage device are removed from the storage device.

9. An apparatus for managing memory access, comprising: At least one processor is configured to implement at least one convolutional layer of a convolutional neural network on input feature map data to generate output feature map data; as well as Storage device, used to store multiple memory address translations, The device is configured as follows: Based on at least one or more features of the convolutional neural network, predict the access process partially related to the memory outside the at least one processor; and In response to the prediction, prior to the predicted access process, operations are performed to obtain a memory address translation corresponding to a portion of the memory and store the memory address translation in the storage device. The one or more features of the convolutional neural network include the order in which portions of the input feature map data are accessed to implement the at least one convolutional layer of the convolutional neural network.

Images