Neural network processing

Abstract

The invention is entitled "Neural network processing". The invention discloses a method for performing neural network processing and a corresponding data processing system. The data processing system is configured to define one or more tiles to be used when reading a portion of an input feature map from memory or writing a portion of an output feature map to memory. The data processing system is further configured to provide information allowing locations falling within the defined one or more tiles to be mapped to memory locations to allow a processor to read data of the input feature map from memory or write data of the portion of the output feature map to memory.

Description

Background Technology

[0001] The techniques described in this article involve the processing of feature maps in neural networks.

[0002] Neural networks can be used in processes such as machine learning, computer vision, and natural language processing. A neural network operates on appropriate input data (e.g., image or sound data) to ultimately provide a desired output (e.g., recognition of objects in an image or speech in a sound clip, or other useful output inferred from the input data). This process is often referred to as "inference" or "classification."

[0003] Neural networks typically process input data (e.g., image or sound data) according to the network of operators, each performing a specific operation. These operations are usually performed sequentially to produce the desired output data (e.g., classification based on image or sound data). Each operation can be referred to as a "layer" of the neural network processing.

[0004] Therefore, neural network processing may include processing a sequence of “layers” such that the output from each layer is used as the input to the next processing layer. Figure 1 An exemplary sequence of neural network processing layers from the initial input layer 101 to the final output layer 107 is shown, with layers including various convolutional layers (C layers) 102, 103, 104 and fully connected layers (FC layers) 105, 106 in between.

[0005] Input layer 101 can be configured to receive input data (e.g., image or sound data) and provide that input data in a suitable form (e.g., as an array of data elements, or referred to as a "feature map") for use by subsequent neural network layers. The feature map will typically consist of a three-dimensional array of data elements, each data element having associated data. The feature map may have a width (W), a height (H), and a depth (C), where the width (W) and height (H) can be defined as the number of data elements in the width and height directions, respectively, and the depth (C) can correspond to the number of data channels. For example, with input data including an image, the width and height of the array provided by the input layer may correspond to numerical values ​​(e.g., pixels) of data positions along the width and height directions of the image, respectively, while the channels may include the RGB channels of the image.

[0006] Following the input layer, one or more other layers of neural network processing may exist (e.g., including convolutional layers, fully connected layers, pooling layers, or any other layers of neural network processing that may exist).

[0007] Generally, each layer of a neural network (e.g., a convolutional layer, a fully connected layer, or a pooling layer) processes the input feature map (IFM) to generate a corresponding output feature map (OFM). The output feature map generated by the neural network processing layer is then used as the input feature map for the next processing layer in the sequence, and so on. This is in... Figure 2 As shown in the image.

[0008] As used in this article, the term "feature map" can refer to either the input feature map or the output feature map.

[0009] Feature maps can be processed in a "batch" manner, where multiple (e.g., unrelated) feature maps are processed simultaneously.

[0010] like Figure 2 As shown, the data of the input feature map (IFM) processed by a specific layer of the neural network can be read from the working memory (e.g., a buffer). The input feature map (IFM) may include data that has previously been written to the buffer as part of the output feature map (OFM) generated by the previous neural network processing layer.

[0011] The operations performed by each layer of a neural network can include any suitable operation that manipulates the input feature map to provide an output feature map. This operation may require process parameters (e.g., the weights of filters or "kernels") that are specific to the layer the neural network is processing. Therefore, as... Figure 2 As shown, appropriate process parameters (e.g., weights) can be read from the working memory (e.g., a buffer) to perform each layer of neural network processing.

[0012] refer to Figure 1 The final layer of neural network processing in the sequence may include an output layer 107. The output layer can process the input feature maps to generate useful output data (e.g., inference or classification).

[0013] Therefore, known neural network processing may include processing the input feature map to provide an associated output feature map.

[0014] Nevertheless, the applicant believes there is a range of ways to improve the processing of feature maps and associated data in neural networks. Attached Figure Description

[0015] Various embodiments of the technology described herein will now be described by way of example only, with reference to the accompanying drawings, wherein:

[0016] Figure 1 An exemplary sequence of neural network processing layers, including input and output layers, is shown, with neural network layers including various convolutional layers (C layers) and fully connected layers (FC layers) in between.

[0017] Figure 2 A series of neural network processing layers is shown, wherein the output feature maps from the neural network processing layers can be written to a suitable buffer and then used as the input feature maps for the next layer in the series, and wherein each neural network processing layer can use processing parameters (e.g., weights) read from a suitable buffer;

[0018] Figure 3 This illustrates how the size of the feature map can change when processed according to the individual neural network processing layers;

[0019] Figure 4 An exemplary partition of a feature map into segments (“strips”), wherein each segment extends across the entire width of the feature map;

[0020] Figure 5 The processing of feature maps of a first successive operation (layer) and a second successive operation (layer) according to an embodiment of the technology described herein is illustrated, wherein each operation (layer) processes a portion of the feature map smaller than the full width of the feature map;

[0021] Figure 6 To show the basis Figure 5 A system diagram of the data flow used to process feature maps;

[0022] Figure 7 This demonstrates how multiple tiles can be used to describe a portion of a feature map;

[0023] Figure 8 It provides a list of parameters that can be used in the command stream to define a portion of the input feature map based on one or more tiles, and a set of memory locations for mapping each tile to the data from which that tile will be read.

[0024] Figure 9 It provides a list of parameters that can be used in the command stream to define a portion of the output feature map based on one or more tiles, and a set of memory locations for mapping each tile to the data to be written to that tile;

[0025] Figure 10 This is an exemplary set of instructions that can be provided in a command stream in an implementation of the technology described herein for processing portions of a feature map by performing successive layers of neural network processing that respectively execute operations A and B;

[0026] Figure 11A and Figure 11B The flowcharts illustrate the processing based on a portion of the feature map of each of operations A and B, and... Figure 11C A flowchart illustrating a process based on a portion of operations A and B in an embodiment of the technology described herein;

[0027] Figure 12 Examples of the contents of the buffer are provided in (i) to (viii), where a portion of the output feature map is written to the buffer when the first neural network processing layer is executed, and a portion of the input feature map is read from the buffer for use in executing the second (next) neural network processing layer.

[0028] Figure 13 It shows the corresponding Figure 12 In cases (i) and (ii), how can one or more tiles be used to define a portion of the input feature map that will be read from the buffer or to define a portion of the output feature map that will be written to the buffer;

[0029] Figure 14 It shows the relationship with Figure 5 Compared to alternative forms that divide the feature map into multiple parts;

[0030] Figure 15 (i) to (viii) provide when using Figure 14 The partial arrangement shown is an example of the possible contents of the buffer. When the first neural network processing layer is executed, a portion of the output feature map is written to this buffer, and a portion of the input feature map is read from this buffer for use in executing the second (next) neural network processing layer; and

[0031] Figure 16 It shows the relationship with Figure 7 Compared to the alternative tiling layout, it can be used to describe different parts of the feature map, and when... Figure 15 The methods shown in (i) to (iii) are appropriate for managing the buffer.

[0032] Similar reference numerals are used for similar features in the accompanying drawings (where appropriate). Detailed Implementation

[0033] A first embodiment of the technology described herein includes a method for controlling a processor operable to perform neural network processing in a data processing system, the data processing system including a processor operable to perform neural network processing and a memory for storing data to be used during the execution of neural network processing, the method comprising:

[0034] Neural network processing is performed on corresponding portions of the input feature map, each portion of which includes an array of data elements, each data element having corresponding data associated with it, and each data element having an associated position within the input feature map; and

[0035] When performing neural network processing on a portion of the input feature map:

[0036] The portion of the input feature map is defined based on one or more patches, wherein each patch corresponds to a corresponding region of the portion of the input feature map, and each region corresponds to one or more locations of data elements within the data element array of the portion of the input feature map, wherein the patches together form the entire portion of the input feature map;

[0037] For each tile, information is provided that allows each data element location forming the tile to be mapped to a corresponding memory location, the corresponding memory location storing data of the data element associated with the location;

[0038] The method further includes:

[0039] When data of the location of data elements within the portion of the input feature map is retrieved from the memory, the processor identifies which of the one or more defined tiles the location falls into, uses provided information to identify the memory location from which data is read from the location, the information allowing each location forming the tile to be mapped to a corresponding memory location, and reading the data from the identified memory location; and

[0040] The processor processes the acquired data of one or more data elements in the input feature map portion to provide one or more data elements for the corresponding portion of the output feature map.

[0041] A second embodiment of the technology described herein includes a data processing system, the data processing system comprising:

[0042] A processor, operable to perform neural network processing; and

[0043] The memory is used to store data to be used when performing neural network processing;

[0044] in:

[0045] The processor is configured to:

[0046] Neural network processing is performed on corresponding portions of the input feature map, each portion of which includes an array of data elements, each data element having corresponding data associated with it, and each data element having an associated position within the input feature map; and

[0047] The data processing system further includes a processing circuit, which is configured to:

[0048] A portion of the input feature map to be processed is defined based on one or more patches, wherein each patch corresponds to a corresponding region of the portion of the input feature map, and each region corresponds to one or more locations of data elements within the data element array of the portion of the input feature map, wherein the patches together form the entire portion of the input feature map; and

[0049] For each tile, information is provided that allows each data element location forming the tile to be mapped to a corresponding memory location, the corresponding memory location storing data of the data element associated with the location;

[0050] and

[0051] The processor is further configured to perform neural network processing on the portion of the input feature map as follows:

[0052] When data on the location of data elements within the portion of the input feature map is retrieved from the memory, it is identified which of the one or more defined tiles the location falls into. The provided information is used to identify the memory location from which data is read from that location. This information allows each location forming the tile to be mapped to a corresponding memory location, and the data is read from the identified memory location.

[0053] The acquired data of one or more data elements in the input feature map portion are processed to provide one or more data elements for the corresponding portion of the output feature map.

[0054] As referenced above Figure 1 and Figure 2 The known neural network processing discussed may include processing input feature maps to provide associated output feature maps, wherein, if necessary, data of these feature maps may be stored in working memory (e.g., buffers) (e.g., between neural network processing layers).

[0055] However, the applicant has recognized that since the feature maps involved in neural network processing can be quite large, a buffer of a considerable size would be required if the feature maps were to be stored as a whole (e.g., between two consecutive layers of neural network processing).

[0056] Furthermore, the applicant has recognized that, depending on the specific neural network processing operation performed, the size of the output feature map generated by the neural network processing (e.g., including neural network processing layers) can differ from the size of the input feature map generated by it. This is, for example, in... Figure 3The diagram illustrates that a feature map FM0 with initial dimensions H×W×C is processed by the layer performing operation A to generate an output feature map FM1 with dimensions H×W×2C. This output feature map is then processed by the next layer performing operation B to generate an output feature map FM2 with dimensions H / 2×W / 2×2C. Therefore, the size of the feature map generated during neural network processing can vary. In practice, if the entire feature map were to be stored between neural network processing layers, a relatively large amount of working memory might be required to accommodate the possible changes in the feature map's size during neural network processing.

[0057] The applicant has recognized that one possibility for reducing the amount of working memory required when performing neural network processing is to process the feature map into multiple separate, smaller parts, for example, one after another. In this case, the first neural network processing operation (layer) can perform processing on a portion of the feature map and write it (only) to the working memory, and the next neural network processing operation (layer) can then use that portion of the feature map that has been written to the working memory to perform processing, and so on, where each individual part of the feature map is processed in this way. Therefore, it is not necessary to store the entire feature map in the working memory at the same time, and a working memory size smaller than that of the feature map can be provided accordingly.

[0058] Such processing of the data used for successive operations (layers) is referred to in this paper as “cascading” those operations, based on the principle that the entire feature map is not stored between those successive operations (layers).

[0059] Therefore, in the technique described herein, the input feature map in neural network processing is processed into corresponding portions of that input feature map. Furthermore, to facilitate reading the relevant input feature map portions from memory (such as working (buffer) memory), each portion of the feature map is defined based on one or more patches, and these patches can then be mapped to corresponding memory locations to allow reading data from that portion of the input feature map.

[0060] Specifically, each patch describes a region of the input feature map corresponding to one or more locations of data elements within that portion of the input feature map. When data for that portion of the input feature map is read from memory, each data element location forming the patch is mapped to a corresponding location in the memory. Thus, the one or more locations forming the patch are mapped to a set of one or more (corresponding) locations in the memory from which data is to be read.

[0061] Defining a portion of a feature map based on one or more tiles, as described in this paper, provides a degree of flexibility when reading data from working (buffer) memory. For example, the set (or sets) of memory locations from which data is to be read can be varied by defining fewer or more tiles, by selecting the size of the defined tiles, and by selecting an appropriate mapping from the locations forming the tiles to memory locations. As described below, this can be used (and indeed allows) situations where a relatively small working (buffer) memory is provided, and further, where the working (buffer) memory is managed in a manner that takes into account filter margins.

[0062] For example, the applicant has recognized that when processing a partial feature map, the input feature map portion to be processed by the neural network may include data from more than one previously generated output feature map portion as a result of neural network processing. For example, the input feature map portion may include data from two or more previously generated output feature map portions that are adjacent (adjacent) to each other within the output feature map. When the input feature map portion is to be processed according to a neural network processing including filter operations, it is possible that at least some of the data to be read from the working (buffer) memory for that input feature map portion includes filter margins.

[0063] The applicant has recognized that, in this situation, it may be desirable to manage the working (buffer) memory such that the data in the newly generated output feature map portion does not necessarily completely overwrite the data stored from the previously generated output feature map portion. For example, it may be desirable to store the newly generated output feature map portion such that at least some data in the boundary (or vicinity) between the newly generated output feature map portion and the previously generated adjacent output feature map portion is not overwritten (so that the filter margin is maintained (reserved) in the working (buffer) memory for use when reading the input feature map portion from the working (buffer) memory). However, it may be permissible for the data in the newly generated output feature map to overwrite the previously generated data, which no longer needs to be processed as an input feature map portion (e.g., data that does not contribute to the filter margin).

[0064] Therefore, the applicant has recognized that in some cases, it may be inappropriate to allow newly generated output feature map portions to simply overwrite the entire previous output feature map portion within the working (buffer) memory. Thus, it may be desirable to distribute the data of the newly generated output feature map portions across multiple sets of memory locations within the working (buffer) memory so that any existing data still needed to be retained in the working (buffer) memory for "fitting around" (e.g., retaining data at or near the boundary of an adjacent previously generated output feature map portion within the output feature map that forms the filter margin to be read when processing the input feature map portion). Therefore, the data of the input feature map to be read can also be distributed across multiple sets of memory locations. In this respect, these tiles of the technique described herein allow data (of the input feature map portion) to be read from multiple sets of memory locations by defining a suitable number of tiles with appropriate sizes and mapping those tiles to multiple sets of memory locations.

[0065] Therefore, these tiles of the technology described herein allow for the management of working (buffer) memory in a manner that takes into account filter margins.

[0066] Furthermore, these tiles of the technique described herein allow for management of the working (buffer) memory, enabling the provision of a relatively small working (buffer) memory while still taking filter margins into account. For example, because these tiles allow data from the output feature map portion (as well as the input feature map portion) to be written to (or read from) multiple sets of memory locations in a relatively flexible manner, it is not necessary to provide a working (buffer) memory large enough to store most (or the entire) of the feature map.

[0067] The input feature map may include any suitable feature map (where the term "input" refers only to a feature map that is to be read from memory for processing according to the neural network processing).

[0068] Neural network processing of this portion of the input feature map may include a neural network processing layer. This neural network processing layer may be "cascaded" with a preceding neural network processing layer, as described above, such that both neural network processing layers are executed on a partial basis. Therefore, the portion of the input feature map to be processed may correspond to at least a portion of the output feature map that was written to memory during the execution of the preceding neural network processing layer.

[0069] The input feature map includes an array of data elements, each data element having corresponding associated data and an associated position within the input feature map. The input feature map may include a rectangular array of data element positions. The input feature map may have a width corresponding to multiple data elements in the width direction (horizontal, x-direction) and a height corresponding to multiple data elements in the y-direction (vertical, y-direction). The input feature map may include multiple arrays of data elements, each array forming a channel of the input feature map, such that the input feature map has a depth corresponding to the number of channels in the input feature map.

[0070] Each portion of the input feature map processed in the manner described herein includes an array of data elements, which is a subset of the array of data elements forming the input feature map (and thus includes some but not all of the data elements in the input feature map). Each data element forming that portion of the input feature map has an associated position within that portion of the input feature map. That portion of the input feature map may include an array of data element positions, in one embodiment including a rectangular array of data element positions. That portion of the input feature map may have a width (in the x-direction) and a height (in the y-direction) corresponding to a plurality of data elements in the width and height directions, respectively. That portion of the feature map is smaller than the input feature map. Therefore, at least one of the width and height of that portion of the input feature map (and both in one embodiment) is smaller than the width and height of the input feature map, respectively.

[0071] When processing is performed on this portion of the input feature map, this portion of the feature map is defined based on one or more patches. Therefore, this portion of the feature map is divided into one or more defined patches.

[0072] Each tile should (and in one embodiment does) correspond to a continuous region of that portion of the input feature map, the region including one or more consecutive locations of data elements within that portion of the input feature map. Each tile may be rectangular. Each tile will have a width (in the x-direction of the tile) and a height (in the y-direction of the tile), corresponding to a plurality of data elements forming the tile in the width and height directions, respectively. In one embodiment, the width (x) and height (y) directions of the tile are aligned with the width (x) and height (y) directions of the input feature map.

[0073] The one or more tiles together form the entire portion of the input feature map. Therefore, each data element forming the portion of the input feature map resides within at least one tile (and in one embodiment, only within one tile, such that these tiles do not overlap). In one embodiment, the one or more tiles constituting the portion of the input feature map have a predefined layout (relative to each other) across the portion of the input feature map (within the portion). (Alternatively, information defining the relative positions of these tiles within the portion of the input feature map may be provided). The number of tiles defined by the processor for use by the processor when performing processing on a portion of the input feature map can vary (e.g., depending on how the portion of the input feature map is stored in memory). In one embodiment, the number of tiles that can be defined can vary up to a predetermined maximum number of (allowed) tiles. In one embodiment, the maximum number of tiles is four tiles (however, if both the tiles and the portion of the input feature map are rectangular, other maximum numbers of tiles, such as 2, 3, 5, 6, 7, 8... etc., or even numbers such as 2, 6, 8... etc., can be used). In this regard, the applicant has recognized that when processing of feature maps is performed on a partial basis (even when the feature map portions are distributed in the working (buffer) memory in a manner that takes into account filter margins), a maximum of 4 tiles may be sufficient.

[0074] The one or more blocks used for the input feature map portion can be defined in any suitable and desired manner, for example, using any suitable and desired parameters (variables).

[0075] In one implementation, the one or more blocks are defined by providing information from which the dimensions of each block can be derived. In this implementation, the blocks are rectangular, and the information allows the height and width of each block to be used to be derived. As described above, the height and width may correspond, respectively, to numerical values ​​representing the position of data elements in the height and width directions of the block.

[0076] In one implementation, constraints are imposed on the height and / or width of available tiles relative to each other, such that it is not necessary to explicitly define the height and width of each tile to be used in that portion of the input feature map. In one implementation, the processor may infer the height of one or more of these tiles to be used from the defined height of another tile and / or from the defined height of that portion of the input feature map. Alternatively or additionally, the processor may infer the width of one or more of these tiles to be used from the defined width of another tile and / or from the defined width of that portion of the input feature map. In one implementation, the one or more tiles have a defined (predefined in one implementation) layout across that portion of the input feature map comprising one or more columns of tiles. In such implementations, in one implementation, constraints are imposed on the height and width of the one or more tiles such that tiles within the same column have the same width relative to each other (corresponding to the width of the column). In one implementation, tiles within different columns may (are permitted) have different widths relative to each other (such that the width of each column may be different). In one implementation, the one or more tiles are defined such that tiles within the same column may (are permitted) have different heights relative to each other. In one embodiment, the one or more blocks are defined such that blocks in different columns (adjacent or, in one embodiment, any) can (be allowed) have different heights relative to each other (e.g., the height of the blocks can be set independently for each column). In one embodiment, when a maximum of four blocks can be defined, these blocks comprise a maximum of two columns of blocks, with each column comprising a maximum of two blocks.

[0077] As mentioned above, by imposing constraints on the height and / or width of tiles relative to each other, the data processing system does not necessarily have to define (and is defining) the height and / or width of each tile (because the height and / or width of some tiles can be inferred from the height and / or width of other tiles).

[0078] Furthermore, the applicant has recognized that, in cases where the working (buffer) memory is managed as a scrolling buffer that scrolls in both the vertical (y) and horizontal (x) directions, and introduces a vertical offset when scrolling in the horizontal direction (but not any horizontal offset when scrolling in the vertical direction), a configuration using multi-column tiles (as discussed above) may be effective for reading a portion of the input feature map from that working (buffer) memory. For example, the applicant has recognized that when the working (buffer) memory is managed in this way, the data of the input feature map portion tends to be distributed across one or more sets of memory locations, each of which can be mapped using tiles, even when constraints are applied (when these tiles are "pieced together" to form the input feature map), and these tiles have a layout containing one or more columns with the aforementioned constraints.

[0079] In an alternative implementation of this type of embodiment, the one or more tiles have a defined (predefined in one implementation) layout across the portion of the input feature map comprising one or more rows of tiles. In one implementation, constraints are imposed on the height and width of the one or more tiles such that tiles within the same row have the same height relative to each other (corresponding to the height of the row). In one implementation, tiles within different rows may (are allowed) have different heights relative to each other (such that the height of each row can be different). In one implementation, the one or more tiles are defined such that tiles within the same row may (are allowed) have different widths relative to each other. In one implementation, the one or more tiles are defined such that tiles within different rows (adjacent or, in one implementation, any) may (are allowed) have different widths relative to each other (e.g., the width of the tiles can be set independently for each row). In one implementation, when a maximum of four tiles can be defined, these tiles comprise at most two rows of tiles, with each row comprising at most two tiles.

[0080] The applicant has recognized that, in cases where the working (buffer) memory is managed as a scrolling buffer that scrolls in both the vertical (y) and horizontal (x) directions, and introduces a horizontal offset when scrolling in the vertical direction (but not any vertical offset when scrolling in the horizontal direction), a configuration using multi-row tiles (as discussed above) may be effective for reading a portion of the input feature map from that working (buffer) memory. For example, the applicant has recognized that when the working (buffer) memory is managed in this way, the data for portions of the input feature map tends to be distributed across one or more sets of memory locations, each of which can be mapped using tiles, even when constraints are applied (when these tiles are "pieced together" to form the input feature map), resulting in a layout containing one or more rows with the aforementioned constraints.

[0081] Therefore, in one embodiment, the information provided to the processor to define the tiles constituting the portion of the input feature map includes the overall height and overall width of the portion of the input feature map (in one embodiment, in terms of the number of data element positions in the respective direction), and: the width of each column of tiles except the last column (the width of the last column can be inferred from the width of the feature map portion and the widths of the other columns), and for each column, the tile height of each tile except the final tile in that column (the final tile height can again be determined from the overall height of the input feature map portion and the heights of the other tiles in that column); or the height of each row of tiles except the last row (the height of the last row can be inferred from the height of the feature map portion and the heights of the other rows), and for each row, the tile width of each tile except the final tile in that row (the final tile width can again be determined from the overall width of the input feature map portion and the widths of the other tiles in that row).

[0082] Therefore, in an implementation where up to four tiles can be defined for reading a portion of an input feature map from memory, these tiles have a layout spanning the input feature map portion comprising up to two columns. Information provided to the processor to define all four tiles includes: the height of the portion of the input feature map, the width of the portion of the input feature map, the width of a single tile (corresponding to the width of the column containing the tile), and the height of a single tile in each of these columns. The processor is configured to infer the height and width of any tile not explicitly defined from this information. If fewer than four tiles are to be defined, the information provided to the processor will, as appropriate, include selections of that information.

[0083] In an alternative embodiment where up to four tiles can be defined for reading a portion of an input feature map from memory, these tiles have a layout spanning the input feature map portion comprising up to two rows. Information provided to the processor to define all four tiles includes: the height of the portion of the input feature map, the width of the portion of the input feature map, the height of an individual tile (corresponding to the height of the row containing the tile), and the width of an individual tile in each of these rows. The processor is configured to infer the height and width of any tile not explicitly defined from this information. If fewer than four tiles are to be defined, the information provided to the processor will, as appropriate, include selections of that information.

[0084] In cases where the input feature map portion includes multiple data channels, in one embodiment, the processor also provides an indication of the relevant depth (i.e., the number of channels) of the input feature map portion.

[0085] As described above, in one embodiment, the one or more tiles constituting that portion of the input feature map have a predefined layout (relative to each other) across that portion of the input feature map. For example, with up to four tiles available, the first, second, third, and fourth tiles may correspond to the top-left, top-right, bottom-left, and bottom-right tiles, respectively. In one embodiment, the relative positions (layout sequence) of these tiles are maintained regardless of the number of tiles defined for that portion of the input feature map and regardless of the exact dimensions (height and width) of each tile. (Alternatively, information defining the relative positions (layout sequence) of these tiles within that portion of the input feature map may be provided). Therefore, it is not necessary to provide information indicating a specific (precise) position of the tiles within that portion of the input feature map (because the position of the tiles will depend on the (predefined) layout sequence of the tiles and the dimensions of the tiles).

[0086] As described above, the one or more patches are defined as spanning the entire input feature map portion such that the location of each data element within the input feature map falls within (only) a single patch. In this respect, each data element can be considered to have a location not only within the input feature map portion but also within a patch.

[0087] In one embodiment, each block has a reference position within the block, such that each data element forming the block has a position (which can be identified) within the block measured (counted) relative to the reference position of the block. In one embodiment, the position of the data element within the block corresponds to a plurality of data positions at a distance from the reference position of the block.

[0088] The reference position within a block is, in one implementation, a specific location, in another implementation, a selected location, or in yet another implementation, a predefined location within or relative to the block, such as a corner of the block, such as the top left corner.

[0089] The reference position of each tile can be explicitly defined, for example, using appropriate instructions in the instruction stream. However, in one implementation, the reference position of each tile is predefined, so it does not need to be explicitly indicated to the processor. In one implementation, the reference position of each tile is predefined as a corner of the tile (and in one implementation, the top-left corner).

[0090] In one implementation, the reference position of the block is the origin of the block's x, y coordinate system. Therefore, in one implementation, each data element forming the block has an x, y position (identifiable by it) relative to the block's origin, where the x, y position of the data element corresponds to the numerical values ​​from the origin to the data element's position in the x and y directions of the block.

[0091] In one implementation, the x and y directions of the patch correspond to the x and y directions (aligned) of that portion of the input feature map, respectively. Therefore, data elements with x and y positions within a portion of the input feature map can (additionally) be identified by their x and y positions within the patch, where the considered data element falls relative to the origin of the patch.

[0092] When reading a portion of an input feature map from memory, one or more defined tiles are used by mapping locations within each defined tile to a corresponding set of memory locations from which data is to be read. Specifically, information is provided that allows each location within a tile to be mapped to a corresponding memory location, which stores data of the data element associated with that location. This information, allowing each location within a tile to be mapped to a corresponding memory location, can be defined in any suitable and desired manner, for example, using any suitable and desired parameters (variables).

[0093] The information provided that allows each location in a tile to be mapped to a corresponding memory location (memory address) storing data of the data element associated with that location may include any suitable and desired information that allows locations within a tile to be mapped to corresponding memory locations (memory addresses) storing data of the data element associated with that location.

[0094] In one embodiment, the provided information includes information that allows mapping a reference location of a tile (within) to a "reference" memory location (address) that stores data of a data element located at the reference location of the tile. In one embodiment, this information is provided in the form of a reference memory location (address) of the tile's reference location. Then, in one embodiment, the provided information also includes a memory location (address) that allows identifying the data at the location of a data element in memory using the relative position of the data element within the tile (relative to the reference location of the tile). In one embodiment, the information includes information that allows determining the location (as an offset) of the data at the reference memory location of the data element relative to the reference location of the tile based on the data element's position within the tile relative to the reference location of the tile.

[0095] Therefore, in one embodiment, the information provided to allow each location forming a tile to be mapped to a corresponding memory location includes information indicating the span (memory address spacing) between data corresponding to adjacent data element locations within the tile. Thus, the information indicating the span (memory address spacing) can be used in conjunction with the relative position of a data element within the tile to identify the memory location where the data element's data is stored (e.g., by scaling the data element's position within the tile relative to a reference position of the tile using the information indicating the span to obtain the position of the data element's data in memory relative to the reference memory address of the tile). Therefore, in one embodiment, the information indicating the span (of data) between adjacent data elements within the tile in memory includes information indicating the span of at least one of the following: the horizontal (x) direction (and thus the memory spacing of each (position) "step" in the horizontal direction of the tile) and the vertical (y) direction (and thus the memory spacing of each (position) step in the vertical direction of the tile). This span can be used to indicate the appropriate (relative) memory address spacing using any suitable and appropriate desired unit of memory space, such as indicating a number of bytes in one embodiment.

[0096] In an implementation where the feature map includes multiple channels, the information provided to allow each location forming a tile to be mapped to a corresponding memory location may further include: information indicating the span (memory address spacing) between data elements that have the same horizontal and vertical position within the feature map but have depths (in other words, "channel span") corresponding to adjacent channels of the feature map.

[0097] In implementations, the channels of a feature map can be divided into multiple groups (referred to herein as “blocks”), with data for each group of channels stored in a different area of ​​memory (e.g., this provides convenience and efficiency in accessing data when the feature map comprises a large number of channels). For example, these channels can be divided into groups (“blocks”), each group consisting of 16 channels. Thus, the starting point of each group (“block”) of channels can be spaced apart in memory by a span (referred to herein as a “block span”). In such cases, data elements in groups (“blocks”) with the same horizontal and vertical positions but different depths within the feature map, for example, the number of channels (e.g., 16 channels), will be spaced apart in memory by the “block span”. In such implementations, where these channels of the feature map are divided into blocks for storage in memory, information provided to allow each location forming a block to be mapped to a corresponding memory location may also include information indicating the “block span”. In implementations, for example, multiple feature maps (corresponding to “batches” of feature maps) can be processed simultaneously, such that a first (identical) portion of each feature map in a batch is processed before processing the next portion for the next feature map in the batch. In such implementations, the information provided to allow each location forming a tile to be mapped to a corresponding memory location may include information indicating the span (memory address spacing) between batches in memory (e.g., between data elements that have the same horizontal and vertical position and fall within the same channel but within consecutive feature maps to be processed). Such spans may be referred to herein as “batch spans”.

[0098] Therefore, in one embodiment, the information provided to the processor to allow each location in a tile to be mapped to a corresponding memory location storing data for that tile location includes the memory address of the tile's reference (origin) location (reference memory address), and the span between memory locations of data locations along the horizontal direction in the tile and the span between memory locations of data locations along the vertical direction in the tile. When the feature map includes multiple channels, information including channel spans (and optionally block spans) is also provided, and when batching feature maps, information including batch spans is also provided.

[0099] As will be discussed in more detail below, horizontal and vertical spans can be used to scale the relative position of data elements within a tile (relative to the tile's reference position) to obtain the relative position (offset) in memory (relative to the tile's reference memory address). The offset in memory relative to the tile's reference memory address can be further adjusted to account for the channels and / or blocks and / or batches in which the data elements fall (by adding additional offsets based on channel spans and / or block spans and / or batch spans). The effect is that a tile can be used to read data of data elements stored in a set of memory locations, each memory location having an offset in memory that can be derived from the tile's reference memory address, and the appropriate scaled position of the data element within the tile (relative to the tile's reference position) (and, where applicable, any of the channels, blocks, and batches).

[0100] In one implementation, the same span (span value) is used for all patches constituting a portion of the input feature map, but each patch has its own, separate, indicative, and provided reference location memory address. However, the span value used may be different for different feature maps and / or for different portions of the feature map to be processed by the neural network.

[0101] In cases where the input feature map can be formed from at most a certain maximum number of blocks, in one embodiment, a base memory address can be provided for each of the maximum number of blocks that can be used, and for any block that is not actually defined for the input feature map (i.e., does not contain data of the input feature map), there is a base address indicating such as an empty base address or a (repeated) base address corresponding to a block in another block.

[0102] However, in one implementation, a reference memory address is provided for each (and only each) block actually used to (define) the considered input feature map portion. In such implementations, it is not necessary to provide reference addresses for any blocks not used to (define) read data from the considered input feature map portion.

[0103] In one implementation, the base memory address provided for a block is indicated as an offset relative to some other memory addresses (which may be set independently of the base memory address of the block, for example, by a driver or compiler at runtime). Therefore, when the location of a data element within a block is mapped to a memory address location, a memory address spacing relative to the base memory address of the block is determined (and subsequently specified relative to the other memory addresses). Thus, if the other memory addresses are changed (e.g., by a driver or compiler), any defined block will be mapped to a different region of memory. In this way, the memory location to which the data element location within a defined block is mapped can be repositioned by changing the other memory addresses (wherein the repositioning occurs without changing the block definition, such that any instructions issued to define the block (e.g., within a command stream) remain valid despite the repositioning). Of course, other arrangements are also possible.

[0104] Information defining one or more patches of the input feature map, along with information allowing each location forming a patch to be mapped to a corresponding memory location, is used to acquire data corresponding to that portion of the input feature map for processing.

[0105] To this end, for a location within that portion of the feature map to be processed, the processor identifies which of one or more defined blocks includes the location (the location falls within it), and then uses the provided information to identify the memory location from which data is to be read, which allows each location forming a block to be mapped to a corresponding memory location, and reads the data from the identified memory location (address).

[0106] The location of the data element to be read from memory within the input feature map portion can be indicated and defined using any suitable variables, such as, in one implementation, using the horizontal (x) position and vertical (y) position of the data element within the feature map portion.

[0107] To read data from a data element within a portion of the input feature map, the processor first identifies which definition block the data element falls into. The processor may identify which definition block the data element falls into based on one or more of the following, and in one embodiment, based on both: the (relative) position of the data element within that portion of the feature map; and the size of one or more definition blocks within the definition map.

[0108] When identifying which tile a data element's location falls into, the processor considers tiles in one implementation according to a specific order, in another according to a selected order, and in yet another according to a predefined order. If it is determined that the location does not fall into a given tile, the processor considers the next tile in that order, and so on. In one implementation, the processor compares the location of the data element with the dimensions (e.g., height and / or width) of a first tile to determine whether the data element falls into that first tile. In one implementation, the first tile is located at a corner of that portion of the input feature map (e.g., the top-left corner of the defining tile). If the data element does not fall into the first tile, the processor may, if necessary, determine whether the location of the data element falls into a second tile of the defining tile, for example, the second tile may be adjacent to the first tile horizontally or vertically, and so on for a third tile, a fourth tile, and so on. In one implementation, the processor considers tiles row by row (thus considering all tiles in a row sequentially, then moving to the next row and considering all tiles in that row, and so on), or column by column.

[0109] In one implementation, the processor also determines the relative position of the data element within the tile it falls into (relative to a reference position of the tile). For example, and in one implementation, the processor may set (and, if necessary, modify) x and y variables describing the position of the data element within a portion of the feature map such that they correspond to x and y positions within the tile.

[0110] For example, when the tiles are arranged in at most two columns (as described above), the processor can compare the horizontal (x) position of a data element with the width of the first (e.g., the leftmost) column to determine whether the position falls within that first column. If the data element falls within the first column, the processor can identify which tile within that column the data element's position falls into (using the data element's vertical (y) position) (and can modify the variables describing the data element's vertical (y) position so that they represent the relative position within the identified tile). If the data element does not fall within the first column, the processor can identify which tile within the second column the data element's position falls into (and can adjust the variables describing the data element's horizontal (x) and / or vertical (y) position so that they represent the relative position within the identified tile).

[0111] Accordingly, when the tiles are arranged in at most two rows (as described above), the processor can compare the (vertical) position of a data element with the height of the first (e.g., topmost) row to determine whether the position falls within that first row. If the data element falls within the first row, the processor can identify which tile within that row the data element's position falls into (and can adjust the variable describing the data element's horizontal (x) position so that it represents the relative position within the identified tile). If the data element does not fall within the first row, the processor can identify which tile within the second row the data element's position falls into (and can adjust the variables describing the data element's horizontal (x) and / or vertical (y) position so that they represent the relative position within the identified tile).

[0112] Once the processor has identified which tile the considered input feature map data location falls into, the processor uses information mapping the data location within that tile to a memory location to determine the memory location (address) from which to read the considered data element (location). This can be done in any suitable and desired manner, for example, and in one embodiment, based on information provided to map the data location within the tile to a memory location.

[0113] Therefore, in one embodiment, the operation uses the relative position of the data element in the tile (which in the embodiment determined as described above) to determine (identify) the memory location of the data for that data location, in one embodiment based on an indication of a reference memory location (address) provided for a reference location for the tile, and the indicated span for the data location.

[0114] Therefore, in one embodiment, the processor multiplies (scales) the relative spacing in the x-direction between the data location in the tile and the reference location in the tile by the corresponding memory address span provided to the processor in the x-direction, and correspondingly multiplies (scales) the relative spacing in the y-direction between the data location in the tile and the reference location in the tile by the corresponding memory address span provided to the processor in the y-direction, to determine the memory location (offset) of the data element relative to the reference memory address of the tile. In one embodiment, the scaled relative spacings in the x and y directions are summed to obtain the memory location (offset) of the data element relative to the reference memory address of the tile.

[0115] When the input feature map comprises multiple channels, optionally these channels are divided into multiple blocks. The memory location (offset) of the data for the data element location is obtained by additionally summing the offsets to account for the channels under consideration, and optionally by summing the offsets to account for the blocks under consideration. The offset of the channel under consideration is obtained by multiplying the value indicating which channel the data element location falls into by (scaling) the span between channels (“channel span”). The offset of the block is obtained by multiplying the value indicating which block the data element falls into by the span between blocks (“block span”).

[0116] When multiple input feature maps are processed as batches, the memory location (offset) of the data element location is obtained by additionally summing the offsets to account for the batches under consideration. The offset of the batch under consideration is obtained by multiplying (scaling) the span between batches (“batch span”) by the value indicating which batch the data element location under consideration falls into.

[0117] Therefore, in one embodiment, the processor identifies the memory location of the data of the data element to be read from the portion of the input feature map by: identifying which definition block of the definition block of the portion of the input feature map the data element falls into; identifying the relative position of the data element within the block relative to a reference position of the block; identifying the memory location corresponding to the reference position of the block; and using the relative position of the data element within the block relative to the reference position of the block and the identified memory location corresponding to the reference position of the block, to determine the memory location for storing the data of the data element under consideration.

[0118] The above describes the operations used to obtain the data values ​​of a given data element from the input feature map. The processor will accordingly obtain the data values ​​of as many different data elements as possible that need to be processed in the input feature map portion (e.g., obtain them together) to generate one or more appropriate data elements for the corresponding output feature map portion.

[0119] In one implementation, the process is repeated for an entire portion of the input feature map in order to generate an appropriate and corresponding portion of the output feature map.

[0120] Therefore, in one implementation, the processor reads data of (all) data elements of the input feature map portion from memory in an appropriate order over and within the entire portion of the input feature map in the manner discussed above, thereby generating a corresponding output feature map portion (and repeats the above steps for each data element within that portion of the input feature map until the entire portion of the input feature map has been read from memory and processed).

[0121] As described above, in the techniques described herein, neural network processing includes processing feature maps on a part-by-part basis. Therefore, once the above steps have been performed to read and process a first part of the input feature map from memory, they can (and in one embodiment) be performed again when reading and processing the next part of the input feature map from memory. Thus, in one embodiment, the process is then repeated for each part of the (overall) input feature map until the entire input feature map has been read from memory and processed on a part-by-part basis.

[0122] Accordingly, the output feature map is generated by generating a corresponding portion of the output feature map from a corresponding portion of the input feature map. The neural network processing performed on and in connection with a portion of the input feature map can include any suitable and desired neural network processing for generating a portion of the output feature map from a portion of the input feature map. Thus, in one embodiment, the neural network processing includes neural network processing layers, such as convolutional layers or pooling layers. In one embodiment, the neural network processing includes applying one or more filters (e.g., weights) to that portion of the input feature map to generate a portion of the output feature map. Therefore, in the latter case, the processor reads data elements from the portion of the input feature map to which the appropriate filters are applied within an appropriate window (kernel), thereby generating one or more corresponding output feature map data elements.

[0123] The neural network processing can be part of an inference or classification process. Therefore, a portion of the output feature map may include data that can be used (directly or after processing by one or more additional neural network layers) for inference or classification. This inference or classification may include inferring or classifying features present in the input data used for neural network processing (e.g., recognition of objects in an image or speech in a sound clip, or other useful outputs inferred from the input data). Thus, neural network processing performed by the techniques described herein can contribute to producing useful outputs.

[0124] These input and output feature map portions can be configured as needed, but in one implementation, each portion of the input feature map (and the corresponding output feature map) is rectangular. The input feature map (and the corresponding output feature map) may include multiple columns and / or rows. Each portion of the input feature map can be processed by neural network processing to provide the corresponding portion of the output feature map. Therefore, the output feature map may have the same number of portions as the input feature map, and in one implementation, the same number of rows and / or columns as the input feature map.

[0125] In embodiments of the technology described herein, the input feature map (and correspondingly, the output feature map) comprises two column portions, each of which may include multiple row portions. However, the input feature map (and correspondingly, the output feature map) may (and does in embodiments) include portions with more than two columns.

[0126] For example, depending on the neural network operation (e.g., a filter) applied to the input feature map portion to generate the output feature map portion, the input feature map portion may not necessarily have the same size as the output feature map portion.

[0127] The size of any particular (or each) input feature map portion may depend on the data that can be used (in the working (buffer) memory) to process the input feature map portion (e.g., depending on the available data that forms the filter margin). Therefore, the (horizontal and / or vertical) dimensions of the input feature map portions may differ from one another.

[0128] A portion of the input feature map can be read from memory and then processed according to neural network processing. A portion of the input feature map can be read from memory and processed according to neural network processing in a specific (e.g., predetermined) order (such that corresponding portions of the output feature map are generated according to the order).

[0129] In one implementation, the input feature map comprises multiple rows and / or columns (in one implementation, two columns, however other numbers of columns are also possible), and portions of the input feature map are read from memory and processed row by row (such that corresponding portions of the output feature map are generated row by row). In this case, the first portion of the input feature map to be processed may be the first (leftmost) portion of the first (topmost) row of the portions forming the input feature map. The portions forming the first (topmost) row can be processed sequentially (from left to right along the row). Once the portions forming the first row have each been processed, the next row portion can be processed, and so on, until the entire input feature map has been processed.

[0130] In this context, when reading and processing portions of the input feature map line by line, in one implementation, one or more tiles in each portion comprise one or more columns of tiles, where tiles within the same column have the same width. As mentioned above, in this case, it may be appropriate to manage the memory (from which portions of the input feature map are read) such that the memory scrolls in both the horizontal (x) and vertical (y) directions, and introduces a vertical offset when scrolling in the horizontal direction.

[0131] In an alternative implementation, the input feature map comprises multiple rows and / or columns (in one implementation, two columns, however other numbers are also possible), and portions of the input feature map are read from memory and processed column by column (such that corresponding portions of the output feature map are generated column by column). In this case, the first portion of the input feature map to be processed may be the first (top) portion of the first (leftmost) column forming the portion of the input feature map. The portions forming the first (leftmost) column can be processed sequentially (from top to bottom along the column). Once the portions forming the first column have each been processed, the next column portion can be processed, and so on, until the entire input feature map has been processed.

[0132] In this context, when reading and processing portions of the input feature map column by column, in one implementation, one or more tiles in each portion comprise one or more rows of tiles, where tiles within the same row have the same height. As mentioned above, in this case, it may be appropriate to manage the memory (from which portions of the input feature map are read) such that the memory scrolls in both the horizontal (x) and vertical (y) directions, and introduces a horizontal offset when scrolling in the vertical direction.

[0133] As described above, the number of tiles defined to allow reading a portion of the input feature map from memory can depend on the distribution of that portion of the input feature map's data in memory. For example, there may be enough space in memory for a first portion of the input feature map to be stored as a set of memory locations, from which that first portion can be read by defining only one (single) tile. However, subsequent portions may require multiple tiles to read the data for that portion (e.g., if the data for that portion is distributed in memory in a way that preserves the required filter margin when performing neural network processing). Therefore, the parameters defining the tiles to be used (and defining the mapping from locations within each tile to memory locations) in one implementation is an updated set of each portion of the input feature map to be read from memory.

[0134] Therefore, in one embodiment, operations are performed on multiple consecutive input feature map portions in a manner described herein, wherein each portion has its own defined layout of one or more tiles and a corresponding mapping from tile locations to memory locations. In one embodiment, at least some of the input feature map portions have a different number of tiles than each other.

[0135] The above discussion covers defining and using input feature maps piece by piece using the techniques described in this paper.

[0136] The applicant has recognized that, for example, in cases where cascading operations may exist, corresponding operations can be used when writing output feature maps to memory in parts, where the output feature map from one neural network layer will serve as the input feature map for the next neural network layer.

[0137] Similarly, when writing a portion of the output feature map to memory, using the tiles in the manner described herein can benefit from using less (working) memory to store (a portion) of the output feature map (e.g., while preserving the desired filter margin).

[0138] Therefore, in one implementation, the operations discussed above concerning the processing of input feature maps for neural network processing, performed in the manner described herein, are accordingly used to process and generate and store output feature maps during neural network processing.

[0139] Therefore, in one embodiment, the method of the technology described herein further includes (and the processor and data processing system are configured to):

[0140] When performing neural network processing to generate a portion of an output feature map from a corresponding portion of an input feature map, the portion of the output feature map includes an array of data elements, each data element having corresponding data associated with it, and each data element having an associated position within the output feature map;

[0141] The portion of the output feature map is defined based on one or more tiles, wherein each tile corresponds to a corresponding region of the portion of the output feature map, and each region corresponds to one or more positions of data elements within the data element array of the portion of the output feature map, wherein the tiles together form the entire portion of the output feature map;

[0142] For each tile, information is provided that allows each location forming the tile to be mapped to a corresponding memory location for storing data associated with the data element at that location; and

[0143] When writing data corresponding to the data element of the portion of the output feature map to memory, the processor: identifies which of the one or more defined blocks include the location; identifies the memory location to which the data of the location is written using provided information that allows each location forming the block to be mapped to a corresponding memory location; and writes the data to the identified memory location.

[0144] The techniques described in this article also extend to such output feature map operations themselves.

[0145] Therefore, embodiments of the technology described herein include a method for controlling a processor operable to perform neural network processing in a data processing system, the data processing system including a processor operable to perform neural network processing and a memory for storing data to be used during the execution of neural network processing, the method comprising:

[0146] Perform neural network processing to generate corresponding portions of an output feature map, each portion of the output feature map comprising an array of data elements, each data element having corresponding data associated with it, and each data element having an associated position within the output feature map; and

[0147] When performing neural network processing to generate a portion of the output feature map:

[0148] The portion of the output feature map is defined based on one or more tiles, wherein each tile corresponds to a corresponding region of the portion of the output feature map, and each region corresponds to one or more positions of data elements within the data element array of the portion of the output feature map, wherein the tiles together form the entire portion of the output feature map;

[0149] For each tile, information is provided that allows each data element location forming the tile to be mapped to a corresponding memory location for storing data of the data element associated with that location;

[0150] The method further includes:

[0151] When storing data of the data element location within the portion of the output feature map in memory, the processor identifies which tile among the one or more defined tiles the location falls into, uses the provided information to identify the memory location where the data of the location is stored, the information allowing each location forming the tile to be mapped to a corresponding memory location, and storing the data at the identified memory location.

[0152] The implementation scheme of the technology described herein includes a data processing system, which includes:

[0153] A processor, operable to perform neural network processing; and

[0154] A memory for storing data generated during neural network processing;

[0155] in:

[0156] The processor is configured to:

[0157] Perform neural network processing to generate corresponding portions of an output feature map, each portion of the output feature map comprising an array of data elements, each data element having corresponding data associated with it, and each data element having an associated position within the output feature map; and

[0158] The data processing system further includes a processing circuit, which is configured to:

[0159] Define a portion of an output feature map to be generated from one or more tiles, wherein each tile corresponds to a corresponding region of the portion of the output feature map, each region corresponds to one or more locations of data elements within the data element array of the portion of the output feature map, wherein the tiles together form the entire portion of the output feature map; and

[0160] For each tile, information is provided that allows each data element location forming the tile to be mapped to a corresponding memory location for storing data of the data element associated with that location;

[0161] and

[0162] The processor is further configured to:

[0163] When performing neural network processing to generate a portion of the output feature map:

[0164] When storing the data element location data within the portion of the output feature map into the memory, it is identified which of the one or more defined tiles the location falls into, and the memory location where the data of the location is stored is identified using the provided information, which allows each location forming the tile to be mapped to a corresponding memory location, and the data to be stored at the identified memory location.

[0165] As those skilled in the art will understand, implementations of the technology described herein may (and in one implementation, certainly) include any one or all of the features of the technology described herein.

[0166] For example, in one embodiment, the tiles of the output feature map portion are defined in the manner discussed above regarding the tiles of the input feature map portion. Accordingly, information that allows mapping data locations within the tiles of the output feature map to memory locations for storing data locations is provided in one embodiment in the manner discussed above regarding the input feature map portion. In one embodiment, the processor uses this information to map the output feature map tile locations to memory locations in the manner discussed above for reading the input feature map portion, thereby determining the memory locations in the output feature map portion for storing data element locations.

[0167] Generally speaking, in the case of processing the output feature map portion (and storing the data elements of the output feature map portion into memory), (and in one embodiment) any of the above-described features regarding the processing of the input feature map portion can be used and applied accordingly.

[0168] The neural network processing that generates this portion of the output feature map may include a neural network processing layer. This neural network processing layer may be "cascaded" with a subsequent neural network processing layer, as described above, such that both neural network processing layers are executed on a partial basis. Therefore, the portion of the output feature map that is generated and written to memory can be (at least partially) used as a portion of the input feature map for use in subsequent neural network processing layers.

[0169] Accordingly, once the above steps have been performed to write the first part of the output feature map into memory, the above steps can be performed again when the next part of the output feature map is written into memory, and so on, until the entire output feature map has been generated.

[0170] As described above, the neural network processing of the techniques described herein may include performing a first neural network processing layer and a next (second) neural network processing layer on a partial basis (such that the first and second layers are "cascaded"). Therefore, the neural network processing may include generating and storing a portion of the output feature map by the first processing layer in a manner described herein, and then reading at least some data from the data written to memory by the next (second) processing layer in a manner described herein as a portion of the input feature map of the next (second) processing layer (and so on, for further processing layers).

[0171] When a first neural network processing layer and a second (next) neural network processing layer are "cascaded," the first and second neural network processing layers can be executed sequentially, such that the second neural network processing layer reads data from memory a portion of the input feature map that was written to memory when a portion of the output feature map was generated by the first (previous) neural network processing layer. For example, when the first (previous) layer generates a portion of the output feature map (which will be used as part of the input feature map by the second layer) and writes it to memory, the second layer can begin reading a portion of the input feature map from memory. Alternatively, once the first (previous) layer has completed generating and writing a portion of the output feature map to memory (which will be used as part of the input feature map by the second layer), the second layer can begin reading a portion of the input feature map from memory (only).

[0172] Parts of the input and output feature maps can be stored in any suitable and desired memory of the neural network processor and / or in memory accessible to the neural network processor. In one embodiment, they are stored in the processor's "local" working memory. In one embodiment, a portion of memory (buffer) is allocated for storing feature map parts, and this buffer is, in one embodiment, a "cascaded" buffer, for storing parts of the output feature maps generated by the first neural network processing layer, which are then to be used as parts of the input feature maps for subsequent neural network processing layers.

[0173] In one implementation, the size of the "cascaded" buffer is smaller than the output feature map to be generated by the first neural network processing layer (and thus smaller than the input feature map to be processed by subsequent neural network processing layers), such that the cascaded buffer is insufficient to store the entire input (or output) feature map. In one implementation, the cascaded buffer has a size capable of storing a portion of the entire (input or output) feature map. In another implementation, the cascaded buffer has a size capable of storing a portion of the entire (input or output) feature map as well as at least a portion of another (input or output) feature map (e.g., corresponding to filter margin).

[0174] In one implementation, the "cascaded" buffer is configured and operated as a scrolling buffer that scrolls in both the horizontal (x) and vertical (y) directions. Therefore, when data is written to the "cascaded" buffer, upon reaching the horizontal or vertical edge of the buffer, the data is scrolled to the next data position to be written to the opposite horizontal or vertical edge, respectively. In one implementation, there is a horizontal offset when scrolling vertically, or a vertical offset when scrolling horizontally.

[0175] It is possible to control and trigger neural network processing performed in any suitable and desired manner, in accordance with the techniques described herein. In one implementation, this is accomplished using an appropriate set of commands (instructions) and other data included in a command stream provided to and executed by the processor, thereby controlling and triggering the operation in accordance with the techniques described herein. Thus, the processor performing the neural network processing executes the command stream to perform the neural network processing.

[0176] The command stream can be provided by any suitable and desired component or processor. In one implementation, the command stream is provided and generated by a driver of a neural network processor, for example, executed on a host processor (such as the central processing unit (CPU) of an overall data processing system). The instruction set forming the command stream can be stored in memory accessible to the processor performing the neural network processing.

[0177] In one implementation, the command stream includes appropriate commands (instructions) that, when executed by the neural network processor, cause the neural network processor to perform desired neural network processing (including reading portions of the input feature map from memory for processing and / or writing portions of the output feature map to memory, and processing portions of the input feature map to generate corresponding portions of the output feature map). The commands (instructions) of the command stream may include commands (instructions) that set various parameters (such as those described above) to be used when the neural network processor performs neural network processing. Parameters set by the commands (instructions) in the command stream may include, for example, parameters for defining one or more patches for a portion of the feature map and / or parameters for providing information that allows locations within that portion of the feature map to be mapped to corresponding locations in memory.

[0178] In one implementation, the parameters set by the commands (instructions) in the command stream include one or more parameters from which the size (height and / or width) of each defined tile can be derived, and one or more parameters indicating a reference memory address corresponding to the reference position of each defined tile.

[0179] As described above, the processor can be configured to determine the relative position of a data element within a tile based on information indicating the tile size, and then determine the location of the data (relative to the base memory address of the tile) in memory by scaling the relative position by any appropriate span.

[0180] Therefore, other information that can be used by the processor (e.g., parameters) includes the various spans discussed above. In one embodiment, information about the span is set by means of appropriate instructions in the command flow, but other arrangements are possible if desired. As mentioned above, in the embodiment, the information about the span is common (the same) for all defined blocks of a specific portion of the feature map (however, the information about the span may differ for different feature maps and / or feature map portions). Therefore, information about the span can be set by means of appropriate instructions for each portion of the feature map to be processed.

[0181] The processor performing neural network processing can be an accelerator or processor (neural network accelerator / processor (NNA)) specifically configured for (or dedicated to) performing neural network processing. Alternatively, the processor may include any processor suitable for performing neural network processing, such as a microcontroller unit (MCU), central processing unit (CPU), graphics processing unit (GPU) (graphics processor), video processor, audio processor, image signal processor (ISP), or digital signal processor.

[0182] The data processing system can be implemented as part of any suitable electronic device that may require neural network processing, such as a desktop computer, portable electronic device (e.g., a tablet computer or mobile phone), or other electronic device. Therefore, the technology described herein also extends to electronic devices that include a data processing system (and on which the data processing system operates in accordance with the technology described herein). In one embodiment, the data processing system of the present invention can be implemented as part of a portable electronic device (such as a mobile phone, tablet computer, or other portable device).

[0183] A data processing system may include any desired components and elements that a data processing system can include, such as one or more or all of the following: a display processing unit (display processor), a central processing unit (CPU), a graphics processing unit (GPU) (graphics processor), a video processor, a digital signal processor, one or more neural network processors, and a display.

[0184] These processors can be placed on a system-on-a-chip.

[0185] The data processing system may include one or more memories (such as the memories described above) for storing the data described herein and / or storing software for performing the processes described herein, and / or communicate with said one or more memories. As described above, the data processing system may communicate with a host microprocessor and / or with a display for displaying output data associated with neural network processing.

[0186] The memory may include one or more local memories that can be located on the chip. The local memory may include one or more buffers (e.g., "cascaded" buffers).

[0187] The memory may also include main memory, which can be external memory that can be located off-chip. The main (external) memory can be any suitable type of memory, such as SDRAM.

[0188] The various functions of the technology described herein can be performed in any desired and suitable manner. For example, the functions of the technology described herein can be implemented in hardware or software as needed. Thus, for example, the various functional elements of the technology described herein may include one or more suitable processors, one or more controllers, functional units, circuits, processing logic units, microprocessor arrangements, etc., which are operable to perform various functions, such as appropriate dedicated hardware elements (processing circuits) and / or programmable hardware elements (processing circuits) that can be programmed to operate in a desired manner.

[0189] It should also be noted here that, as those skilled in the art will understand, the various functions of the techniques described herein can be copied and / or executed in parallel on a given processor. Similarly, various processing circuits can share processing circuits if needed.

[0190] Those skilled in the art should also understand that all embodiments of the technology described herein may include any one or more of the features described herein, as appropriate.

[0191] The methods described herein can be implemented at least in part using software, such as computer programs. Therefore, it should be understood that, when viewed from another embodiment, the techniques described herein include: computer software, which, when installed on a data processor, is particularly suitable for performing the methods described herein; a computer program element including computer software code portions for performing the methods described herein when the program element is run on a data processor; and a computer program including code adapted to perform the methods described herein or all steps of the methods when the program is run on a data processing system.

[0192] The techniques described herein also extend to computer software carriers that include such software, which, when used to operate a data processing system, causes a processor or system to perform the steps of the methods described herein. Such computer software carriers can be physical storage media, such as ROM chips, CD-ROMs, RAM, flash memory, or disks, or they can be signals, such as electronic signals transmitted through wires, optical signals, or radio signals, such as signals to satellites.

[0193] It should also be understood that not all steps of the methods described herein need to be performed by computer software, and thus, from another broad implementation, the techniques described herein include computer software and such software installed on a computer software carrier for performing at least one step of the methods described herein.

[0194] Therefore, the techniques described herein can be suitably embodied as a computer program product used with a computer system. Such implementations may include a series of computer-readable instructions fixed on a tangible, non-transitory medium, such as a computer-readable medium, for example, a disk, CD-ROM, ROM, RAM, flash memory, or hard disk. It may also include a series of computer-readable instructions that can be transmitted to a computer system via a modem or other interface device through a tangible medium (including, but not limited to, optical or analog communication lines) or passively using wireless technologies (including, but not limited to, microwave, infrared, or other transmission technologies). This series of computer-readable instructions embodies all or part of the functions previously described herein.

[0195] Those skilled in the art will understand that such computer-readable instructions can be written in a variety of programming languages ​​to be used with many computer architectures or operating systems. Furthermore, such instructions can be stored using any current or future memory technology (including, but not limited to, semiconductor, magnetic, or optical technologies), or transmitted using any current or future communication technology (including, but not limited to, optical, infrared, or microwave technologies). It is conceivable that such computer program products can be distributed as removable media with accompanying printed or electronic documentation (e.g., shrink-wrapped software), pre-loaded onto a computer system (e.g., system ROM or a fixed disk), or distributed from a server or electronic bulletin board via a network (e.g., the Internet or the World Wide Web).

[0196] Neural network processing typically involves multiple processing layers, where each layer performs operations on the input feature map to generate an output feature map, such as... Figure 1 and Figure 2 As shown, and as stated above.

[0197] Each neural network processing layer can process the entire input feature map and correspondingly generate and write out the entire output feature map to be used as the input feature map for the next layer. However, this will require a significant amount of working memory to store the entire feature map between processing layers (especially since the size of the data array may need to be changed depending on the processing performed by a particular layer).

[0198] For example, Figure 3 This illustrates how the size of the feature map can change due to the various processing layers of the neural network. Specifically, in Figure 3 In this process, the feature map FM0 with initial size H×W×C is processed according to the layer performing operation A to generate an output feature map FM1 with size H×W×2C. This output feature map is then used as the input feature map for the next processing layer, which performs operation B to generate a corresponding output feature map FM2 with size H / 2×W / 2×2C. The applicant has recognized that, in this case, it may be advantageous to avoid storing feature maps that appear at least between consecutive operations A and B and have an overall size of H×W×2C.

[0199] Therefore, the applicant has recognized that it may be advantageous to process the feature map into multiple parts smaller than the entire feature map. Specifically, the first operation (layer) can perform processing of the corresponding input feature map part by part by processing the corresponding parts of the input feature map sequentially (e.g., until the entire input feature map has been processed). When processing a part of the input feature map, the first operation (layer) can generate the corresponding part of the output feature map and write it into a suitable buffer. Then, the successive operations (layers) can perform processing using at least partially the parts of the output feature map that have been written into the buffer. In this way, the successive operations (layers) also process the feature map part by part, because the parts of the output feature map from the first operation (layer) are written into the buffer and are therefore available for processing according to the successive operations (layers). In this way, it is not necessary to store the entire feature map between the first and second operations, and thus a smaller amount of memory can be provided than would be required to store the entire feature map between the first and second operations.

[0200] Such an arrangement, in which the feature map is processed through sequential operations, is suitable for, for example... Figure 3 The situation illustrated may be particularly useful in which the first operation (operation A) increases the size of the feature map (to W×H×2C) compared to the input feature map (which has dimensions W×H×C) received for processing via operation A. Therefore, part-based processing may be particularly useful when the feature map existing between two consecutive operations (layers) is larger than the feature map before (and / or after) the consecutive operations (layers).

[0201] As described above, a feature map may include an array of data elements, each data element having a position within the feature map (which may be defined as its x and y positions within the feature map), and each data element having associated data. A portion of the feature map similarly includes an array of data elements (where this portion of the feature map has an array of data elements smaller than the array of data elements forming the entire feature map (i.e., a subset thereof)). Therefore, a portion of the feature map includes an array of data elements, each data element having a position within that portion of the feature map (which may be defined as its x and y positions within that portion of the feature map), and each data element having associated data.

[0202] Figure 4 This illustrates one possible way to divide a feature map into multiple parts. This feature map could be an input feature map to which a neural network processing layer is applied, or an output feature map generated as a result of a neural network processing layer.

[0203] exist Figure 4In the example, the feature map is divided into sections including "Strip 0", "Strip 1", and "Strip 2". In this example, each section is rectangular and has the width and height of multiple data elements corresponding to the width (x-direction) and height (y-direction) of the feature map. Figure 4 In the example shown, each part has a width equal to the width W of the feature map and a height less than the height H of the feature map. However, due to Figure 4 The portion shown includes "strips" that extend across the entire width of the feature map, so a relatively large amount of memory may still be needed to store these portions of the feature map.

[0204] Figure 5 An alternative form of subdividing a feature map into multiple segments (“strips”) is shown. Specifically, the input feature map 500 used for the first operation (operation A) is divided into segments A0 to A7, each segment being rectangular and having a width W and a height H smaller than those of the input feature map 500, respectively. Figure 5 In the example shown, each of sections A0 through A7 has the same height and the same width. Alternatively, these sections can have different heights and widths if desired. Figure 5 In the example shown, feature map 500 is divided into two columns, each containing multiple rows. Alternatively, the feature map can be divided into more than two columns if desired.

[0205] Operation A will process each of portions A0, A1, A2, A3, A4, A5, A6, and A7 sequentially to generate the corresponding portion of the output feature map stored in buffer 502 (also referred to herein as the "cascaded buffer"). Figure 5 In the example shown, these parts are arranged such that processing is performed line by line, with parts A0 and A1 forming the first line, parts A2 and A3 forming the next line, and so on.

[0206] like Figure 5 The assigned feature map shown allows for a relatively small amount of space to be reserved in buffer 502 (compared to using, for example) Figure 4 The working memory (compared to the entire width strip shown) is used to store a portion of the output feature map generated when performing operation A.

[0207] The next neural network processing layer, including the second operation (operation B), can use the output feature map from operation A as its input feature map. Operation B can be "cascaded" with the first operation (operation A) because operation B uses a portion of the output feature map generated by operation A to perform processing (instead of waiting until operation A has generated the entire output feature map). Operation B can read a portion of its input feature map from the cascade buffer and generate the corresponding portions B0 to B7 of the output feature map 501. Figure 5 As shown, portions B0 to B7 can each be rectangles and have a width W' and height H' smaller than those of the output feature map 501. Figure 5 In the example shown, each of sections B0 through B7 has the same height and the same width. Alternatively, these sections can have different heights and widths if desired. Figure 5 In the example shown, feature map 501 is divided into two columns, each containing multiple rows. Alternatively, the feature map can be divided into more than two columns if desired.

[0208] exist Figure 5 In the example shown, the number of parts processed by operation B is the same as the number of parts processed by operation A (i.e., the number of rows and columns of the parts is the same). However, the size of parts B0 to B7 may differ from the size of parts A0 to A7, depending on whether, for example, operation A or operation B causes a change in the size of the feature map.

[0209] Furthermore, each portion read from buffer 502 for processing by operation B is not necessarily identical to the portion previously output by operation A. For example, if operations A and B include filtering operations (which are typically used in neural network processing), the portions written by operation A and read by operation B may differ due to consideration of filter margin.

[0210] Alternatively, the feature map processed by the first operation A can be as follows: Figure 14 The allocation is shown, where portions A0 to A7 form two columns. In this case, these portions are arranged such that processing is performed column-by-column, with portions A0 to A3 forming the first column and portions A4 to A7 forming the second column. Any subsequent "cascaded" operations based on similarly performed portions will generate an output feature map including the same arrangement of the two columns.

[0211] Figure 6 It is shown, for example, according to Figure 5 A system diagram of an implementation of a data processing system and data flow for processing feature maps is provided, wherein operations A and B are "cascaded". The system includes a CPU 600 and a neural network processing unit (NPU) 604. The NPU 604 may include a processor specifically configured to perform neural network processing (e.g., operations for performing neural network processing layers). Alternatively, the NPU 604 may include any other suitable processor capable of performing neural network processing, such as the CPU itself or a graphics processing unit (GPU).

[0212] Figure 6The system also includes a memory 601 for storing a command stream 603. This command stream includes a sequence of instructions for controlling the operation of the NPU 604, and this sequence of instructions will be generated by the CPU 600 (e.g., by a driver 606 of the NPU 604, which executes on the CPU 600). The memory 601 also stores processing parameters 602 for neural network processing. The processing parameters 602 may include, for example, weights of operations to be performed by the NPU 604 (e.g., weights of filter operations). Figure 6 The memory 601 in the diagram is shown as flash memory; however, other types of memory, such as dynamic random access memory (DRAM), may be used if needed.

[0213] Figure 6 The system also includes a “working” memory 605 for storing feature maps (or portions thereof) associated with neural network processing. The “working” memory 605 is shown as static random access memory (SRAM); however, other suitable types of memory may be used if desired. For example, memory 601 and working memory 605 may be components of the same physical memory if desired. One or more components of working memory 605 may be provided for storing entire feature maps (such as feature maps 500 and 501). A buffer 502 (referred to herein as a “cascaded buffer”) is stored in the working memory for storing portions of the feature map between consecutive “cascaded” operations. Buffer 502 should be stored in memory that allows relatively fast access, such as SRAM. Buffer 502 may not be sufficient to store the entire feature map.

[0214] When performing "cascaded" operations A and B, the data flow typically occurs when the NPU 604 receives instructions or other commands from the CPU to begin neural network processing. Figure 6 The process begins with step 1). The NPU 604 then accesses (e.g., reads from memory) the command flow 603 (step 2) associated with performing neural network processing.

[0215] exist Figure 6 In this process, the neural network processing to be performed includes operations A and B, which form successive processing layers. According to the instructions in command stream 603, NPU 604 reads a portion (“strip”) of input feature map 500 (step 3), processes the portion according to operation A, and writes the corresponding portion of output feature map into buffer 502 (step 4).

[0216] When operation B is executed, NPU 604 reads a portion of the input feature map for operation B from buffer 502 (step 5), wherein the portion read includes data previously output to buffer 502 due to the execution of operation A (step 5). NPU 604 processes the portion of the input feature map according to operation B, and then writes the corresponding portion of the output feature map 501 into working memory 605 (step 6).

[0217] It is possible to process portions of the feature map in this way according to operation A and operation B until the entire feature input feature map 500 for operation A has been processed (and the entire output feature map 501 from operation B has been generated accordingly).

[0218] exist Figure 6 In the illustrated implementation, only operations A and B are "cascaded". Before operation A, the entire input feature map 500 is stored in working memory 605, and after operation B, the entire output feature map 501 is stored in working memory 605. In contrast, buffer 502 does not need to store the entire output feature map generated by operation A (corresponding to the entire input feature map of operation B).

[0219] It should be understood that the aforementioned "cascading" is not limited to two operations A and B corresponding to the processing of two layers of a neural network. In practice, multiple (e.g., two or more) consecutive operations (layers) can be "cascaded," such that each operation (layer) processes the feature map partially, one by one. In such an arrangement, multiple buffers ("cascaded buffers") can be provided, where each cascaded buffer stores a portion of the feature map between consecutive operations (layers). In this case, the feature map can be stored completely only before the first operation of the multiple consecutive operations and after the last operation of the multiple consecutive operations.

[0220] Figure 11A This is a flowchart illustrating the processing of a portion of the input feature map based on operation A. Figure 11B This is a flowchart illustrating the processing of a portion of the input feature map according to operation B, which can be compared with, for example, relative to... Figure 5 and Figure 6 The operation A described above is "cascaded".

[0221] from Figure 11AAs can be seen, once processing of a portion of the input feature map begins (step 1101), the first position to be processed by operation A is identified by setting the x and y positions within that portion of the input feature map to zero (step 1102). Therefore, in this example, the first position to be processed is the top-left position within that portion of the feature map. The data corresponding to the identified position is then processed according to operation A (step 1103) to generate data for the corresponding position in a portion of the output feature map, where the generated data is subsequently written to a cascaded buffer.

[0222] As can be seen from steps 1103 to 1106, the identified positions (and the positions where processing is performed) gradually increase until the entire portion of the input feature map has been processed. In this example, processing is performed row by row from left to right. Specifically, the identified positions increase along the row (in the x-direction) until the edge of that portion of the input feature map is reached, and then the next row is selected (by setting x = 0 and incrementing the y-position). This continues until the bottom edge of that portion of the input feature map is reached.

[0223] according to Figure 11A The processing can be repeated for each part of the input feature map until the entire input feature map has been processed according to operation A.

[0224] Operation B can process a portion of the corresponding input feature map, such as Figure 11B As shown in the figure, once this portion of the input feature map begins to be processed (step 1109), it can be similar to... Figure 11A The method described sequentially identifies positions within that portion of the input feature map and can be processed according to steps 1110 to 1114. However, the difference lies in that, at step 1111, the application operation B will include reading data corresponding to the identified positions from the cascaded buffer, and performing operation B to generate data for the corresponding positions in the output feature map, which can be written to any suitable memory.

[0225] according to Figure 11B The processing can be repeated for each part of the input feature map until the entire input feature map has been processed according to operation B.

[0226] As described above, for “cascaded” operations A and B, the portion of the input feature map used when performing operation B includes at least some data from the data already written into the cascade buffer when the portion of the output feature map was generated based on operation A.

[0227] After operation A completes processing of a portion of the input feature map (to generate a portion of the output feature map for use by operation B), the NPU 604 can perform processing on the portion of the input feature map via operation B. In such implementations, once completed according to... Figure 11A As part of the flowchart processing, the NPU 604 therefore executes according to... Figure 11B The processing of a portion of the flowchart.

[0228] Alternatively, the NPU 604 does not need to wait until operation A has completed processing a portion of the input feature map before starting processing according to operation B. For example, it can use... Figure 11C The flowchart shown illustrates that the processing of operations A and B is performed in parallel, such that once operation A has completed processing at each position in its corresponding input feature map, operation B performs processing at the same position in its corresponding input feature map.

[0229] A potential difficulty in processing feature maps piece by piece is how to handle the processing of the individual parts that will be written to memory (or read from memory).

[0230] For example, as described above, when "cascading" operations A and B consecutively, it may be advantageous to provide a relatively small cascading buffer 502, insufficient in size to store the entire output feature map generated by operation A (which forms the input feature map for operation B). In this case, portions of the output feature map generated by operation A that are later written into the cascading buffer during neural network processing may need to overwrite portions previously written into that cascading buffer. However, if operation A or B includes a filtering operation, portions of the output feature map generated by operation A may need to be written into the cascading buffer in a manner that preserves any filter margin that may be required when performing processing according to operation B. For example, to preserve filter margin, it may be necessary to write newly generated portions of the output feature map into the cascading buffer to avoid overwriting at least the last of the previous output feature map portions already written into that cascading buffer. Therefore, if only a relatively small cascading buffer is provided, it may be desirable to manage the cascading buffer such that the data of the newly generated output feature map portions "fit" at least some of the data from the preceding portions. The applicant has recognized that, in this case, data from the output feature map portion can be efficiently written to (and read from) the input feature map portion using one or more tiles, each tile being mapped to a corresponding set of memory locations. Exemplary cases for using such tiles will be discussed in more detail below.

[0231] Therefore, according to the technique described herein, when performing neural network processing based on a portion, a portion of the output feature map to be written to memory (or a portion of the input feature map to be read from memory) is described according to one or more patches. Each patch corresponds to a region of that portion of the feature map, which includes one or more consecutive locations of the data elements forming that portion of the feature map.

[0232] It also provides information that allows the data locations that form each tile (within each tile) to be mapped to the corresponding memory locations, thereby allowing data corresponding to each location to be written to (or read from) memory (cascaded buffer).

[0233] Figure 7 A possible definition of a portion of feature map 700 based on one or more blocks is shown. Figure 7 In the example shown, four tiles can be defined as tile 0, tile 1, tile 2, and tile 3, where each tile can be used to describe a region of that part of the feature map, which includes the consecutive positions of data elements in the data element array that forms part of the feature map. Figure 7 In the example shown, each tile is rectangular. Each tile has a width and height that correspond to the numerical values ​​of the data element's position in the width or height direction, respectively.

[0234] exist Figure 7 In the example shown, the tiles consist of two columns of tiles, where tiles within the same column have the same width. In this case, tile 0 and tile 2 have the same width (which is WIDTH0), and similarly, tile 1 and tile 3 have the same width. However, tiles in different columns can have different widths. Tiles within a column can have different heights. Tiles in different columns can also have different heights. For example, in Figure 7 In the example shown, each of the tiles 0, 1, 2, and 3 has a different height.

[0235] Each tile has a reference position (origin) (see...) Figure 7 (BASE0, BASE1, BASE2, BASE3), which correspond to the origin of the x and y coordinate system of the tile. Figure 7 In the example shown, each base position corresponds to the top left corner of the tile.

[0236] Figure 16The diagram illustrates an alternative way to define blocks. In this case, there are four blocks: block 0, block 1, block 2, and block 3. These blocks form rows, where blocks within the same row have the same height. In the example shown, block 0 and block 1 have the same height (HEIGHT0), and similarly, block 2 and block 3 have the same height. The widths of blocks within a row can differ. Blocks in different rows can also have different widths. Figure 16 In the example shown, tile 0, tile 1, tile 2, and tile 3 do not have the same width. Figure 16 In the middle, each tile has a similar Figure 7 The reference positions are BASE0, BASE1, BASE2, and BASE3.

[0237] During neural network processing, in embodiments of the invention, information is provided to define a portion of a feature map based on one or more patches. Information is also provided to allow mapping to each location within each patch to a memory location in the cascaded buffer 502. This information is provided via descriptors within command stream 603.

[0238] The information provided for defining each of these tiles includes information that allows determining the height and width of each tile. (e.g.) Figure 7 and Figure 16 As shown, the arrangement of the tiles relative to each other is known or predetermined.

[0239] The height and width parameters for each tile to be used can be set. However, as in this embodiment, the width and height of these tiles have a predetermined relationship relative to each other, so this is not necessary. For example, as in... Figure 7 In the example shown, the tiles in the same column have the same width, and all four tiles can be defined using (only) the width WIDTH0 of tile 0, the height HEIGHT0 of tile 0, the height HEIGHT1 of tile 1, the height HEIGHT of this portion of the feature map, and the width WIDTH of the feature map.

[0240] exist Figure 7 In the example shown, the width of tile 2 is the same as the width WIDTH0 of tile 0 because they are in the same column. The widths of tiles 1 and 3 can be inferred from the difference between the width WIDTH of that portion of the feature map and the width WIDTH0 of tile 0. The height of tile 2 can be inferred from the difference between the height HEIGHT of that portion of the feature map and the height HEIGHT0 of tile 0. The height of tile 3 can be inferred from the difference between the height HEIGHT of that portion of the feature map and the height HEIGHT1 of tile 1.

[0241] exist Figure 16 In the example shown, since the tiles are arranged in rows, the tiles are defined using (only) the height HEIGHT0 of tile 0, the width WIDTH0 of tile 0, the width WIDTH2 of tile 2, the width WIDTH of that portion of the feature map, and the height HEIGHT of that portion of the feature map. The width of tile 1 can be inferred from the difference between WIDTH and WIDTH0, and the height of tile 1 is equal to HEIGHT0. The heights of tiles 2 and 3 can be inferred from the difference between HEIGHT and HEIGHT0. The width of tile 3 can be inferred from the difference between WIDTH and WIDTH2.

[0242] Figure 8 A list of descriptors (parameters) is provided in this embodiment that can be used in the command stream to indicate a portion of the input feature map to be read from memory. Specifically, Figure 8 The descriptor provides information based on Figure 7 The arrangement of variables used to define blocks.

[0243] As mentioned above, due to the predetermined relationship between the height and width of these tiles, it is not necessary to explicitly provide the height and width of each tile. Therefore, regarding... Figure 7 The discussion reached a consensus, in Figure 8 In the code, the parameter IFM_HEIGHT is provided for the height of this portion of the input feature map, and the parameter IFM_WIDTH is provided for the width of this portion of the input feature map (where the preceding "IFM..." only identifies the parameter as specific to the input feature map). The parameter IFM_DEPTH (which is the number of channels in the feature map) is also provided for the depth of this portion of the input feature map. Parameters are also provided for the following: the width of patch 0 IFM_WIDTH0, the height of patch 0 IFM_HEIGHT0, and the height of patch 1 IFM_HEIGHT_1. (See also: Regarding...) Figure 7 The height and width of the tiles discussed, which are not explicitly described by parameters, can be inferred from these listed parameters.

[0244] Each of the above parameters corresponding to width, height, or depth indicates the dimension in that direction based on the value of the data element's position in the width (x-direction), height (y-direction), and depth (z-direction) of the block.

[0245] Additional parameters are provided to allow the position of each data element within the tile to be mapped to a corresponding memory location (from which the data element's data is to be read).

[0246] Therefore, as Figure 8As shown, parameters IFM_BASE0, IFM_BASE1, IFM_BASE2 and IFM_BASE3 are provided. These parameters can be used to map the base position of each tile (BASE0, BASE1, BASE2, BASE3 respectively) (top left corner in this case) to the corresponding memory position in the cascade buffer 502.

[0247] It also provides parameters to indicate the span between data elements in the cascaded buffer. This span can be used to convert the position of a data element within a tile relative to the tile's reference position into an offset (space) in memory relative to the memory address mapped to the tile's reference position.

[0248] The span in a specific direction corresponds to the distance between the starting points of adjacent data element locations in the considered direction in memory (in this case, the numerical value of the memory location). For example... Figure 8 As shown, the parameters IFM_STRIDE_X and IFM_STRIDE_Y are provided to indicate the span in the x and y directions, respectively. IFM_STRIDE_C indicates the "block span" and is used when the input feature map comprises multiple channels that are grouped into "blocks" stored in different regions of memory, such that the "block span" indicates the memory address spacing between said blocks. IFM_STRIDE_N indicates the "batch span" that can be used when performing batch processing on multiple input feature maps, where IFM_STRIDE_N indicates the memory address spacing between feature maps in a batch of feature maps being processed.

[0249] The above parameters can be used to read data elements that form part of the input feature map from corresponding locations in memory based on the defined tiles. Specifically, in this embodiment, the location from which data elements are read from memory is determined as follows:

[0250]

[0251] This determination yields the location of the data element within that portion of the feature map to be read from memory (defined by variables n, x, y, and c, where x represents the position of the data element of interest in that portion of the input feature map in the x-direction, y represents the position in that portion of the input feature map in the y-direction, c represents the channel to be read for the data position, and n represents the batch (i.e., the specific input feature map) in which the data position is located).

[0252] Then determine which tile the data element of interest falls into, and the relative position of that data element within that tile.

[0253] In the example above, a variable t is provided to represent the tile under consideration. Initially, the variable t is set such that t = 0, that is, the topmost tile in the leftmost column (tile 0) is identified as a potentially interesting tile. Then it is determined whether x > WIDTH0, and if so, the variable t is incremented by 1. In other words, it is identified whether the data location of interest is located outside the first (leftmost) column with a width WIDTH0 in the x-direction, and if so, the second column (adjacent in the x-direction) is identified as the column of interest by designating the tile under consideration as the topmost tile in the next column (tile 1).

[0254] If it is determined that x > WIDTH0, meaning the data location of interest is outside the first column, then the value of x is adjusted accordingly so that x describes the relative position of the data location of interest within the second column. This is done by setting x = WIDTH0.

[0255] Next, determine if y >= HEIGHT[t], and if so, increment the variable t by 2. In other words, identify if the data location of interest is outside the topmost tile in the relevant column (which has a height HEIGHT[t], where t depends on being previously determined to be 0 or 1), and if so, identify the lower tile in the column of interest, which is tile 2 or tile 3.

[0256] If it is determined that y >= HEIGHT[t], meaning the data of interest is located outside the topmost tile in the relevant column, then the value of y is adjusted accordingly so that y describes the relative position of the data of interest within the relevant tile. This is done by setting y -= HEIGHT[t].

[0257] Once the relevant map block has been identified and the relative positions of the data elements within that block have been determined, the memory location (address) from which the data element is read is determined as follows:

[0258] a = BASE[t] + n * STRIDE_N +

[0259] y*STRIDE_Y+x*STRIDE_X+

[0260] (c / BRICK)*STRIDE_C+(c%BRICK)*ElemSize;

[0261] In other words, the memory location is determined as follows: The reference position (top left corner) of the considered block is determined by the indicated memory (address) location (BASE[t]), and then the offset of the considered block (calculated by n*STRIDE_N), the relative position of the data element within the block (calculated by y*STRIDE_Y+x*STRIDE_X), the channel block into which the considered data element falls (calculated by (c / BRICK)*STRIDE_C), and the channel within the channel block (calculated by (c%BRICK)*ElemSize) are added to that location. Then, the expression "Return*( <type>*)a” reads data from the obtained identified memory location.

[0262] The data read from the identified memory location can then be processed according to the corresponding neural network layer (operation).

[0263] Data can be read from memory in this way and each data element within that portion of the input feature map can be processed, for example, by relative to Figure 11A and Figure 11B The method described is to increment the x and y positions sequentially.

[0264] Figure 9 This demonstrates how to define (specify) a portion of the output feature map within a command flow. Descriptors (parameters) can be used to define features similar to those relative to... Figure 8 The method described defines the map blocks, where the preceding "OFM..." only identifies those parameters as specific to the output feature map.

[0265] Specifically, parameters are provided for the following: the height of the portion of the output feature map (OFM_HEIGHT), the width of the portion of the output feature map (OFM_WIDTH), and the depth of the portion of the output feature map (IFM_DEPTH, which is the number of channels in the feature map). Parameters are also provided for the following: the width of patch 0 (OFM_WIDTH0), the height of patch 0 (OFM_HEIGHT0), and the height of patch 1 (OFM_HEIGHT1).

[0266] The system also provides parameters OFM_BASE0, OFM_BASE1, OFM_BASE2, and OFM_BASE3 for the memory location (address) of the reference position for each tile.

[0267] The parameters OFM_STRIDE_X and OFM_STRIDE_Y are provided to indicate the span between data elements in the cascaded buffer, which have adjacent positions in the x and y directions of that portion of the output feature map, respectively. OFM_STRIDE_C indicates the "block span" and is used when the output feature map comprises multiple channels that are grouped into "blocks" to be stored in different regions of memory, such that the "block span" indicates the memory address spacing between said blocks. OFM_STRIDE_N indicates the memory address spacing between data elements in consecutively output feature maps when performing batch processing of feature maps. The above parameters are used to determine the memory location of data elements written to a portion of the generated output feature map when performing neural network processing in this embodiment, as described below:

[0268]

[0269] The determination first selects the location of the data element within that portion of the feature map to be written to memory (defined by variables n, x, y, c, and value, where x represents the position of the data element of interest in that portion of the output feature map in the x-direction, y represents the position in that portion of the output feature map in the y-direction, c represents the channel where the data location is located, and n represents the batch (i.e., the specific output feature map) in which the data location is located). The variable "value" corresponds to the value of the generated data to be written to memory.

[0270] Then determine which tile the data element of interest falls into, and the relative position of that data element within that tile.

[0271] In the example above, this is accomplished by providing a variable t representing the patch under consideration. Similar to the discussion above relative to the input feature map, the variable t is modified such that it represents the patch in which the data element is located, and the variables x and y are also modified to provide the relative position of the data element within that patch.

[0272] Then determine the memory location where the data element will be written. In this example, the integer "a" represents the memory location where the data will be written, and is calculated as follows:

[0273] a = BASE[t] + n * STRIDE_N +

[0274] Y*STRIDE_Y+x*STRIDE_X+

[0275] (c / BRICK)*STRIDE_C+(c%BRICK)*ElemSize;

[0276] In other words, the memory location is determined as follows: The memory location (address) of the reference position of the considered tile is obtained (BASE[t]), and then the offset of the considered batch (calculated by n*STRIDE_N), the relative position of the data element within the tile (calculated by y*STRIDE_Y+x*STRIDE_X), the channel block into which the considered data element falls (calculated by (c / BRICK)*STRIDE_C), and the channel within the channel block (calculated by (c%BRICK)*ElemSize) are added to that location. Then, the expression "*( <type>*)a = value" writes the data to the identified memory location. When using, such as Figure 16 In other tiled arrangements as shown, similar principles as those discussed above apply when determining the location in memory from which data is read when processing a portion of the input feature map (or the location in memory from which data is written to a portion of the output feature map). For example, the process could similarly include: for a location of interest in that portion of the feature map, identifying which tile the location of interest is located in, determining the relative position of the location of interest within that tile, and then reading data from (or writing data to) the corresponding location in memory, which has an offset relative to a base address of the tile in memory, wherein the offset is determined based on the relative position of the location of interest within the tile and any appropriate span in memory.

[0277] Figure 10 It shows the use of from Figure 8 and Figure 9 An exemplary portion of the command stream of the descriptor, which is used to control the NPU 604 to process portions of the feature map according to cascaded operations A and B. Specifically, Figure 10 The command stream shown includes portions A0 and A1 of the feature map processed by cascading operations A and B to generate output feature maps B0 and B1, similar to... Figure 5 As shown.

[0278] Figure 10 Instruction 1000 includes instructions for setting various parameters to define blocks used when reading a portion A0 of the input feature map from memory for processing according to operation A. Instruction 1001 includes instructions for setting various parameters to define blocks used when writing a portion A0a of the output feature map generated by operation A into a concatenated buffer. The next instruction 1002 is an instruction for performing processing on operation A. When executed, instruction 1002 causes the NPU to read a portion A0 of the input feature map from memory, process the portion A0 to generate a portion A0a of the output feature map, and write the portion A0a into the concatenated buffer. Reading data from memory and writing data into the concatenated buffer utilizes blocks defined as discussed above.

[0279] The next instruction in command stream 1003 includes instructions for setting various parameters to define blocks used when reading a portion B0a of the input feature map from the cascade buffer for processing according to operation B. Instruction 1004 includes instructions for setting various parameters to define blocks used when writing a portion B0 of the output feature map generated by operation B into memory. The next instruction 1005 is an instruction for performing the processing of operation B. When executed, instruction 1005 causes the NPU to read a portion B0a of the input feature map from the cascade buffer, process the portion B0a to generate a portion B0 of the output feature map, and write the portion B0 into memory. Reading data from the cascade buffer and writing data into memory will utilize the defined blocks.

[0280] Instructions 1006, 1007, and 1008 include instructions for: reading the next portion A1 of the input feature map from memory, processing portion A1 according to operation A to generate portion A1a, and writing portion A1a to a cascade buffer. Similar to instructions 1000, 1001, and 1002, instructions 1006, 1007, and 1008 include instructions for defining associated blocks for reading portion A1 from memory and for writing portion A1a to the cascade buffer.

[0281] The next instruction in command streams 1009, 1010, and 1011 includes instructions for: reading a portion B1a of the input feature map from the cascade buffer for processing according to operation B; processing the portion B1a according to operation B to generate a portion B1; and writing the portion B1 into the memory cascade buffer. Similar to instructions 1003, 1004, and 1005, instructions 1009, 1010, and 1011 include instructions for defining a related map block for reading the portion B1a from the cascade buffer and for writing the portion B1 into the cascade buffer.

[0282] The cascade buffer can provide additional instructions for processing part A2 to generate part B2 of the output feature map by cascading operations A and B, and so on for parts A3, B3, A4, B4, etc., until all parts of the input feature map have been processed by operation A (and therefore all parts of the output feature map have been generated by operation B).

[0283] The parameters set to define blocks for reading a portion of the input feature map from memory (or a cascaded buffer) and included in the command stream can vary depending on the number and size of the blocks to be used. This, in turn, depends on how the data for that portion is stored in memory (or a cascaded buffer). Similarly, the parameters set to define blocks for writing a portion of the output feature map to memory (or a cascaded buffer) can vary depending on the number and size of the blocks to be used, which in turn depends on how the data for that portion is stored in memory (or a cascaded buffer).

[0284] For example, a relatively large amount of memory can be provided (allocated) to store the input feature map 500 that will be processed by operation A, such that the input feature map 500 can be stored entirely in memory (as per [reference to...]). Figure 5 (As discussed). Therefore, each of the portions A0 to A7 of the input feature map 500 does not need to be subdivided and distributed across multiple groups of memory locations in order to be "fitted" into the allocated memory. Therefore, only a single tile is needed to read any one of the portions A0 to A7 forming the input feature map 500 from memory. Therefore, Figure 10 Instructions 1000 and 1006 set values ​​only for a single block (block 0) with a base IFM_BAR0, width IFM_WIDTH0, and height IFM_HEIGHT0.

[0285] Similarly, sufficient space can be provided (allocated) in memory to store the output feature map 501 generated during the processing according to operation B, such that the output feature map 501 can be stored entirely in memory (as per [reference to...]). Figure 5 (As discussed). Therefore, each of portions B0 to B7 of the output feature map 501 does not need to be subdivided and distributed across multiple groups of memory locations in order to be "fitted" into the allocated memory. Therefore, only a single tile is needed to read any one of portions B0 to B7 forming the output feature map 501 from memory. Therefore, Figure 10 Instructions 1004 and 1110 set values ​​only for a single tile (tile 0) with a base OFM_BASE0, width OFM_WIDTH0, and height OFM_HEIGHT0.

[0286] Depending on how the data will be stored (or stored) in the cascade buffer, the portion written to (or read from) the cascade buffer may require a larger number of tiles. As mentioned above, the ability to use multiple tiles when writing (or reading from) data to the cascade buffer for a portion of the feature map allows the cascade buffer to be managed in a relatively flexible manner. This allows data for a portion of the feature map to be distributed across different groups of memory locations, making the data a "fit around" data for the other portions of the feature map, for example, preserving filter margin even when only a relatively small cascade buffer is provided (allocated).

[0287] For example, in instruction 1001, the parameters OFM_BASE0, OFM_WIDTH0, and OFM_HEIGHT0 are used to define a single block (block 0) used when writing a portion of A0a to the cascade buffer (because there is enough space in the cascade buffer to write a portion of A0a to a set of memory locations that can be described using a single block). Similarly, instruction 1003 only sets the required parameters IFM_BASE0, IFM_WIDTH0, and IFM_HEIGHT0 to define a single block (block 0) for reading a portion of B0a from the cascade buffer. The parameters used to define blocks 1, 2, and 3 are... Figure 7 Other available parameters for the available tiles shown can be simply omitted from the command stream.

[0288] For portions processed later, it may be necessary to "fit" the data of those portions around at least some of the data from the previous portions in the cascade buffer. Therefore, such portions can be adapted to define multiple blocks for writing data to (or reading data from) the cascade buffer, where each block can be mapped to a different memory location containing a different set of data for that portion. This is illustrated, for example, in instruction 1007, where parameters OFM_BASE0, OFM_WIDTH0, OFM_HEIGHT0, OFM_BASE1, OFM_HEIGHT1, and OFM_BASE3 are set to define three blocks: block 0, block 1, and block 3 for writing portion A1a to the cascade buffer. Similarly, instruction 1009 sets parameters IFM_BASE0, IFM_WIDTH0, IFM_HEIGHT0, IFM_BASE1, IFM_HEIGHT1, and IFM_BASE3 to define blocks 0, block 1, and block 3 for reading portion B1a from the cascade buffer.

[0289] When a portion of the input feature map is written to the cascade buffer (or a portion of the output feature map is read from the cascade buffer), various combinations of available tiles (tile 0, tile 1, tile 2, and tile 3) can be defined as needed for use.

[0290] Figure 12 (i) through (viii) show examples of the contents of the cascade buffer when neural network processing for consecutive "cascaded" operations A and B is performed in the manner described herein.

[0291] The cascaded buffer consists of memory locations allocated (reserved) for storing data elements of the feature map. As mentioned earlier, the cascaded buffer may consist of a relatively small number of allocated memory locations, such that the feature map cannot be stored entirely within those allocated memory locations.

[0292] The memory locations that form a cascaded buffer may not be adjacent to each other. However, the memory locations that form a cascaded buffer will be known or can be derived.

[0293] For example, a cascaded buffer can be logically viewed as an array of locations with known or predefined ranges in the horizontal (x) and vertical (y) directions, where each location corresponds to (maps) a memory location allocated for the cascaded buffer, and data of data elements can be written to (or read from) that memory location.

[0294] Figure 12 (i) through (viii) illustrate such cascaded buffers that can be viewed as arrays of x, y positions. Specifically, the array shown is a 13×10 array of x, y positions. Each x, y position within the array can be identified by its x and y coordinates (spacing) relative to a reference position of the array. Figure 12 In (i) to (viii), the reference position of the cascaded buffer is the top left position of the array.

[0295] As described above, each (x, y) position within the array corresponds to a memory location allocated for the cascading buffer. The mapping from x and y positions to their respective memory locations can be any suitable mapping. For example, the memory location of each (x, y) position in the array can be identified (calculated) based on scaling the x-coordinate of the position within the cascading buffer using the span (memory address spacing) between adjacent positions in the x-direction of the cascading buffer, and the y-coordinate of the position can be scaled using the span (memory address spacing) between adjacent positions in the y-direction of the cascading buffer.

[0296] Figure 12 The array of x, y positions shown in (i) through (viii) can be mapped to a set of memory locations forming a cascaded buffer suitable for storing data for a single channel of a feature map. When the feature map under consideration comprises multiple channels (optionally grouped into channel "blocks"), and / or when multiple feature maps are processing a batch of feature maps, a set of memory locations for a cascaded buffer can be provided for each channel of each feature map. Appropriate scaling can be used to identify the memory locations for different channels to account for the span between channels (memory address spacing) (and optionally, the span between channel blocks (memory address spacing) and / or the span between feature maps in the batch of feature maps (memory address spacing).

[0297] Therefore, it will be apparent that the mapping of each x, y position within the array logically describing the cascaded buffer to a memory location involves scaling that x, y position in a manner similar to that previously discussed for mapping each x, y position within a defined tile to a memory location. Thus, any region in the array logically describing consecutive x, y positions of the cascaded buffer will correspond to (map to) a set of memory locations having known (or derivable) offsets relative to each other. Therefore, any region in the array logically describing consecutive x, y positions of the cascaded buffer can be equivalently described using a single tile defined in the manner described herein.

[0298] Regarding the contents of the cascaded buffer Figure 12 The examples shown in (i) through (viii) illustrate the contents of a cascade buffer when used to store a portion of the feature map between cascaded operations A and B, such as Figure 5 As shown.

[0299] Specifically, in Figure 12 In (i) to (viii), the numbers 0 to 7 at each position of the array represent data metadata, which are written to the corresponding memory location of the cascaded buffer when each of the parts A0 to A7 of the feature map 500 is processed according to operation A.

[0300] exist Figure 12 In (i) to (viii), the dark-colored positions correspond to the data metadata read from the corresponding memory location of the cascaded buffer to perform the processing according to operation B, and the light-colored positions correspond to the data metadata that can be read if operation B needs to take filter margin into account.

[0301] In the example shown, the processing of part A0 (which is the first part of the input feature map processed by operation A) generates a part of the output feature map, the data of which is... Figure 12 The number 0 in (i) is shown. Since the cascade buffer is initially empty, data can be written to a set of memory locations corresponding to a single set of consecutive x, y positions (from which can be derived). Therefore, a single tile can be used when writing data to the cascade buffer.

[0302] Operation B then reads from memory and performs processing using a portion of the input feature map, which includes at least some data from the data output from Operation A. This data is from the corresponding... Figure 12 (i) Read the memory locations at the dark x and y positions. Figure 12 In the case shown in (i), the memory location from which data is read corresponds to (from which a) single set of consecutive x, y positions. Therefore, when reading this data from the cascaded buffer, a single tile can be used.

[0303] When processing the next portion A1 of the input feature map (and similarly, each of the subsequent portions A2, A3, etc.) according to operation A to generate the corresponding portion of the output feature map, there may be limited space remaining in the cascade buffer, making it possible for newly generated data to overwrite some data in the previously generated data. However, the newly generated data should be written to the cascade buffer in a way that avoids overwriting any data that operation B might still need (e.g., data that will be used as filter margin when performing operation B).

[0304] For example, in Figure 12 In the case shown in (ii), data is written to the output portion generated when processing part A1 according to operation A, such that the data at the right and lower boundaries of the output portion from processing part A0 is not overwritten (retained), so that this data can be used later as a filter margin for processing according to operation B. Therefore, the data generated when processing part A1 needs to "fit around" the data from processing part A0, and this data is held in the cascade buffer. Therefore, the data from processing part A1 is written to three sets of memory locations, each set of memory locations corresponding to (from which a set of consecutive x, y positions 703a, 703b, 703c) is derived. Therefore, data can be written to the cascade buffer by defining three blocks. The portion of the feature map (such as) then read for performing operation B is then used. Figure 12 (ii) The dark x and y positions shown similarly occupy three sets of memory locations, each set of memory locations corresponding to (from which a) set of consecutive x and y positions 704a, 704b, and 704c. Therefore, data can be read from the cascaded buffer by defining three tiles.

[0305] like Figure 12 The cascaded buffers shown in (i) to (viii) are "rolling buffers," in which a portion of the feature map's data is written to memory locations corresponding to consecutive x, y positions until it reaches the edge of the x or y range of the array describing the cascaded buffer. Upon reaching the edge of the array describing the buffer, the x, y position "rolls" around that edge, and data continues to be written to memory locations corresponding to consecutive x, y positions starting from the opposite edge of the array describing the buffer. Figure 7 In the example shown, the buffer "rolls" around all its edges, causing it to "roll" in both the x (horizontal) and y (vertical) directions. Figure 12 In the example shown, when "rolling" in the x (horizontal) direction, a y (vertical) offset is introduced. This can be seen, for example, from... Figure 12 (ii) It can be seen that the data output as a result of processing part A1 of the feature map is initially written to memory locations corresponding to the x and y positions, which are the rightmost columns at the top of the array describing the buffer. However, upon reaching the rightmost edge, the remaining data continues to be written to memory locations corresponding to the x and y positions starting from the leftmost edge, where the three positions have a positive vertical (y) offset (which also causes the data to roll vertically around the top edge). Figure 12 As shown in (i) through (viii), when such a rolling buffer is used, this can result in a portion of the feature map output from operation A being stored, such that the portion occupies (is distributed) multiple sets of memory locations, each set of memory locations corresponding to (mapped to) a consecutive set of x, y positions of an array describing the cascaded buffer, so that each set of memory locations can be described using a block of memory defined and mapped in the manner described herein.

[0306] Similarly (and since the output from operation A is distributed across multiple sets of memory locations, each set of memory locations corresponds to (from which can be derived) consecutive sets of x, y positions of the array describing the cascaded buffer), the portion that needs to be read to perform the processing according to operation B can also be distributed across multiple sets of memory locations, each set of memory locations corresponding to (from which can be derived) consecutive sets of x, y positions of the array describing the cascaded buffer.

[0307] Figure 13 yes Figure 12 The simplified representations of data written to and read from the cascaded buffer in (i) and (ii) outline each set of consecutive x, y positions within the array describing the cascaded buffer corresponding to memory locations where data is written (or read) from, and each memory location can be described using a single tile in a manner described herein using the techniques.

[0308] like Figure 13 As shown, when the first portion A0 of the input feature map is processed according to operation A, the data of the corresponding portion of the output feature map is written to a set of memory locations corresponding to (and derived from) a single set of consecutive x, y positions 701 of the cascaded buffer. These memory locations each have a known (and deriveable) offset relative to each other, and can therefore be described using a single patch A0a according to the technique described herein. When operation B is performed, data corresponding to a portion of the input feature map used for operation B is read from a set of memory locations corresponding to a single set of consecutive x, y positions 702, which can also be described using a single patch B0a according to the technique described herein.

[0309] Figure 13 The result of processing the next part A1 of the input feature map according to operation A is also shown. Data of the corresponding part of the output feature map 703 from processing part A1 is written to three sets of memory locations in the cascaded buffer, each set of memory locations corresponding to consecutive x, y positions 703a, 703b and 703c of different groups (e.g., relative to...). Figure 12 (ii) As described herein, the writing can be performed using three blocks A1a, A2a, and A3a in the manner described herein. When operation B is subsequently performed, the data 704 of the portion of the input feature map 704 processed by operation B is distributed across three sets of memory locations in the cascaded buffer, each set of memory locations corresponding to different sets of consecutive x and y positions 704a, 704b, and 704c, where the reading can also be performed using three blocks B1a, B1b, and B1c in the manner described herein.

[0310] Therefore, one or more tiles can be defined for reading data from (or writing data to) memory, depending on how the data is stored (or will be stored) in the cascading buffer. In fact, to illustrate, Figure 10 The instructions shown define the relevant blocks (A0a, B0a, A1a, A1b, A1c, B1a, B1b, B1c) for use in... Figure 13 The example shown illustrates reading data and writing data to the cascaded buffer.

[0311] As from Figure 12 As can be seen from (i) to (viii), when the cascaded buffer is managed such that it rolls horizontally without vertical offset, and rolls vertically with horizontal offset, the memory location storing a portion of the feature map tends to correspond to one or more sets of consecutive x,y positions describing the array of the cascaded buffer, each set of x,y positions having one of two possible widths in the x-direction, while the height of each set of x,y positions in the y-direction can vary. Therefore, Figure 7 A tile arrangement (having up to two columns, each column comprising one or more tiles, wherein tiles in the same column have the same width and wherein the heights of the tiles may be different) is suitable for use when writing partial data of the output feature map to a memory location of a cascaded buffer (or reading partial data of the input feature map from it).

[0312] Alternatively, if the feature map is divided into two columns, they will be as follows: Figure 14 If the data is processed column by column as shown, then the data can be processed as follows: Figure 15 Writes to and reads from the cascaded buffer in the manner shown in (i) through (viii). Similarly, the cascaded buffer can be viewed as an array of (x, y) positions, each corresponding to a position in memory mapped to the cascaded buffer. Figure 15 In (i) through (viii), numbers 0 through 7 represent metadata written to the cascade buffer as part of the output feature map generated by processing portions A0 through A7 respectively according to operation A. The darker positions correspond to metadata that will be read when a portion of the input feature map is processed by the next "cascade" operation (such as operation B).

[0313] exist Figure 15 In the examples shown in (i) to (viii), the cascaded buffer is managed as a scroll buffer that scrolls in the horizontal (x) direction without a vertical (y) offset, and scrolls in the vertical (y) direction with a negative horizontal (x) offset at two positions. This can be seen from, for example... Figure 15 As seen in (iii), the data of a portion of the output feature map generated as a result of processing part A2 is initially written to memory locations corresponding to consecutive x and y positions, which start from the bottom row of the array describing the cascaded buffers, such that these x and y positions are then rolled to the first row of the array describing the buffers, which have a negative horizontal offset of two positions.

[0314] When Figure 15 When managing cascaded buffers as shown in (i) to (viii), a portion of the feature map's data tends to be stored in one or more sets of consecutive x, y locations (corresponding to one or more sets of memory locations), where each set of consecutive x, y locations has one of two different heights in the y direction, while the width of each set in the x direction can be different. Therefore, Figure 16 A tile arrangement (having at most two rows of tiles, each row comprising one or more tiles, wherein tiles within the same row have the same height and wherein the widths of the tiles may differ) is suitable for use when writing a portion of the output feature map data to a cascade buffer (or reading a portion of the input feature map data from it).

[0315] Although it is relatively Figure 7 and Figure 16 A specific arrangement of tiles is described, but other arrangements are also possible. For example, a particular tile arrangement (e.g., the relationship between the height and width of tiles) can be appropriately selected based on data elements, which tend to be distributed across memory when this data is written to (and thus read from) memory.

[0316] Therefore, as can be seen from the above, the techniques described herein provide a mechanism for describing portions of a feature map based on one or more tiles, which allows those portions to be written to (working) memory (e.g., a cascaded buffer) (or read from it) that has a relatively small size compared to the size of the entire feature map.

[0317] The specific embodiments described above are presented for illustrative and descriptive purposes only. They are not intended to be exhaustive or to limit the technology to the precise forms disclosed. Many modifications and variations are possible in accordance with the above teachings. The described embodiments were chosen to best explain the principles of the technology and its practical application, thereby enabling others skilled in the art to best utilize the technology in various embodiments and to have various modifications suitable for the particular intended use. The scope of the invention is intended to be defined by the appended claims.< / type> < / type>

Claims

1. A method of controlling a processor, the processor being operable to perform neural network processing in a data processing system, the data processing system including a processor operable to perform neural network processing and a memory for storing data to be used in performing the neural network processing, the method comprising: Neural network processing is performed on a corresponding portion of the input feature map, each portion of the input feature map comprising an array of data elements, each data element having corresponding data associated with it, each data element having an associated position within the input feature map, and each portion of the input feature map being smaller than the input feature map itself. as well as When performing neural network processing on a portion of the input feature map: The portion of the input feature map is defined based on one or more patches, wherein each patch corresponds to a corresponding region of the portion of the input feature map, and each region corresponds to one or more locations of data elements within the data element array of the portion of the input feature map, wherein the patches together form the entire portion of the input feature map; For each tile, information is provided that allows each data element location forming the tile to be mapped to a corresponding memory location, the corresponding memory location storing data of the data element associated with the location; The method further includes: When data of the location of data elements within the portion of the input feature map is retrieved from the memory, the processor identifies which of the one or more defined tiles the location falls into, uses provided information to identify the memory location from which data is read from the location, the information allowing each location forming the tile to be mapped to a corresponding memory location, and reading the data from the identified memory location; and The processor processes the acquired data of one or more data elements in the input feature map portion to provide one or more data elements for the corresponding portion of the output feature map. The one or more map patches defined for the portion of the input feature map include: One or more columns of tiles, wherein the processing system is configured to impose a constraint on the width of the one or more tiles such that when a column contains multiple tiles, the multiple tiles in the column have the same width relative to each other but can have different heights relative to each other, and wherein the processing system is configured to define the height of the tiles independently for each column such that when the one or more tiles form multiple columns of tiles, the tiles in different columns can have different heights relative to each other; or A row or multiple rows of tiles, wherein the processing system is configured to impose a constraint on the height of the one or more tiles such that when a row contains multiple tiles, the multiple tiles within the row have the same height relative to each other but can have different widths relative to each other, and wherein the processing system is configured to define the width of the tiles independently for each row such that when the one or more tiles form multiple rows of tiles, the tiles within different rows can have different widths relative to each other.

2. The method of claim 1, wherein the neural network processing for said portion of the input feature map includes a neural network processing layer, wherein said portion of the input feature map corresponds to at least a portion of a portion of the output feature map written to the memory when the preceding neural network processing layer is executed.

3. The method of claim 1, wherein there exists a predetermined maximum number of available tiles that can be defined.

4. The method of claim 1, wherein defining the portion of the feature map based on one or more blocks comprises setting values ​​of one or more parameters from which the height and width of each block in the map can be determined.

5. The method of claim 4, wherein the one or more parameters include the overall height of the portion of the feature map, the overall width of the portion of the feature map, and the height and / or width of at least one tile; When only one tile needs to be defined, values ​​are set for the parameters that include the overall height of the portion of the feature map and the overall width of the portion of the feature map; and When more than one tile is to be defined, values ​​are set for the parameters of the overall height of the portion of the feature map, the overall width of the portion of the feature map, and the height and / or width of the at least one tile.

6. The method of claim 1, wherein information allowing each location forming a tile to be mapped to a corresponding memory location storing data associated with the data element of said location comprises: Information that allows mapping the reference position of the tile to the reference memory position; and And information that allows determining the location of the data element's data in memory relative to the reference memory location of the tile from the relative position of the data element within the tile.

7. The method of claim 6, wherein the information that allows determining the location of the data of the data element in the memory relative to the reference memory location of the tile includes information indicating at least one of the following: the span between adjacent data elements in the horizontal direction within the tile in the memory; the span between adjacent data elements in the vertical direction within the tile in the memory; the span between memory locations storing data of different channels of the feature map; and the span between memory locations storing data of different feature maps in a batch of feature maps.

8. A method of controlling a processor, the processor being operable to perform neural network processing in a data processing system, the data processing system including a processor operable to perform neural network processing and a memory for storing data to be used in performing the neural network processing, the method comprising: Neural network processing is performed to generate corresponding portions of an output feature map, each portion of which includes an array of data elements, each data element having corresponding data associated with it, each data element having an associated position within the output feature map, and each portion of the output feature map being smaller than the output feature map itself. as well as When performing neural network processing to generate a portion of the output feature map: The portion of the output feature map is defined based on one or more tiles, wherein each tile corresponds to a corresponding region of the portion of the output feature map, and each region corresponds to one or more positions of data elements within the data element array of the portion of the output feature map, wherein the tiles together form the entire portion of the output feature map; For each tile, information is provided that allows each data element location forming the tile to be mapped to a corresponding memory location for storing data of the data element associated with that location; The method further includes: When storing data on the location of data elements within the portion of the output feature map in memory, the processor identifies which tile among the one or more defined tiles the location falls into, uses provided information to identify the memory location where the data for that location is stored, and allows each location forming the tile to be mapped to a corresponding memory location, and the data to be stored at the identified memory location. The one or more map patches defined for the portion of the output feature map include: One or more columns of tiles, wherein the data processing system is configured to impose constraints on the width of the one or more tiles such that when a column contains multiple tiles, the multiple tiles in the column have the same width relative to each other but can have different heights relative to each other, and wherein the data processing system is configured to define the height of the tiles independently for each column such that when the one or more tiles form multiple columns of tiles, the tiles in different columns can have different heights relative to each other; or A row or multiple rows of tiles, wherein the data processing system is configured to impose a constraint on the height of the one or more tiles such that when a row contains multiple tiles, the multiple tiles within the row have the same height relative to each other but can have different widths relative to each other, and wherein the data processing system is configured to define the width of the tiles independently for each row such that when the one or more tiles form multiple rows of tiles, the tiles within different rows can have different widths relative to each other.

9. The method of claim 8, wherein the neural network processing for generating data for a portion of the output feature map includes a neural network processing layer, the method further comprising: Once the portion of the output feature map has been written to memory, at least a portion of the portion of the output feature map that has been written to memory is used as part of the input feature map and processed by the next neural network processing layer.

10. A data processing system, comprising: A processor capable of operating to perform neural network processing; and The memory is used to store data to be used when performing neural network processing; in: The processor is configured to: Neural network processing is performed on corresponding portions of the input feature map, each portion of which includes an array of data elements, each data element having corresponding data associated with it, each data element having an associated position within the input feature map, and each portion of the input feature map being smaller than the input feature map itself; and The data processing system further includes a processing circuit, which is configured to: A portion of the input feature map to be processed is defined based on one or more patches, wherein each patch corresponds to a corresponding region of the portion of the input feature map, and each region corresponds to one or more locations of data elements within the data element array of the portion of the input feature map, wherein the patches together form the entire portion of the input feature map; and For each tile, information is provided that allows each data element location forming the tile to be mapped to a corresponding memory location, the corresponding memory location storing data of the data element associated with the location; and The processor is further configured to perform neural network processing on a portion of the input feature map: When data on the location of data elements within the portion of the input feature map is retrieved from the memory, it is identified which of the one or more defined tiles the location falls into. The provided information is used to identify the memory location from which data is read from that location. This information allows each location forming the tile to be mapped to a corresponding memory location, and the data is read from the identified memory location. The data processing system processes acquired data from one or more data elements in a portion of the input feature map to provide one or more data elements for a corresponding portion of the output feature map, wherein the data processing system is configured to, when defining a portion of the input feature map to be processed based on one or more map patches: Define one or more tiles comprising one or more columns of tiles, wherein the data processing system is configured to impose constraints on the width of the one or more tiles relative to each other, such that tiles within the same column have the same width relative to each other but can have different heights relative to each other, and wherein the data processing system is configured to define the height of the tiles independently for each column, such that tiles within different columns can have different heights relative to each other; or The definition includes one or more tiles comprising one or more rows of tiles, wherein the data processing system is configured to impose constraints on the height of the one or more tiles relative to each other, such that tiles in the same row have the same height relative to each other but can have different widths relative to each other, and wherein the data processing system is configured to define the width of the tiles independently for each row, such that tiles in different rows can have different widths relative to each other.

11. The data processing system of claim 10, wherein the processor is configured to perform neural network processing, the neural network processing including a neural network processing layer for a portion of an input feature map, wherein the portion of the input feature map corresponds to at least a portion of an output feature map written to the memory when the preceding neural network processing layer is executed by the processor or a different processor.

12. The data processing system of claim 10, wherein the data processing system is configured to define up to a predetermined maximum number of available tiles to describe a portion of a feature map.

13. The data processing system of claim 10, wherein the data processing system is configured to define the portion of the feature map based on one or more blocks by setting one or more parameters, wherein the height and width of each block in the map can be determined from the one or more parameters.

14. The data processing system of claim 10, wherein information provided by the data processing system, allowing each location of a tile to be mapped to a corresponding memory location storing data associated with the data element of said location, includes: Information that allows mapping the reference position of the tile to the reference memory position; and And information that allows determining the location of the data element's data in memory relative to the reference memory location of the tile from the data element's relative position within the tile.

15. A data processing system, comprising: A processor capable of operating to perform neural network processing; and A memory for storing data generated during neural network processing; in: The processor is configured to: Neural network processing is performed to generate corresponding portions of an output feature map, each portion of which includes an array of data elements, each data element having corresponding data associated with it, each data element having an associated position within the output feature map, and each portion of the output feature map being smaller than the output feature map itself; and The data processing system further includes a processing circuit, which is configured to: Define a portion of an output feature map to be generated from one or more tiles, wherein each tile corresponds to a corresponding region of the portion of the output feature map, and each region corresponds to one or more locations of data elements within the data element array of the portion of the output feature map, wherein the tiles together form the entire portion of the output feature map; and For each tile, information is provided that allows each data element location forming the tile to be mapped to a corresponding memory location for storing data of the data element associated with that location; and The processor is further configured to, when performing neural network processing to generate a portion of the output feature map: When storing the data element locations within the specified portion of the output feature map into the memory, it is identified which of the one or more defined tiles the location falls into. The provided information is used to identify the memory location where the data for that location is stored, allowing each location forming the tile to be mapped to a corresponding memory location, and the data to be stored at the identified memory location. The data processing system is configured to, when defining a portion of the output feature map to be processed based on one or more tiles: Define one or more tiles comprising one or more columns of tiles, wherein the data processing system is configured to impose constraints on the width of the one or more tiles relative to each other, such that tiles within the same column have the same width relative to each other but can have different heights relative to each other, and wherein the data processing system is configured to define the height of the tiles independently for each column, such that tiles within different columns can have different heights relative to each other; or The definition includes one or more tiles comprising one or more rows of tiles, wherein the data processing system is configured to impose constraints on the height of the one or more tiles relative to each other, such that tiles in the same row have the same height relative to each other but can have different widths relative to each other, and wherein the data processing system is configured to define the width of the tiles independently for each row, such that tiles in different rows can have different widths relative to each other.

16. The data processing system of claim 15, wherein the processor is configured to perform neural network processing, the neural network processing including a neural network processing layer to generate data for a portion of an output feature map, wherein at least a portion of the data generated for said portion of the output feature map will be used as a portion of an input feature map to be processed according to a next neural network processing layer executed by said processor or one or more different processors.

17. A computer-readable storage medium storing computer software code, which, when executed on a processor, performs a method for controlling the processor, the processor being operable to perform neural network processing in a data processing system, the data processing system including a processor operable to perform the neural network processing and a memory for storing data to be used during the execution of the neural network processing, the method comprising at least one of the following: Neural network processing is performed on one or more portions of an input feature map, each portion of the input feature map comprising an array of data elements, each data element having corresponding data associated with it, each data element having an associated position within the input feature map, and each of the one or more portions of the input feature map being smaller than the input feature map itself. as well as A neural network process is performed to generate one or more portions of an output feature map, each portion of the output feature map comprising an array of data elements, each data element having corresponding data associated with it, each data element having an associated position within the output feature map, and each of the one or more portions of the output feature map being smaller than the output feature map itself. The method includes performing neural network processing on a portion of an input feature map: The portion of the input feature map is defined based on one or more patches, wherein each patch corresponds to a corresponding region of the portion of the input feature map, and each region corresponds to one or more locations of data elements within the data element array of the portion of the input feature map, wherein the patches together form the entire portion of the input feature map; For each tile, information is provided that allows each data element location forming the tile to be mapped to a corresponding memory location, the corresponding memory location storing data of the data element associated with the location; When data of the location of data elements within the portion of the input feature map is retrieved from the memory, the processor identifies which of the one or more defined maps the location falls into, uses the provided information to identify the memory location from which data of the location is read, the information allowing each location forming the map to be mapped to a corresponding memory location, and the data to be read from the identified memory location; as well as The processor processes the acquired data of one or more data elements in the input feature map portion to provide one or more data elements for the corresponding portion of the output feature map; and The method includes performing neural network processing to generate a portion of an output feature map: The portion of the output feature map is defined based on one or more tiles, wherein each tile corresponds to a corresponding region of the portion of the output feature map, and each region corresponds to one or more positions of data elements within the data element array of the portion of the output feature map, wherein the tiles together form the entire portion of the output feature map; For each tile, information is provided that allows each data element location forming the tile to be mapped to a corresponding memory location for storing data of the data element associated with that location; When storing data of data element locations within the portion of the output feature map in memory, the processor identifies which of the one or more defined tiles the location falls into, uses provided information to identify the memory location where the data for that location is stored, and allows each location forming the tile to be mapped to a corresponding memory location, and the data to be stored at the identified memory location. Wherein, the one or more map patches defined for the portion of the input feature map or for the portion of the output feature map include: One or more columns of tiles, wherein the data processing system is configured to impose constraints on the width of the one or more tiles such that when a column contains multiple tiles, the multiple tiles in the column have the same width relative to each other but can have different heights relative to each other, and wherein the data processing system is configured to define the height of the tiles independently for each column such that when the one or more tiles form multiple columns of tiles, the tiles in different columns can have different heights relative to each other; or A row or multiple rows of tiles, wherein the data processing system is configured to impose a constraint on the height of the one or more tiles such that when a row contains multiple tiles, the multiple tiles within the row have the same height relative to each other but can have different widths relative to each other, and wherein the data processing system is configured to define the width of the tiles independently for each row such that when the one or more tiles form multiple rows of tiles, the tiles within different rows can have different widths relative to each other.

Images