Method for avoiding bank conflicts and pipeline conflicts in tensor memory layout

Abstract

A method for optimizing tensor memory layout defines at least one hard constraint for allocating a plurality of input / output (I / O) vectors to read and write data of a task in a tensor memory. The at least one hard constraint is applied to determine one or more potential conflicts among the plurality of I / O vectors. One or more soft constraints can also be generated to reduce the one or more potential conflicts among the plurality of I / O vectors. The at least one hard constraint is applied in a maximum satisfiability (MaxSAT) solver. The one or more soft constraints can also be applied in the MaxSAT solver. The MaxSAT solver determines locations of the data in the tensor memory. In the tensor memory, a starting address of input data to be read and output data to be written for each of the I / O vectors is updated.

Description

TECHNICAL FIELD

[0001] The present disclosure relates generally to the field of computer systems, and more particularly to a method for optimizing tensor memory layout and a tensor memory layout optimization module. BACKGROUND

[0002] Some large computing environments include one or more powerful processors and a large amount of memory capable of storing GBs or TBs of information. A practical approach to improve computing speed includes moving data related to a task from a global memory to a smaller local memory for the duration of the task. The processing unit can then perform operations on the data contained in the local memory without the need to perform read and write operations in the global memory. After the task is completed, the data, which is typically modified by the task processing, is returned to the global memory.

[0003] Several tasks related to the same data content stored in the local memory can be executed in parallel (simultaneously or concurrently). Memory storage conflicts and pipeline conflicts can occur between different types of operations due to the parallel execution of operations in a vector, scalar, load store unit pipeline, and between the same type of operations due to the pipelining operations in the pipeline. Pipeline conflicts result in performance degradation by losing instruction level parallelism. Memory storage conflicts can occur when, for example, an attempt is made to read or write information in overlapping portions of the local memory. Such memory storage conflicts can result in read stalls or write stalls and increase instruction execution time.

[0004] In applications implemented on a graphical processing unit (GPU) or a digital signal processor (DSP), a compiler typically cannot efficiently allocate data to various portions of the local memory. Therefore, manual allocation of memory is often required.

[0005] Therefore, there is a need for techniques to optimize memory allocation in the local memory. SUMMARY

[0006] It is an object of the present disclosure to provide a method for optimizing tensor memory layout. Optimization generally includes improving the layout, but does not necessarily involve finding the best layout under all conditions. At least one hard constraint is defined for allocating a plurality of input / output (I / O) vectors for reading input data or writing output data in the tensor memory, wherein the tensor memory stores data for performing a task. The at least one hard constraint is applied to determine one or more potential conflicts among the plurality of I / O vectors. The at least one hard constraint is applied in a maximum satisfiability (MaxSAT) solver to determine locations of the input data and the output data in the tensor memory. A starting address of the input data to be read and the output data to be written for each of the plurality of I / O vectors is updated in the tensor memory.

[0007] According to other aspects of the present disclosure, the method further includes generating one or more soft constraints to reduce the one or more potential conflicts among the plurality of I / O vectors, wherein the one or more soft constraints are also applied in the MaxSAT solver to determine the locations of the input data and the output data in the tensor memory.

[0008] According to other aspects of the present disclosure, the tensor memory is a local memory. The method further includes transferring the data from a global memory to the local memory for processing the data by a processing unit performing the task. The method further includes the processing unit using a first I / O vector of the I / O vectors to simultaneously read first data from the local memory; the processing unit using a second I / O vector of the I / O vectors to read second data from the local memory; the processing unit using a third I / O vector of the I / O vectors to write third data into the local memory, wherein the reading of the first and second data and the writing of the third data are performed while complying with the at least one hard constraint.

[0009] According to other aspects of the present disclosure, the local memory includes a plurality of tiers, wherein each tier includes a plurality of banks. A bank group contains a corresponding bank of each tier of the plurality of tiers. A portion of the first data is written into one or more first banks. A portion of the second data is written into one or more second banks. A portion of the third data is written into one or more third banks. There is no intersection between the first banks, the second banks, and the third banks. The one or more first banks and the one or more second banks are not part of the same bank group.

[0010] According to other aspects of the present disclosure, the method further includes: performing at least one of activity analysis, pipeline analysis, and bank conflict analysis on the local memory before generating the one or more soft constraints and the at least one hard constraint.

[0011] According to other aspects of the present disclosure, the at least one hard constraint is from each of: (i) preventing simultaneous read and write in the same memory bank; (ii) preventing simultaneous reading of multiple inputs in the same memory bank group; (iii) start addresses of input data and output data are integer multiples of 0.5·W, where W is a layer width; (iv) between two different inputs to be read from the local memory simultaneously, a start address of a first input is an even integer multiple of 0.5·W, and a start address of a second input is an odd integer multiple of 0.5·W; (v) a combination thereof.

[0012] According to other aspects of the present disclosure, the one or more soft constraints and the at least one hard constraint are applied to the MaxSAT solver in a Boolean clause format.

[0013] According to other aspects of the present disclosure, the generating one or more soft constraints includes: (a) generating soft read-read conflict clauses using start address Boolean variables, where the start address Boolean variables are true when an I / O vector i starts at an address 0.5·W·j, j is an odd value; (b) generating soft read-write conflict clauses using read-write Boolean layer variables, where the read-write Boolean layer variables are true when an I / O vector i for a read operation is in the same layer as an I / O vector j for a write operation.

[0014] According to other aspects of the present disclosure, the generating one or more soft constraints further includes: (c) calculating a cost of pipeline conflict, adding 1 to a sum of costs of all clauses calculated in (a) and (b); (d) generating pipeline conflict clauses using Boolean overlap variables, where the Boolean overlap variables are true when the I / O vector i overlaps with the I / O vector j.

[0015] According to other aspects of the present disclosure, the defining at least one hard constraint includes: (e) calculating a cost of hard clauses, adding 1 to a sum of all costs produced in (a), (b), (c), (d), and (f); (f) generating hard clauses for non-overlapping I / O vectors with intersecting active ranges using Boolean coverage variables, where the Boolean coverage variables are true when an I / O vector i has any data at an address 0.5·W·j; (g) generating hard clauses for all Boolean variables used in (a), (b), (c), (d), (e), and (f) using original I / O vectors i, j, where i starts at an address 0.5·W·j; (h) generating hard clauses while ensuring that all I / O vectors have start addresses and all I / O vectors are allocated in the local memory.

[0016] According to other aspects of the present disclosure, the MaxSAT solver determines locations of the input data and the output data in the local memory by applying the Boolean clauses.

[0017] According to other aspects of the present disclosure, the local memory includes 64 banks distributed among 4 ranks and 16 bank groups; each bank includes 128 rows, each row including 32 bytes, wherein a rank width W is equal to 512 bytes.

[0018] Another object of the present disclosure is to provide a tensor memory layout optimization module. The tensor memory layout optimization module includes: an input device configured to receive information about a tensor memory and information about a task to be performed on data contained in the tensor memory; an output device configured to output a starting address of a data element contained in the tensor memory; a processor operatively connected to the input device and the output device; and a non-transitory computer readable medium configured to store instructions executable by the processor to implement a method for optimizing the tensor memory layout.

[0019] Another object of the present disclosure is to provide a tensor memory layout optimization module. The tensor memory layout optimization module includes: an input device configured to receive information about a tensor memory and information about a task to be performed on data contained in the tensor memory; an output device configured to output a starting address of a data element contained in the tensor memory; a processor operatively connected to the input device and the output device. The processor is configured to: receive the information about the tensor memory and the information about the task to be performed on data contained in the tensor memory from the input device; define at least one hard constraint for allocating a plurality of input / output (I / O) vectors, the I / O vectors being used to read input data or write output data in the tensor memory; apply the at least one hard constraint to determine one or more potential conflicts among the plurality of I / O vectors; apply the at least one hard constraint in a maximum satisfiability (MaxSAT) solver to determine locations of the input data and the output data in the tensor memory; and cause the output device to update, in the tensor memory, a starting address of input data to be read and output data to be written for each of the plurality of I / O vectors.

[0020] According to other aspects of the present disclosure, the processor is further configured to generate one or more soft constraints to mitigate one or more potential conflicts among the plurality of I / O vectors; and apply the at least one hard constraint and the one or more soft constraints in the MaxSAT solver to determine the locations of the input data and the output data in the tensor memory.

[0021] According to other aspects of the present disclosure, the tensor memory layout optimization module further comprises an internal memory device operably connected to the processor. The input device is further configured to receive information about the structure of the tensor memory, information about the I / O vectors, information about the active ranges of the I / O vectors, and information about potential pipeline conflicts among operations being performed as part of the task. The processor is further configured to store information about the structure of the tensor memory, information about the I / O vectors, information about the active ranges of the I / O vectors, and information about potential pipeline conflicts among operations being performed as part of the task in the internal memory device.

[0022] According to other aspects of the present disclosure, the output device is configured to forward the starting address of a data element contained in the tensor memory to a processing unit of a computing environment or a global memory of the computing environment. BRIEF DESCRIPTION OF DRAWINGS

[0023] The features and advantages of the present disclosure will be appreciated upon reading of the following detailed description and upon reference to the accompanying drawings, in which:

[0024] Figure 1 Block diagram of a computing environment provided for an embodiment of the present disclosure;

[0025] Figure 2 Mapping of a local memory portion of a memory configuration provided for an embodiment of the present disclosure; Figure 1

[0026] Figure 3A Schematic view of the first two memory banks of a local memory provided for an embodiment of the present disclosure; Figure 2

[0027] Schematic view of the last two memory banks of a local memory provided for an embodiment of the present disclosure; Figure 3B Figure 2

[0028] Figures 4A-4E Figure 2

[0029] Figure 5 ​​​​​A timing diagram of operations integrated in a tensor memory layout optimization module for local memory provided for one embodiment of the present disclosure;

[0030] Figure 6 A timing diagram of details of generating tensor memory layout optimization soft constraints and hard constraints in a Boolean clause format provided for one embodiment of the present disclosure;

[0031] Figure 7 A block diagram of a tensor memory layout optimization module provided for one embodiment of the present disclosure;

[0032] Figure 8 A histogram of relative bank conflict reduction for a simulation test case;

[0033] Figure 9 A histogram of relative run time reduction for a simulation test case;

[0034] Figure 10 A histogram of relative run time reduction for a plurality of hardware based test cases.

[0035] It should be noted that in all the drawings and accompanying descriptions, identical features are identified with the same reference numerals. Further, it should be understood that the drawings and the following descriptions are intended to be illustrative only, and that the disclosure is not intended to be limited by the scope of the claims. DETAILED DESCRIPTION

[0036] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the described embodiments belong.

[0037] In general, the present disclosure describes a tensor memory layout optimization technique with the goal of mitigating and alleviating potential conflicts between simultaneous read and write operations of data in a tensor memory, e.g., in a local memory. The allocation of input / output (I / O) vectors (also referred to as I / O buffers, I / O arrays, or I / O tensors) that are used to read input data or write output data in a local memory defines one or more hard constraints. These hard constraints are used to determine potential conflicts between the I / O vectors. Soft constraints can be generated to mitigate the potential conflicts, which can be understood to represent optimization objectives. These soft and hard constraints can be applied in the form of Boolean clauses to a maximum satisfiability solver to position the data to be read and written in the local memory. The starting addresses of the input data to be read and the output data to be written in the local memory are updated. In this way, potential conflicts between the read and / or write operations of various I / O vectors in the local memory are mitigated. In some systems, a large number of parallel tasks can be performed on data contained in the local memory, so some conflicts can not be completely avoided. Regardless, the number of potential conflicts is reduced, and processing performance is greatly improved.

[0038] Figure 1 A block diagram of a computing environment is provided for one embodiment of the present disclosure. The computing environment 100 includes a global memory 110, a processing unit 120, a read port 130, and a write port 140. The computing environment 100 includes other components, such as but not limited to additional buffers, memory caches, synchronizers, interface ports, etc. (not shown); for simplicity, these components are not depicted in Figure 1 FIG. 1.

[0039] The global memory 110 can be very large, e.g., include GBs or TBs of data. To allow fast read and write access and fast vector operations for a given task, a portion of the data in the global memory 110 is copied into a smaller local memory 200 of the computing environment 100 and copied back into the global memory 110 when the given task is complete. For example, but not by way of limitation, the local memory 200 can include tens or hundreds of KBs of data. The portion of data can be copied from the global memory 110 to the local memory 200 and copied back to the global memory 110 using any suitable conventional mechanism that is not within the scope of the present disclosure.

[0040] During a given cycle, the processing unit 120 can read a 512-byte data vector from the local memory 200 using the read port 130 and write a 512-byte data vector into the local memory 200 using the write port 140. Thus, the read port 130 and the write port 140 are designed such that they can access a limited portion of the local memory 200 to read and / or write data in the local memory 200 in each given cycle.

[0041] Processing unit 120 can perform multiple parallel vector operations on data contained in local memory 200. A situation where a specific information element contained in local memory 200 is modified by one operation and then read by another operation may lead to data errors caused by conflicting operations. Such data errors can be relatively easily avoided by ensuring that two operations cannot be executed simultaneously in computing environment 100, but the performance impact due to processing latency would be unacceptable. The following paragraphs describe the structure of local memory 200 and the methods for reading and writing therein, which prevent data errors caused by conflicts while minimizing processing latency.

[0042] Figure 2 Provided for one embodiment of this disclosure Figure 1 The local memory configuration maps to the local memory portion. In a non-limiting example, local memory 200 comprises 64 memory banks numbered B00 to B63, each storing 4KB of data. The memory banks are distributed along one dimension in 16 memory bank groups, each containing 4 memory banks. For example, a memory bank group 201 includes memory banks B02, B18, B34, and B50. The memory banks are also distributed along another dimension across 4 tiers, each tier containing 64KB of memory distributed across the 16 memory banks. For example, a 64KB tier 202 includes memory banks B32 to B47. Other configurations of local memory 200 may contain a larger or smaller number of memory banks distributed across different numbers of memory bank groups and tiers. For example, local memory 200 may contain 64 memory banks distributed across 8 memory bank groups, each containing 8 memory banks, and local memory 200 defines 8 32KB tiers. In another example, local memory 200 can contain 32 memory banks distributed across four memory bank groups, with each group containing eight memory banks. Local memory 200 defines eight 16KB tiers. Memory banks with sizes greater than or less than 4KB are also possible. Other configurations can also be considered. The configuration of local memory 200 is chosen to be consistent with the configurations of read port 130 and write port 140, as described below.

[0043] Figure 3A Provided for one embodiment of this disclosure Figure 2 A schematic diagram of the first two storage banks of the local memory. Figure 3B Provided for one embodiment of this disclosure Figure 2 This is a schematic diagram of the last two storage banks of the local memory. In this embodiment, each storage bank B00 to B63 can contain 4096 bytes (4KB) of data, therefore the local memory 200 can contain 262144 bytes (256KB) of data. Each storage bank is 32 bytes wide and 128 bytes deep. Figure 3A and Figure 3BAs shown, the local memory 200 is addressable such that the first address 0 is found in the first row (row number 0) of the bank B00. Addresses 0 to 31 form the first row of the bank B00, addresses 32 to 63 form the first row of the next bank B01, until the address 511 is found at the end of the first row (not shown) of the bank B15. Then, the address 512 is found in the second row (row number 1) of the bank B00. The sequence continues until the last address 262143 is found in the last row (row number 127) of the last bank B63.

[0044] Returning to Figure 2 The structure of the local memory 200, as well as the hardware structure of the read port 130 and the write port 140, makes it impossible for data to be read in more than one bank of the same bank group (e.g., banks B02, B18, B34, and B50 in the bank group 201) at the same time, and for data to be read and written in the same bank at the same time.

[0045] When two or more vector operations are reading and writing data contained in the same bank, or reading two input data in the same bank group, these operations cause bank conflicts and cannot be executed in parallel. Bank conflicts can cause the processing to stall, resulting in a slowdown of the computing system 100. Considering the local memory 200, attempting to read two inputs in the same bank group (e.g., in banks B00 and B16) can increase the time required to read these inputs by a factor of 16, as reading these inputs in banks B00 and B16 at the same time is prohibited.

[0046] Considering that, in frequent cases, the operation of the computing environment 100 will result in reading and writing data in overlapping parts of the local memory 200, pipeline conflicts can also occur. Performing a second operation can require waiting until a first operation is completed. Pipeline conflicts can be extended to multiple operations that cannot be executed in parallel. In this context, the term “cost” (sometimes referred to as “weight”) applied to a given conflict represents the amount of time the computing environment 100 takes to overcome the given conflict. Typically, pipeline conflicts can be more expensive than bank conflicts, as pipeline conflicts can cause the next operation to be delayed until the current operation is completed.

[0047] Accordingly, the structure of the computing environment 100 imposes some hard constraints on read and write operations, which are defined to prevent conflicts. The present technology uses these hard constraints to optimize the tensor memory layout in order to allocate data in the local memory 200 at compile time:

[0048] Parallel operations cannot read and / or write in the same area of the local memory 200 at the same time, otherwise, the active range of an input or output of one operation cannot intersect with the active range of another input or output of another parallel operation.

[0049] Although in a given period, it is possible to read data from one memory bank at a time from each memory bank group ( Figure 2 The example contains a total of 16 memory accesses, but data cannot be read from multiple memory banks in the same memory bank group within each cycle.

[0050] Although data can be read from any memory bank and written to any other memory bank simultaneously, data cannot be read from or written to the same memory bank in any given period.

[0051] Data blocks that can be read from or written to local memory 200 have starting addresses that are integer multiples of 0.5·W, where W is the layer width. Figure 2 In the example, the layer width is 512 bytes. In this example, the starting address of the data block is n·256, where n is within the range [0-1023] used for alignment with the structures of read port 130 and write port 140. In the example of local memory 200, for a final data block of 256 bytes extending from address 261888 in memory bank B63 to address 262143, there can be 1024 different starting addresses, with the final possible starting address M. i It is 261888 (for n = 1023). Some implementations of local memory 200 can impose a smaller possible range on the value n defined, for example, in the range [0-991], with the last possible starting address M. i It is 253696 (for n=991).

[0052] When the starting address of one of these inputs is an even multiple of 256 bytes, and the starting address of another of these inputs is an odd multiple of 256 bytes, read-read conflicts between two different inputs to be read from local memory 200 can be avoided if there is no overlap between the two inputs in local memory 200.

[0053] Figures 4A-4E Provided for one embodiment of this disclosure Figure 2 Simultaneous read and write operations are performed on the read and write ports in the local memory portion. Read port 130 and write port 140 ( Figure 1 It has a vector width corresponding to a layer width of 200 in local memory. Figure 2In the specific example of the local memory 200 having a width of 512 bytes (defined as 32 bytes per bank times 16 bank groups), the read port 130 and the write port 140 have a vector width of 512 bytes. Two halves 204A and 204B of the vector width of the read port 130 can be defined. In the case of a binary operation, the full 512 byte width of the read port 130 can be used, reading 256 bytes from each of the two inputs. In the case of a unary operation, only half of the read port 130 capacity is used, reading 256 bytes. Write operations can be applied frequently on 256 bytes, although copy operations in the local memory 200 can read and write 512 bytes at a time. Figures 4A-4E How data is read and written in various banks of the local memory 200 over multiple consecutive cycles is illustrated.

[0054] In more detail, in the non-limiting example, Figure 4A The first input 206 is shown to be read in the portion of banks B00 to B15, while the output 208 is to be written in the portion of banks B16 to B31, and the input 210 is to be read in the portion of banks B16 to B47. In this example, a read-read conflict can occur when banks B00, B16, and B32 are read simultaneously, as they are part of the same bank group including B00, B16, B32, and B48. In fact, potential conflicts can occur between all bank groups when the first input 206 and the second input 210 are read simultaneously. In addition, a read-write conflict can occur when the portion of the second input 210 in banks B16 to B31 and the portion of the output 208 are read and written simultaneously. In an aspect of the disclosure, as shown in Figures 4B-4E Such read-read and read-write conflicts are avoided.

[0055] In the example of Figure 4A The first input 206, the second input 210, and the output 208 all have a size of 49152 bytes (48 KB) in the example. The first input 206 starts at address 0 in bank B00 (0x0000) and ends at address 49151 in bank B15 (0x3FFF). The second input 210 starts at address 49152 in bank B16 (0x4000) and ends at address 98303 in bank B47 (0x7BFF). The output 208 starts at address 98304 in bank B16 (0x7C00) and ends at address 147455 in bank B31 (0x23FF). Figure 3A), and ends at address 49151 in bank B15. The first input 206 fills 96 lines of the possible 128 lines (line numbers 0 to 95) of banks B00 to B15. The output 208 starts at the first address of the top line of bank B16, i.e. address 65536. The output 208 occupies the first 96 lines (line numbers 0 to 95) of each of banks B16 to B31. The second input 210 occupies the last 24 lines (line numbers 104 to 127) of banks B24 to B31 (6144 bytes), the last 12 lines (line numbers 116 to 127) of banks B16 to B23 (3072 bytes), the first 84 lines (line numbers 0 to 83) of banks B32 to B39 (21504 bytes) and the first 72 lines (line numbers 0 to 71) of banks B40 to B47 (18432 bytes), for a total of 49152 bytes (48 KB). Figure 4A The spans of the inputs and outputs shown in the following figures are for illustration purposes and are not to scale.

[0056] Figures 4B-4E A non-limiting example of a sequence is shown, which comprises a number of cycles implemented by repeating 4 different phases, for reading the inputs 206 and 210 while writing the output 208. Each of the 4 phases is repeated a respective number of times, sufficient to read the inputs 206 and 210 and write the output 208, as will be indicated in the following paragraphs. In any given phase, a maximum of 512 bytes of the inputs 206 and / or 210 can be read in the local memory 200, and a maximum of 512 bytes of the output 208 can be written in the local memory 200, these read and write operations being performed taking into account the above-mentioned hard constraints, in order to minimize the number of conflicts. The order of the phases can be changed.

[0057] Figure 4B It is shown that, in a first instance of the first phase, data of the first input 206 is read by the read ports 130 in the first line (i.e. line number 0) of banks B00 to B07, while data of the second input 210 is read by the read ports 130 in the first line (i.e. line number 96) of banks B24 to B31. At the same time, data is written by the write ports 140 in the first line (line number 0) of banks B16 to B23. In this case, the term "first line" does not need to designate the topmost line of the various banks, but the first line of the designated bank in which data is read for a given input or written for a given output, where. The first phase is repeated 24 times (i.e. over 24 cycles), i.e. until the content of the second input 210 stored in the bottom 24 lines (lines 104 to 127) of banks B24 to B31 is read. As Figure 4BAs shown, the data for the first and second inputs 206 and 210 are read from banks that are never part of the same bank group, and data is not simultaneously read and written in the same bank.

[0058] Figure 4C As shown, in the first instance of the second phase, the data for the first input 206 is read by read ports 130 in the first row (i.e., row number 0) of banks B08 to B15, while the data for the second input 210 is read by read ports 130 in the first row (i.e., row number 116) of banks B16 to B23. At the same time, data is written by write ports 140 in the first row (i.e., row number 0) of banks B24 to B31. The second phase repeats 12 times (i.e., over 12 cycles), i.e., until the contents of the second input 210 stored in the bottom 12 rows (rows 116 to 127) of banks B16 to B23 are read. As Figure 4B As shown, the data for the first and second inputs 206 and 210 are read from banks that are never part of the same bank group, and data is not simultaneously read and written in the same bank.

[0059] Figure 4D As shown, in the first instance of the second phase, the data for the first input 206 is read by read ports 130 in the first row (i.e., row number 0) of banks B08 to B15, while the data for the second input 210 is read by read ports 130 in the first row (i.e., row number 116) of banks B16 to B23. At the same time, data is written by write ports 140 in the first row (i.e., row number 0) of banks B24 to B31. The second phase repeats 12 times (i.e., over 12 cycles), i.e., until the contents of the second input 210 stored in the bottom 12 rows (rows 116 to 127) of banks B16 to B23 are read. As Figure 4B As shown, the data for the first and second inputs 206 and 210 are read from banks that are never part of the same bank group, and data is not simultaneously read and written in the same bank. Figure 4C

[0060] Figure 4E ​It is shown that in the first instance of the fourth phase, the data of the first input 206 is read by the read ports 130 in the 13th row (i.e., row number 12) of the memory banks B08 to B15, while the data of the second input 210 is read by the read ports 130 in the 13th row (i.e., row number 0) of the memory banks B32 to B39. At the same time, data is written by the write ports 140 in the 13th row (i.e., row number 12) of the memory banks B24 to B31. The fourth phase is repeated 84 times (i.e., over 84 cycles), i.e., until the contents of the second input 210 stored in the top 84 rows (rows 0 to 83) of the memory banks B32 to B39 are read, at which time the contents of the first input 206 stored in the top 96 rows (rows 0 to 95) of the memory banks B08 to B15 are also read, and the contents of the output 208 have been written in the top 96 rows (rows 0 to 95) of the memory banks B24 to B31. Again, the data of the first and second inputs 206 and 210 are read from memory banks that are never part of the same memory bank group, and the data is not simultaneously read and written in the same memory bank.

[0061] Figures 4B-4E The operations of the sequence of FIG. 6 are repeated multiple times, as the read ports 130 and the write ports 140 repeatedly access consecutive 32-byte rows in each memory bank until the first input 206 and the second input 210 are fully read, and until the output 208 is fully written. The following paragraphs give illustrative, non-limiting examples of sequences for fully reading the first input 206 and the second input 210 while also fully writing the output 208.

[0062] It is noted that in the non-limiting example of FIG. 6, only half of the available vector width of the write ports 140 is used in each phase, and the output 208 is written in each phase only in the memory banks B16 to B23 or in the memory banks B24 to B31. Another example can show that the output 208 is written in some phases in rows of the memory banks B16 to B31, and some other phases have no write operation. Figures 4B-4E

[0063] Table I summarizes the number of cycle instances of each read and write operation distributed in the four phases in the example of FIG. 6. Figures 4B-4E

[0064] Table I

[0065] Input 206 Input 210 Output 208 First stage 24 24 24 Second stage 12 12 12 Third stage 72 72 72 Fourth stage 84 84 84

[0066] Thus, in the above example, the complete sequence of FIG. 6 is executed in 24 instances of the first phase, 12 instances of the second phase, 72 instances of the third phase, and 84 instances of the fourth phase. A total of 192 cycle instances are sufficient for the processing unit 120 to fully read the first input 206 and the second input 210, and to fully write the output 208. Figures 4B-4E Figure 1 ​​​) can perform the operation of reading the entire first input 206 and second input 210 and writing the entire output 208 without any bank conflicts.

[0067] In one example, each stage can be performed multiple times before the next stage is performed, as shown in Table I. In another example, the first, second, third, and fourth stages can be interleaved and performed sequentially 12 times, followed by the first, third, and fourth stages performed 12 times, the third and fourth stages performed 48 times, and the fourth stage finally performed 12 times.

[0068] The contents of the output 208 are not directly related to the first input 206 or the second input 210, which is why it is stored in a different space within the local memory 200. While there is no functional requirement to write the output 208 to the local memory 200 at the same time as reading the first input 206 and the second input 210, performing these operations concurrently improves the processing speed of the computing environment 100.

[0069] In the case where the first input 206, the output 208, and the second input 210 have different sizes, Figures 4B-4E The stages of

[0070] Any other logical order of the number of stages can be considered. Many variations of the sequence can be mathematically equivalent, so the actual implementation of the sequence can be chosen based on hardware capabilities or ease of implementation of the sequence. Other variations can be considered as long as they do not result in reading in different banks in the same bank group at the same time, or reading and writing in the same bank at the same time.

[0071] The following paragraphs will now present a technique that can be used to select memory allocation in the local memory 200 to mitigate the likelihood of bank conflicts and / or pipeline conflicts. In one embodiment, the effects can be mitigated by expressing hard constraints and soft constraints for the allocation of data in the local memory 200 as conjunctive normal form Boolean clauses. The hard constraints and soft constraints can then be solved using a weighted maximum satisfiability problem. Figure 5 A timing diagram of operations integrated in a local memory tensor memory layout optimization module provided for one embodiment of the present disclosure. The tensor memory layout optimization module is shown in the following figure. In Figure 5 In the sequence 300, some of the operations can be performed in different orders, some of the operations can be performed concurrently, and some of the operations are optional. In operation 310, the tensor memory layout optimization module performs one or more of an activity analysis, a pipeline conflict analysis, and a bank conflict analysis on an intermediate representation (IR) of a computer program.

[0072] In this context, an activity analysis is performed to evaluate the activity of each I / O vector and detect potential overlaps between I / O vectors. An I / O vector exists from its creation in a syntax node until it is eliminated in another syntax node at the last use. The range between these two syntax nodes is its activity.

[0073] If the activities of two I / O vectors whose memory ranges overlap, they can not be allocated. For example, the following 3 steps can be considered to illustrate the context of "activity":

[0074] B = 1.0 (step SI);

[0075] C = A + B (step S2);

[0076] C = C + 1.0 (step S3).

[0077] In these 3 steps, the activity of B is [SI, S2] and the activity of C is [S2, S3]. B and C have a connected activity at step S2. As long as other rules (same I / O vector size, appropriate instruction type, other hardware-based rules) are met, B and C can reuse the same area in local memory 100. This hard constraint can be used as an exception to the rules defined in relation (2) (below) to reduce the memory to be allocated to various I / O vectors.

[0078] In this case, the pipeline conflict analysis makes the performance of the computing environment 100 higher by optimizing the possibility of executing different operations in parallel in different pipeline units. If two I / O buffers are allocated to some (or overlapping) memory ranges, their read and write operations will be executed consecutively to avoid data conflicts, and this consecutive execution will use additional synchronization to control. The pipeline conflict analysis searches for each two operations that can be executed in parallel. For two such operations, a hard constraint is defined that tries not to allocate a read or write I / O buffer in overlapping memory ranges.

[0079] The analysis of the IR of a computer program provides information about the variables that belong to the same operation, as input or output; this information is very useful to identify potential read-read conflicts and read-write conflicts and to identify variables that can cause pipeline conflicts.

[0080] The hard and soft constraints for memory layout optimization are generated in operation 320 in Boolean clause format. To this end, the tensor memory layout optimization module can use a maximum satisfiability (MaxSAT) solver. A non-limiting example of a suitable MaxSAT solver that can be used for this purpose is available at the Open-WBO website http: / / sat.inesc-id.pt / open-wbo / . Given a conjunctive normal form formula containing non-negative costs, as is the case in this technology, the MaxSAT solver allows finding truth values for its variables that maximize the combined cost of satisfying the Boolean clauses. In more detail, a variable is defined to be true if vector i can start from address j. Boolean clauses (or formulas) involving these variables are generated in conjunctive normal form that correspond to the hard and soft constraints of the tensor memory layout optimization. Using a MaxSAT solver reduces the tensor memory layout allocation problem to a maximum satisfiability problem. While which of these variables are true or false is a priori unknown, the MaxSAT solver identifies which variables are true or false according to the hard and soft constraints. The MaxSAT solver provides various truth values or false values that satisfy the hard constraints as well as at least some of the soft constraints with the highest total cost. A non-limiting example implementation of operation 320 is described in more detail below.

[0081] In operation 330, the MaxSAT solver can compute a solution to the problem of locating information elements in local memory 200. The result provided by the MaxSAT solver can be interpreted as the addresses at which the I / O vectors can be allocated in local memory 200. Then, in operation 340, the tensor memory layout optimization module updates the start addresses of the input / output (I / O) vectors in the compute IR for reading each information element in local memory 200 using read port 130 or storing using write port 140.

[0082] The MaxSAT solver can optimize the task of allocating memory in the banks of local memory 200, as follows.

[0083] A Boolean variable X i，j is declared, where i specifies an I / O vector and j specifies its start address. By definition, it is stated that X i，j = 1 when I / O vector i starts at address j. The Boolean clauses expressed in the following relation define the hard and soft constraints that the MaxSAT solver applies to optimize the memory allocation task.

[0084] Given the last possible start address M i introduced above, the following relation (1) defines the hard clauses used by the MaxSAT solver:

[0085] (X i，0 ∨...∨X i，Mi ,∞) (1)

[0086] Relation (1) uses an infinite cost (∞) to ensure that each I / O vector i starts at the respective start address.

[0087] The following relation (2) defines another hard clause used by the MaxSAT solver:

[0088]

[0089] Relation (2) prescribes the application of an infinite cost (∞) to prevent any physical overlap between I / O vectors. Relation (2) applies to all j and l and prevents a physical overlap if I / O vector i starts at address 256 · j and I / O vector k starts at address 256 · l. Exceptions can be defined for I / O vectors that can be reused by the same operation (e.g., when A + B is added to obtain C and C is stored in the same memory space as B) and have the same start address, i.e., j = i.

[0090] Relation (3) introduces a soft clause in which the aim of the cost model is to minimize read-read conflicts within local memory 200:

[0091] If j = l (mod 2), then

[0092] In relation (3), RR ik is the cost value attributed to a read-read conflict between I / O vectors i and k. This cost value is smaller than infinite, and such a read-read conflict can be acceptable because its aim is to mitigate the impact of such a conflict when it is not possible to avoid the impact of such a conflict completely.

[0093] For read-write conflicts, relation (4) introduces another soft clause similar to relation (3), except that it applies to a different pair of addresses:

[0094] If j and l are in the same layer, then

[0095] In relation (4), RW ik is the cost value attributed to a read-write conflict between I / O vectors i and k.

[0096] Relation (5) describes a soft clause in which a cost value is applied to a pipeline conflict:

[0097]

[0098] In relation (5), PC ik is the pipeline conflict cost value attributed to all I / O vectors i and k that have a pipeline conflict, j and l such that a physical overlap occurs if I / O vector i starts at address 256 j and I / O vector k starts at address 256 l. In a non-limiting embodiment, the cost value attributed to a pipeline conflict can be greater than the total cost of all potential bank conflicts in order to prioritize alleviating pipeline conflicts.

[0099] Applying the soft constraints expressed in relations (1) and (2) and the hard constraints expressed in relations (3) to (5), the size of the model generated by the MaxSAT model can be very large. The need to minimize the likelihood of read-write conflicts can translate into creating a number of clauses equal to the total number of potential starting addresses squared. This can impact the compilation time of the computing environment 100, as the number of clauses of most types is quadratic in the number of I / O vectors and in the number of available starting addresses.

[0100] In one embodiment, the size of the model generated by the MaxSAT solver can be significantly reduced by introducing auxiliary variables. For example, but not limited to, to mitigate read-read conflicts, a new variable can be created for I / O vectors that start at address 256 i, where i has an odd value for each I / O vector. In this way, it is not necessary to generate a clause for each pair of possible addresses. Each pair of I / O vectors that has a potential read-write conflict has 2 clauses enough to ensure that they are in different bank groups.

[0101] In the same or another embodiment, to mitigate read-write conflicts, additional new layer variables can be created for I / O vectors. The layer variable is true when both the input data and the output data have any data in the same 64KB layer of the local memory 200 (i.e., banks B00 to B15, or B16 to B31, or B32 to B47, or B48 to B63).

[0102] Still in the same or another embodiment, to mitigate pipeline conflicts and active range intersection, new "overlap" variables can be added for I / O vectors that have any data at address j. Instead of checking all possible pairs of starting addresses that can cause two I / O vectors to overlap, it is only necessary to check if the two I / O vectors share at least part of the same address range. In this way, the number of many overlapping clauses is linear in the number of starting addresses, instead of quadratic. For pipeline conflicts, the other overlap variable is true if I / O vector i overlaps at least in part I / O vector j. The overlap variable can be defined using the overlap variable. The overlap variable is very useful to optimize the number of I / O vector overlap pairs, instead of the number of individual addresses that overlap.

[0103] Still in the same or another embodiment, to mitigate the impact of memory space reuse, new Boolean variables are introduced for two I / O vectors starting from the same address. To define the auxiliary variables, other hard clauses can be created. In this example, the largest group of clauses is the one that defines the cover variables. Some clauses in this group about n I / O vectors of size buf_sz and available starting addresses nAddr will be according to relation (6):

[0104] n*(nAddr-bur_sz)*bur_sz (6)

[0105] Relation (6) can be used to estimate the problem size and the amount of time a MaxSAT solver can need to perform the memory allocation task.

[0106] Figure 6 A timing diagram providing details of generating the tensor memory layout optimization soft and hard constraints in Boolean clause format for one embodiment of the present disclosure; in Figure 6 The sequence 400 includes a number of operations, some of which can be performed in different order, some of which can be performed concurrently, and some of which are optional. The sequence 400 is a non-limiting example implementation of the operations 320 introduced in the description of the Figure 5 The sequence 400 includes operations 410-480:

[0107] Operation 410: Use of the starting address Boolean variables o ij Generate soft read-write conflict clauses that are true when I / O vector i starts at address 256*j, where j is an odd value.

[0108] Operation 420: Use of the Boolean layer variables r ij Generate soft read-write conflict clauses that are true when I / O vector i has data in 64KB layer j.

[0109] Operation 430: The cost of the pipeline conflict clauses is computed as 1 plus the sum of the cost of all clauses in operations 410 and 420. For example, but not limited to, the cost of a read-read conflict or a read-write conflict can be equal to the maximum buffer size divided by 256.

[0110] Operation 440: Use of the Boolean overlap variables ov ij Generate pipeline conflict clauses that are true when I / O vector i overlaps with I / O vector j.

[0111] Operation 450: The cost of the hard clauses is computed as 1 plus the sum of the cost of all clauses generated in operations 410-440. The cost computed in this operation is expected to be quite high, with the purpose of approximating the infinite cost (0) of relations (1) and (2).

[0112] Operation 460: Use Boolean covering variables c ij Hard clauses are generated with I / O vector non-overlapping I / O vectors i have any data at address 256·j, these Boolean variables are true.

[0113] Operation 470: Use original X i，j I / O vector generates hard clauses for all auxiliary variables used in the before clause, where i starts at address 256·j.

[0114] Operation 480: Generate hard clauses while ensuring all I / O vectors have a starting address and all I / O vectors are allocated in local memory 200.

[0115] The present technology can be applied in a kernel compiler of the computing environment 100. The kernel compiler passes known information about input vectors, output vectors, active ranges, and potential pipeline conflicts, depending on the task to be performed on data contained in the local memory 200, to determine the starting address of the I / O vector. The compiler allocates the starting address to the I / O vector in the local memory 200, or in a local memory with other configurations, such as in various types of on-chip memories.

[0116] Figure 7 A block diagram of a tensor memory layout optimization module is provided for one embodiment of the present disclosure. The tensor memory layout optimization module 500 includes a processor or plurality of cooperating processors (represented for simplicity as processor 510), an internal memory device or plurality of internal memory devices (represented for simplicity as a single internal memory device 520), an input device or plurality of input devices (represented for simplicity as input device 530), and an output device or plurality of output devices (represented for simplicity as output device 540). There can be a combined input / output device instead of separate input and output devices. The processor 510 is operatively connected to the internal memory device 520, the input device 530, and the output device 540. The internal memory device 520 can include a non-transitory computer readable medium for storing instructions executable by the processor 510.

[0117] The input device 530 can receive information about the structure of the local memory 200 and about the operations (i.e., tasks) to be performed on the data contained in the local memory 200, information about the input vectors, output vectors, active ranges, and potential pipeline conflicts. This information is provided by the input device 530 to the processor 510, which can store the information in the internal memory device 520. The processor 510 can implement a MaxSAT solver, for example, obtained from an open source library. The processor 510 can use the MaxSAT solver to perform various operations of the sequences 300 and / or 400 to account for the configuration of the input ports 130 and output ports 140, and to account for the configuration of the local memory 200, to assign the starting addresses of the I / O vectors. The processor 510 then causes the output device 540 to provide these starting addresses to the computing environment 100, for example, to the processing unit 120 and / or the global memory. The data used to perform the tasks can then be copied from the global memory 110 to the local memory 200 according to the starting addresses defined by the tensor memory layout optimization module 500.

[0118] It will be appreciated that the sequences 300 and 400 can also be performed by a computer program, which can exist in a variety of forms both active and inactive. For example, the computer program can exist as software program(s) comprised of program instructions in source code, object code, executable code, or other formats. Each of these can be embodied on a computer readable medium, which include storage devices and signals, in compressed or uncompressed form. Representative examples of computer readable storage devices include conventional computer system RAM (random access memory), ROM (read only memory), EPROM (erasable, programmable ROM), EEPROM (electrically erasable, programmable ROM), and magnetic or optical disks or tapes. Representative examples of computer readable signals include those

[0119] The performance of the present technology has been evaluated in simulation and in actual hardware implementations. Simulation results are shown in Figure 8 and Figure 9 Hardware-based results are shown in Figure 10 ​

[0120] The simulation environment includes a test suite that contains a large number of commonly used machine learning operations with commonly used shapes. In the context of the present technology, a shape can correspond to an array / vector dimension in a typical deep learning operation. These arrays or vectors are divided into smaller segments for processing on the computing environment 100. These segments are treated as simple one-dimensional arrays when performing the tensor memory layout optimization. These original shapes are tested to show the performance of the current technology in real life.

[0121] The tests were run in a simulator in which a MaxSAT solver available in the Open-WBO website (http: / / sat.inesc-id.pt / open-wbo / ) was implemented, both without and with the current tensor memory layout optimization technology. The simulator provides detailed information about the kernel execution, including runtime, bank conflicts, pipeline load, etc. The test cases with more than 50 read-read and read-write bank conflicts were compared when using the current technology without the present technology and the results obtained with the present memory tensor layout optimization technology. A timeout of 4 seconds was applied to the simulator, after which a non-optimal solution was provided. A total timeout of 100 seconds was applied. The limit on the number of I / O vectors was set to 1000 and the limit on the number of clauses was set to 15,000,000. The final write-write conflicts were not part of the simulation. A total of 1141 test cases were run, of which 171 test cases had more than 50 read-read and read-write bank conflicts.

[0122] Figure 8 A histogram of the relative bank conflict reduction for the simulation test cases is produced. The histogram includes information for the 171 test cases with more than 50 read-read and read-write bank conflicts in the simulation run without the present technology. The vertical axis shows the number of test cases distributed on the horizontal axis that provide a percentage change (increase or decrease) in read conflicts. The vast majority of test cases show a significant reduction in conflicts when run with the current tensor memory layout optimization technology.

[0123] A very small number of new conflicts were produced, as evidenced by the bars on the horizontal axis showing a few negative percentages. Since the cost of pipeline conflicts is high, the simulation prioritizes reducing pipeline conflicts before reducing bank conflicts. Therefore, in a small number of test cases, the number of bank conflicts increased. In any case, the overall execution was faster due to the improvement in instruction-level parallelism. In fact, Figure 9A histogram of the relative reduction in run-time for the test cases. The histogram includes information for the same 171 test cases. The vertical axis shows the number of test cases distributed across the horizontal axis, which provides the percentage change (increase or decrease) in the number of cycles (or "ticks") of hardware used to perform its task. In Figure 9 In this case, positive values indicate that certain test cases are faster to execute, and negative values correspond to test cases that are slower to execute. It was found that the reduction in bank conflicts corresponds to a relatively small increase in run-time (up to 8%), while the reduction in pipeline conflicts can result in a 30-40% reduction in run-time.

[0124] There can be two reasons for the increase in run-time for some tests. First, the kernel can contain a very large number of scalar operations, such as operations with tensor element indices, because using larger vector start addresses increases the number of instructions required to process these operations. This can be the case for instructions that copy 16 bits at a time. Second, the code size can increase slightly due to the increase in start addresses, which requires more operations to process, and if the run-time of the operations is initially short, the likelihood of cache misses increases, which can have an impact.

[0125] In any case, considering all the performance changes, the run-time of the 171 test cases was reduced by an average of 4.7% when using the current tensor memory layout optimization technique, despite some local regressions.

[0126] The test cases that were run without using the current tensor memory layout optimization technique on the hardware implementation, and then using the current tensor memory layout optimization technique. Of these, 97 test cases had more than 50 read-read and read-write bank conflicts without using the current tensor memory layout optimization technique. Figure 10 A histogram of the relative reduction in run-time for a number of hardware-based test cases. Figure 10 The histogram of Figure 9 The histogram of

[0127] It should be understood that the operations and functions of the described tensor memory layout optimization module 500, its constituent components, and the associated processes can be implemented by any one or more of hardware-based, software-based, and firmware-based elements. Such operational alternatives do not limit the scope of the present disclosure in any way.

[0128] It will also be understood that, although the embodiments presented herein have been described in reference to particular features and structures, this description is illustrative only and is not intended to limit the scope of the disclosure. Changes can be made in the function and arrangement of elements without departing from the scope of the disclosure as recited in the claims. The disclosure as described herein is thus intended to be illustrative, and not restrictive, of the disclosure as set forth in the following claims, and any and all modifications, changes, combinations, or equivalents (including those substituting privacy for security, security for privacy, or both) that come within the scope of the claims are intended to be embraced by the claims as set forth herein.

Claims

1. A method for optimizing tensor memory layout, characterized in that, comprising: defining at least one hard constraint for allocating a plurality of input / output (I / O) vectors for reading input data or writing output data in a tensor memory storing data for performing a task; applying the at least one hard constraint to determine one or more potential conflicts among the plurality of I / O vectors; applying the at least one hard constraint in a Max-SAT solver to determine locations of the input data and the output data in the tensor memory; and updating, in the tensor memory, a starting address of the input data to be read and the output data to be written for each of the plurality of I / O vectors; wherein the tensor memory is a local memory comprising a plurality of tiers, wherein each tier comprises a plurality of banks; a bank group contains corresponding banks of each tier of the plurality of tiers; wherein the at least one hard constraint is selected from the group consisting of: (i) preventing simultaneous read and write in the same memory bank; (ii) preventing simultaneous reading of multiple inputs in the same memory bank group; (iii) the start address of input data and output data are integer multiples of wherein is the layer width; (iv) between two different inputs to be read from local memory simultaneously, wherein the start address of the first input is an even integer multiple of and the start address of the second input is an odd integer multiple of ; (v) a combination thereof.

2. The method of claim 1, wherein, further comprising: generating one or more soft constraints to reduce the one or more potential conflicts among the plurality of I / O vectors, wherein the one or more soft constraints are further applied in the Max-SAT solver to determine the locations of the input data and the output data in the tensor memory.

3. The method of claim 2, wherein, the method further comprising: transferring the data from a global memory to the local memory for processing the data by a processing unit performing the task; and the processing unit reading first data from the local memory using a first I / O vector of the I / O vectors; the processing unit reading second data from the local memory using a second I / O vector of the I / O vectors; the processing unit writing third data into the local memory using a third I / O vector of the I / O vectors, wherein the reading of the first and second data and the writing of the third data are performed simultaneously while complying with the at least one hard constraint.

4. The method of claim 3, wherein: a portion of the first data is written into one or more first banks; a portion of the second data is written into one or more second banks; a portion of the third data is written into one or more third banks; there is no intersection between the first banks, the second banks, and the third banks; and the one or more first banks and the one or more second banks are not part of a same bank group. further comprising:

5. The method of claim 4, wherein, performing at least one of an activity analysis, a pipeline analysis, and a bank conflict analysis on the local memory prior to generating the one or more soft constraints and the at least one hard constraint. the one or more soft constraints and the at least one hard constraint are applied to the Max-SAT solver in a Boolean clause format.

6. The method of claim 2, wherein, the generating one or more soft constraints comprises:

7. The method of claim 6, wherein, the generating one or more soft constraints further comprises: (a) generating soft read-read conflict clauses using a start address Boolean variable, wherein the start address Boolean variable is true when the I / O vector starts at an address that is an odd number, and wherein the soft read-read conflict clauses are generated by the steps of: (b) generating soft read-write conflict clauses using read-write Boolean layer variables, where the read-write Boolean layer variable is true when the I / O vector for a read operation is in the same layer as the I / O vector for a write operation .

8. The method of claim 7, wherein, (c) calculating a cost of a pipeline conflict as 1 plus a sum of costs of all clauses calculated in (a) and (b); the defining at least one hard constraint comprises: (d) generating a pipeline conflict clause using a Boolean overlap variable, wherein the Boolean overlap variable is true when the I / O vector overlaps with the I / O vector .

9. The method of claim 8, wherein, (e) calculating a cost of a hard clause as 1 plus a sum of all costs produced in (a), (b), (c), and (d); ​ (f) generating hard clauses for I / O vector non-overlaps with intersecting activity ranges using Boolean cover variables, where the I / O vector At address The Boolean cover variable is true with any data. (g) using the original I / O vectors , generating hard clauses for all Boolean variables used in constraints (a), (b), (c), (d), (e), and (f), where, starting from addresses ; and (h) generating hard clauses while ensuring that all I / O vectors have a starting address and that all I / O vectors are allocated in the local memory.

10. The method of claim 7, wherein, The MaxSAT solver determines locations of the input data and the output data in the local memory by applying the Boolean clauses.

11. The method of claim 7, wherein, the local memory comprises 64 banks distributed among 4 tiers and 16 bank groups; Each memory bank contains 128 rows, each row containing 32 bytes, where the layer width equals 512 bytes.

12. A tensor memory layout optimization module, comprising: comprises: an input device for receiving information about a tensor memory and information about a task to be performed on data contained in the tensor memory; an output device for outputting starting addresses of data elements contained in the tensor memory; a processor operatively connected to the input device and the output device, wherein the processor is configured to: receive, from the input device, the information about the tensor memory and the information about the task to be performed on data contained in the tensor memory; define at least one hard constraint for allocating a plurality of input / output (I / O) vectors for reading input data or writing output data in the tensor memory; apply the at least one hard constraint to determine one or more potential conflicts among the plurality of I / O vectors; apply the at least one hard constraint in a MaxSAT solver to determine locations of the input data and the output data in the tensor memory; and cause the output device to update, in the tensor memory, starting addresses of the input data to be read and the output data to be written for each of the plurality of I / O vectors; wherein the tensor memory is a local memory comprising a plurality of tiers, wherein each tier comprises a plurality of banks; a bank group comprises corresponding banks of each tier of the plurality of tiers; wherein the at least one hard constraint is selected from the group consisting of: (i) preventing simultaneous read and write in the same memory bank; (ii) preventing simultaneous reading of multiple inputs in the same memory bank group; (iii) the start address of input data and output data are integer multiples of wherein is the layer width; (iv) between two different inputs to be read from local memory simultaneously, wherein the start address of the first input is an even integer multiple of and the start address of the second input is an odd integer multiple of (v) a combination thereof.

13. The tensor memory layout optimization module of claim 12, wherein, the processor is further configured to: generate one or more soft constraints to reduce the one or more potential conflicts among the plurality of I / O vectors; and apply the at least one hard constraint and the one or more soft constraints in the MaxSAT solver to determine locations of the input data and the output data in the tensor memory.

14. The tensor memory layout optimization module of claim 13, wherein, further comprising an internal memory device operatively connected to the processor, wherein: the input device is further configured to receive information about a structure of the tensor memory, information about the I / O vectors, information about active ranges of the I / O vectors, and information about potential pipeline conflicts among operations being performed as part of the task; and the processor is further configured to store the information about the structure of the tensor memory, the information about the I / O vectors, the information about active ranges of the I / O vectors, and the information about potential pipeline conflicts among operations being performed as part of the task in the internal memory device.

15. The tensor memory layout optimization module of claim 13, wherein, the output device is configured to forward the starting addresses of data elements contained in the tensor memory to a processing unit of a computing environment or a global memory of the computing environment.

Images