Matrix data broadcast architecture

Abstract

The present invention relates to the field of computers, and more specifically to a matrix data broadcast architecture. Systems, apparatuses, and methods are disclosed for efficiently executing multiple work units in a processor in parallel by reducing the number of memory accesses. A computing system includes a processor core having a parallel data architecture. The processor core executes a software application with matrix operations. The processor core supports broadcasting shared data to multiple compute units of the processor core. A compiler or other code assigns a thread group to a compute unit based on detecting shared data between compute units. The processor core generates a single access request instead of sending multiple read accesses to a memory subsystem to fetch the shared data. The single access request includes information identifying the multiple compute units to receive the shared data when broadcast by the processor core.

Description

Technical Field

[0001] This invention relates to the field of computers, and more specifically, to a matrix data broadcasting architecture. Background Technology

[0002] Parallelization of tasks is used to increase the throughput of computing systems. To do this, compilers extract parallelized tasks from program code for parallel execution on system hardware. Processor cores consist of deep pipelines configured to execute multiple threads. To further increase parallel execution on hardware, multi-core architectures include multiple processor cores. Computing systems offload specific tasks to dedicated hardware, overcoming the performance limitations of conventional general-purpose cores. Some types of dedicated hardware include single-instruction multiple-data (SIMD) parallel architectures, others include field-programmable gate arrays (FPGAs), and still others include other dedicated types of processing cores. When an architecture contains multiple cores of different types, it is called a heterogeneous multi-core architecture. For specific tasks such as graphics rendering, neural network training, and cryptography, heterogeneous multi-core architectures offer higher instruction throughput than homogeneous multi-core architectures.

[0003] Designers use a variety of parallel computing platforms and application programming interface (API) models to develop software applications for heterogeneous computing. Function calls within these platforms are called “computation kernels,” or simply “kernels.” Software, such as an operating system scheduler, matches these software kernels with one or more data records (e.g., data items) to produce one or more computational work units. Generally, SIMD architectures offer good computational performance and cost efficiency when executing such data-parallel workloads. However, performance degrades when storage bandwidth is limited and the application has high data reusability, leading to multiple memory accesses to the same data.

[0004] In view of the above, there is a need for efficient systems and apparatuses that can execute multiple work units in a processor in parallel by reducing the number of memory accesses. Summary of the Invention

[0005] In one aspect, a computing system may include: a processor comprising a plurality of computing units, each computing unit configured to process instructions; a control block; and wherein the control block is configured to, in response to detecting a first access request for the shared data from one of the plurality of computing units and determining that a qualifying condition is met, transmit a single access request for the shared data requested by the plurality of computing units, and in response to receiving the shared data, broadcast the shared data to the plurality of computing units.

[0006] In one implementation, the single access request can be transmitted to a first cache controller and the shared data can be received from a second cache controller.

[0007] In one implementation, the qualifying condition may be that the count of access requests for the shared data equals the number of computing units that require the shared data.

[0008] In one implementation, the control block may be configured to increment the count of access requests in response to detecting an access request for the shared data.

[0009] In one embodiment, the control block may be configured to store broadcast characteristics for the shared data, including one or more of the following: a broadcast identifier that identifies the shared data; and a base address that specifies the storage location where the beginning of the shared data is stored.

[0010] In one embodiment, the control block may be configured to store in the broadcast feature a number equal to the sum of the given computing unit and the one or more other computing units.

[0011] In one embodiment, the control block may be configured to insert one or more of the following into the single access request: a broadcast identifier; and a mask identifying the given computing unit and the one or more other computing units.

[0012] In one embodiment, the first cache controller may be configured to: determine the mask by receiving the mask in the single access request; and determine the mask by converting the broadcast identifier in the single access request.

[0013] In one embodiment, the first cache controller may be configured to: store the mask; and transmit the mask to the second processor during the broadcasting of the shared data to the plurality of computing units.

[0014] In one embodiment, the second processor uses the mask to select the plurality of computing units for receiving the shared data.

[0015] In another aspect, a method may include: in response to detecting a first access request for the shared data from one of the plurality of computing units and determining that a qualifying condition is met, transmitting a single access request for the shared data, which is shared by a plurality of computing units among a plurality of computing units of the processor, to a first cache controller; and broadcasting the shared data from a second cache controller to the plurality of computing units via the first cache controller.

[0016] In one embodiment, none of the plurality of computing units receives an instruction from any of the plurality of computing units to share the shared data.

[0017] In one embodiment, the method may further include: receiving an access request from the second processor via the control block; incrementing a count of the access request for each access request for shared data received from the second processor via the control block; and determining, by the control block, that the qualification condition is met in response to determining that the count is equal to the sum of the given computing unit and the one or more other computing units.

[0018] In one embodiment, the method may further include storing broadcast characteristics for the shared data via the control block, including one or more of the following: a broadcast identifier that identifies the shared data; and a base address that specifies a storage location where the beginning of the shared data is stored.

[0019] In one embodiment, the method may further include inserting one or more of the following into the single access request via the control block: the broadcast identifier; and a mask identifying the given computing unit and the one or more other computing units.

[0020] In one embodiment, the method may further include: determining the mask by receiving the mask in the single access request through the first cache controller; and determining the mask by the first cache controller via converting the broadcast identifier in the single access request.

[0021] In another aspect, an apparatus may include: a first interface configured to communicate with a processor comprising a plurality of computing units, each computing unit being configured to process instructions; a second interface configured to communicate with a first cache controller; and circuitry; wherein the first interface is configured to receive access requests generated by the plurality of computing units executing a plurality of computing cores; wherein the circuitry is configured to, in response to receiving a first access request for shared data from one of the plurality of computing units via the first interface and determining that qualifying conditions are met, transmit a single access request for the shared data requested by the plurality of computing units to the first cache controller.

[0022] In one embodiment, none of the plurality of computing units receives an instruction from any of the plurality of computing units to share the shared data.

[0023] In one embodiment, the circuit may be configured to: increment a count of access requests for each access request received from the processor for the shared data; and determine that the qualification condition is met in response to determining that the count is equal to the sum of the given computing unit and the one or more other computing units.

[0024] In one embodiment, the circuit may be configured to store broadcast characteristics of the shared data, the broadcast characteristics including one or more of the following: a broadcast identifier, wherein the broadcast identifier is a unique identifier that identifies the shared data; and a base address, wherein the base address specifies a storage location where the beginning of the shared data is stored. Attached Figure Description

[0025] The advantages of the methods and mechanisms described herein can be better understood by referring to the following description in conjunction with the accompanying drawings, wherein:

[0026] Figure 1 This is a block diagram of one implementation of a matrix.

[0027] Figure 2 This is a block diagram of one implementation of a matrix.

[0028] Figure 3 This is a block diagram of one implementation of a matrix.

[0029] Figure 4 This is a block diagram of one implementation of a matrix.

[0030] Figure 5 This is a block diagram of one implementation of a table used to support the broadcasting of shared data in a matrix.

[0031] Figure 6 This is a block diagram of one implementation of computer program code for supporting the broadcasting of matrix-shared data.

[0032] Figure 7 This is a block diagram of one implementation of a computing system that supports shared data in a broadcast matrix.

[0033] Figure 8 This is a block diagram of an implementation scheme for a computing system that supports shared data in a broadcast matrix.

[0034] Figure 9 This is a flowchart of one implementation of a method for efficiently parallelizing multiple work units in a processor by reducing the number of memory accesses.

[0035] While the invention is readily adaptable to various modifications and alternatives, specific embodiments are illustrated by way of example in the accompanying drawings, and are described in detail herein. However, it should be understood that the drawings and their detailed description are not intended to limit the invention to the specific forms disclosed; rather, the invention is intended to cover all modifications, equivalents, and alternatives falling within the scope of the invention as defined by the appended claims. Detailed Implementation

[0036] In the following description, numerous specific details are set forth to provide a thorough understanding of the methods and mechanisms presented herein. However, those skilled in the art should recognize that various implementations can be practiced without these specific details. In some cases, well-known structures, components, signals, computer program instructions, and techniques have not been shown in detail to avoid obscuring the methods described herein. It should be understood that, for the sake of simplicity and clarity, the elements shown in the accompanying drawings are not necessarily drawn to scale. For example, the dimensions of some elements may be enlarged relative to other elements.

[0037] Various systems, apparatuses, methods, and computer-readable media are disclosed for efficiently parallelizing multiple work units in a processor by reducing the number of memory accesses. In various embodiments, the computing system includes memory for storing an operating system, software applications developed by the designer, and user data and result data of the software applications. The computing system also includes a processing unit that accesses memory and executes instructions from the operating system and the applications. The processing unit includes one or more processor cores having a single-instruction multiple-data (SIMD) parallel architecture. In one embodiment, an accelerated processing unit on a motherboard includes a SIMD processor core in addition to a general-purpose processor. In another embodiment, a video graphics card includes a SIMD processor core. One or more software applications include instructions for directing the SIMD core to perform generalized matrix multiplication (GEMM) operations, such as multiplying two input matrices together to generate a third output matrix.

[0038] In one implementation, GEMM operations are used in neural network applications. For example, such applications include algorithms for training deep neural networks. Another software application is a graphics shader program that instructs the SIMD processor core on how to render pixels to control lighting and shadow effects. Furthermore, this application includes pixel interpolation algorithms for geometric transformations. Pixel interpolation extracts new pixel values ​​at arbitrary coordinates from existing data. Other software applications also include instructions for instructing the SIMD core to perform GEMM operations for other scientific and commercial uses. These software application instructions operate on matrices with hundreds to thousands of rows and columns, resulting in millions to billions of floating-point operations (FLOPs).

[0039] In some implementations, designers use parallel computing platforms and corresponding application programming interface (API) models to develop software applications using GEMM operations. An example of a parallel computing platform is... The OpenCL framework (generally referred to as "OpenCL" in this document) comprises a C-like language. An example of this language for video graphics applications is GLSL (OpenGL Shading Language). Function calls in the C-like language are called OpenCL kernels, software kernels, compute kernels, or simply "kernels." In some implementations, instead of using the operating system's scheduler, a compiler or other software code includes processor-executable instructions to allocate multiple compute kernels of a software application performing GEMM operations to multiple compute units of a SIMD processor core. In some implementations, the software application includes code segments that include instructions for supporting a shared data broadcast protocol, where shared data is broadcast to multiple compute units. In a shared data broadcast protocol, multiple compute units that need the shared data generate requests for the data, but only one data fetch is generated. When the fetched data (e.g., from cache, memory, etc.) is returned, it is broadcast to the compute unit that requested the data. This way, multiple fetch requests do not unnecessarily consume memory bus bandwidth. In various implementations, the computing kernels executing on the computing units of the SIMD processor core do not communicate with each other about shared data, and do not receive instructions from any other computing kernel to specify that the shared data is shared with any other computing unit.

[0040] In various implementations, the SIMD processor core includes hardware, such as circuitry, for supporting shared data broadcast protocols. The SIMD processor core stores broadcast characteristics of the shared data. These characteristics include a broadcast identifier, which is a unique identifier for the shared data, and a base address, which specifies the storage location at the beginning of the shared data storage. In some implementations, these characteristics also include the total number of computing units using the shared data. When the SIMD processor core detects a read access request for the shared data, it increases the number of computing units sending read access requests for that specific shared data. When the number of computing units accessing the shared data equals the total number of stored data, the SIMD processor core issues a single access request for the shared data.

[0041] In other implementations, the SIMD processor core monitors the number of threads requesting shared data among multiple compute units in a thread group. This sum is the total number of threads from multiple compute units. In one implementation, the SIMD processor core generates a single access request when the count equals the sum of 32 threads from each of the four compute units. Although in one implementation, each compute unit in the SIMD processor core includes 64 parallel channels for executing 64 threads, the SIMD processor core generates a single read access request after each of the four compute units generates one or more read access requests to provide data to 32 of the 64 channels. It is possible and predictable to determine when a single read access request is generated from another number of total parallel channels and other portions of the total number of channels. In other implementations, the SIMD processor core monitors the amount of shared data requested by multiple read access requests from multiple compute units, rather than the number of compute units or the number of parallel channels within a compute unit.

[0042] In one implementation, the SIMD processor core inserts a mask using shared data to identify compute units within a single access request. In another implementation, the SIMD processor core generates a mask based on a broadcast identifier. In such an implementation, the single access request includes the broadcast identifier but not the mask. In other implementations, the single access request includes the mask but not the broadcast identifier. When the SIMD processor core determines that the memory subsystem has returned shared data, it broadcasts the shared data to the compute units identified by the mask. Therefore, the performance degradation impact based on memory subsystem bandwidth limitations is reduced due to the single access request and the broadcast of the retrieved shared data.

[0043] Reference Figure 1An embodiment of matrix 100 used in the operation is illustrated. As shown, matrices A 110 and B 120 are input to matrix operation 130, which produces matrix C 150. Matrix operation 130 and matrices 110, 120, and 150 are used for training neural networks, pixel interpolation during video graphics rendering, or one of a variety of other scientific and commercial applications. In various embodiments, matrix operation 130 is a general matrix and matrix multiplication (GEMM) operation. For example, in some embodiments, matrix operation 130 is a multiplication and accumulation (MAC) operation for determining the dot product of two values ​​of two input matrices. In other embodiments, matrix operation 130 is another mathematical operation for generating a third matrix (such as matrix C 150) from two input matrices (such as matrix A 110 and matrix B 120). A computational unit 140 for allocating the input data values ​​of matrices 110 and 120 to processor cores is also shown, utilizing a single-instruction multiple-data (SIMD) parallel architecture. In one embodiment, the SIMD processor includes 16 compute units, each with 64 execution channels. In other embodiments, the SIMD processor core includes another number of compute units and another number of channels for each compute unit.

[0044] Matrix A 110 is shown as comprising 16 blocks labeled A32 to A47. In one embodiment, each of these 16 blocks comprises K data items, where K is a positive non-zero integer. In such an embodiment, matrix A 110 has a number of data items equal to 16 multiplied by K. Similarly, matrix B 120 is shown as comprising one block labeled B32. This single block has a number of data items equal to 1 multiplied by K, or K data items. The value K can be any number, so matrices 110 and 120 can be very large. For example, the number of data items K can range from tens to hundreds or even thousands. When the range reaches thousands, the number of operations in matrix operation 130 reaches the range of millions to billions. In other embodiments, the value K represents the amount of data, which comprises multiple data items. For example, if the value K represents 64 bytes, and the size of the data item is 32 bits (4 bytes), then the value K represents (64 / 4) data items or 16 data items.

[0045] Matrix C 150 is shown as comprising 16x1 blocks labeled (A32, B32) to (A47, B32). Each block of matrix C 150 comprises a combination of a block of matrix A 110 and a block B32 of matrix B 120, executed by matrix operation 130. Therefore, data items in matrices 110 and 120 are combined by matrix operation 130 in the order indicated by the block labels in matrix C 150. In some embodiments, the SIMD processor core executes the steps of matrix operation 130 in a concurrent and parallel manner. In various embodiments, the computing system includes memory for storing software applications written by software developers, as well as user data and result data of the software applications. Here, user data is the data items of matrices A 110 and B 120, while result data is the data items of matrix C 150.

[0046] In one implementation, the SIMD processor core includes 16 compute units (CUs) as shown in allocation 140; however, in other implementations, the SIMD processor core includes a different number of compute units. Using allocation 140, compute unit 0 (CU0) of the SIMD processor core combines data in block A32 of matrix A 110 with data in block B32 of matrix B 120 using the steps of matrix operation 130. Software applications written by software developers include the steps of matrix operation 130. For example, the software application uses function calls with definitions defined in a specific library. Software developers use one of several types of parallel computing platforms and application programming interface (API) models to develop software applications. Function calls in these platforms are called “compute kernels,” or simply “kernels.”

[0047] In various implementations, the memory also stores instructions for the operating system. In some implementations, the operating system instructions are executed by a general-purpose processor core in the computing system. In one implementation, the application or firmware matches these software kernels with one or more data records (e.g., data items) to generate one or more computing work units. Work units are divided into multiple workgroups, also referred to as "thread groups." Each thread group has a unique identifier (ID) assigned to it. Thread groups are not assigned to computing units by a scheduler, but rather the allocation is managed by the SIMD processor core. For example, when a computing unit has 64 parallel execution channels, the application or firmware divides the work units into 64 groups, and the SIMD processor core assigns the thread groups of the 64 work units to specific computing units. The SIMD processor core executes the allocation shown in computing unit allocation 140 (or allocation 140).

[0048] Similarly, computation unit 1 (CU1) of the SIMD processor core is assigned to combine data in block A33 of matrix A 110 with data in block B32 of matrix B 120 using matrix operation 130. Likewise, computation unit 15 (CU15) of the SIMD processor core uses matrix operation 130 to combine data in block A47 of matrix A 110 with data in block B32 of matrix B 120, and so on for computation units between CU1 and CU15. Based on assignment 140, the data from matrices 110 and 120 are loaded into computation units 0-15 of the SIMD processor core multiple times as needed. For example, if K represents 256 data items, each data item is 4 bytes in size, and each computing unit has 64 execution channels, each channel supporting operations on 4-byte data items, then the SIMD processor core uses (256 / 64) loads, or 4 loads, to fill the 64 parallel execution channels with data from blocks A32 and B32.

[0049] Generally, SIMD architectures offer good computational performance and cost efficiency when performing parallel data workloads. However, performance degrades when storage bandwidth is limited and the application has high data reuse, leading to multiple memory accesses to the same data. As shown in allocation 140, data reuse is typical in GEMM computations. For example, each computation unit CU0-CU15 loads and uses the same data from block B32 of matrix B120.

[0050] In the illustrated implementation, without modifying the execution of the software application, the SIMD processor core generates 32 data fetches, with each compute unit having two data fetches. For example, compute unit 0 (CU0) uses two data fetches for data in blocks A32 and B32. Similarly, compute unit 8 (CU8) uses two data fetches for data in blocks A40 and the same block B32. However, if the SIMD processor core is modified to include broadcast support for reused data, the SIMD processor core will generate 17 data fetches instead of 32. For example, if the software application includes instructions to support only one data fetch in block B32 and broadcast the acquired data to compute units 0-15, the SIMD processor core generates 16 data fetches for data in blocks A32-A47 and a single data fetch for data in block B32. The number of data fetches is reduced from 32 to 17.

[0051] In one example, one or more buses in the memory subsystem are smaller than the amount of data in block B32 of matrix B120. Therefore, for a “single” data fetch of data in block B32, multiple data fetches occur, and the bus is used multiple times while data is transferred within block B32. For example, when the width of the bus to the memory storing data in block B32 is 64 bytes, but the amount of data in block B32 is 512 bytes, a data fetch of block B32 will use eight separate data transfers on the bus. When data in block B32 is fetched multiple times, for example, 16 times for each of the sixteen compute units in one implementation, the resources of the memory subsystem are heavily loaded, and latency increases. Using the example above, without broadcast support, there are (16x8) or 128 data fetches for 512 bytes of block B32. Performance degrades with increasing latency. However, if the SIMD processor core supports broadcasting data in block B32 and fetches that data only once, instead of fetching it for each of the sixteen compute units, then there are only 8 data fetches for the 512 bytes of block B32 compared to 128 data fetches. Therefore, instead of loading the memory subsystem as before, latency is reduced and performance is improved.

[0052] Turn now Figure 2 An embodiment of matrix 200 used in operation is shown. The control logic described above is numbered the same. As shown, matrices A 210 and B 220 are input to matrix operation 130, which produces matrix C 250. Similar to matrix 100, matrix operation 130 and matrices 210, 220, and 250 are used for training neural networks, pixel interpolation in video graphics rendering processes, or one of a variety of other scientific and commercial applications. In various embodiments, matrix operation 130 is a general matrix and matrix multiplication (GEMM) operation. The allocation 240 of the input data values ​​of matrices 210 and 220 to the computational units of the SIMD processor core is also shown.

[0053] Matrix A 210 is shown as comprising eight blocks labeled A32 to A39. In one embodiment, each of these eight blocks comprises K data items. In such an embodiment, matrix A 210 has eight times the number of data items equal to K. Similarly, matrix B 220 is shown as comprising two blocks labeled B32 and B33. Matrix B 220 has twice the number of data items equal to K. As mentioned earlier, K is a positive non-zero integer. The value K can be any number, so matrices 210 and 220 can be very large. In other embodiments, the value K represents the amount of data, which comprises multiple data items. For example, if the value K represents 512 bytes, and the size of a data item is 64 bits (8 bytes), then the value K represents (512 / 8) data items or 64 data items.

[0054] The illustrated matrix C 250 comprises 8x2 blocks labeled (A32, B32) and (A32, B33) to (A39, B32) and (A39, B33). Each block of matrix C 250 comprises a combination of a block of matrix A 210 and a block of matrix B 220, executed via matrix operation 130. Therefore, data items in matrices 210 and 220 are combined via matrix operation 130 in the order displayed by block labeling in matrix C 250. In some embodiments, the SIMD processor core executes the steps of matrix operation 130 in a concurrent and parallel manner.

[0055] In one embodiment, the SIMD processor core includes 16 compute units (CUs), as shown in compute unit allocation 240 (or allocation 240), although in other embodiments, the SIMD processor core includes a different number of compute units. Using allocation 240, compute unit 0 (CU0) of the SIMD processor core combines data in block A32 of matrix A 210 with data in block B32 of matrix B 220 using the steps of matrix operation 130. Compute unit 1 (CU1) of the SIMD processor core is allocated to combine data in block A33 of matrix A 210 with data in block B32 of matrix B 220 using the steps of matrix operation 130. Similarly, compute unit 15 (CU15 of the SIMD processor core) combines data in block A39 of matrix A 210 with data in block B33 of matrix B 220 using the steps of matrix operation 130, and so on for compute units between CU1 and CU15.

[0056] In some implementations, based on allocation 240, the data from matrices 210 and 220 is loaded multiple times as needed into computation units 0-15 of the SIMD processor core. For example, if K represents 64 data items, each with a size of 8 bytes, and each computation unit has 32 execution channels, each channel supporting operations on 8-byte data items, then the SIMD processor core uses (64 / 32) loading, or two loadings, to fill the 32 parallel execution channels with data from blocks A32 and B32. As shown in allocation 240, data reuse is typical in GEMM computations. For example, each computation unit CU0-CU7 loads and uses the same data from block B32 of matrix B 220. Similarly, each computation unit CU8-CU15 loads and uses the same data from block B33 of matrix B 220.

[0057] Without modifying the SIMD processor core that executes matrix operations 130, in the illustrated embodiment, the SIMD processor core generates 32 data fetches, with 2 data fetches generated for each of the 16 computation units. For example, computation unit 0 (CU0) uses 2 data fetches for data in blocks A32 and B32. Similarly, computation unit 7 (CU7) uses 2 data fetches for data in blocks A39 and the same block B32. However, if the software application is modified to support a shared data broadcast protocol, the software application will generate 18 data fetches instead of 32. For example, if the SIMD processor core in block B32 fetches data only once and broadcasts the fetched data to the instructions of computation units 0-7, then for the leftmost column of matrix C 250, the SIMD processor core generates 8 data fetches for data in blocks A32-A39 and a single data fetch for data in block B32. The number of data fetches for the leftmost column of matrix C 250 is reduced from 16 to 9.

[0058] If the SIMD processor core extracts data only once in block B33 and broadcasts the extracted data to compute units 8-15, then for the rightmost column of matrix C 250, the SIMD processor core generates 8 data extractions for the data in blocks A32-A39 and a single data extraction for the data in block B33. The number of data extractions for the rightmost column of matrix C 250 decreases from 16 to 9. The total number of data extractions decreases from (16+16) or 32 to (9+9) or 18. If the SIMD processor core extracts shared data in blocks A32-A39 of matrix A 210 only once and broadcasts the extracted shared data to compute units 0-15, then the SIMD processor core generates 8 data extractions for the data in blocks A32-A39 and performs a single data extraction for the data in blocks B32 and B33, thus providing (8+1+1) or 10 data extractions. As previously described for matrix 100, when the SIMD processor core supports data broadcasting, data is fetched only once, and the memory subsystem is not loaded without broadcast support as before, thereby reducing latency and improving performance.

[0059] Now go to Figure 3An embodiment of matrix 300 used in operation is shown. The control logic described above is numbered the same. As shown, matrices A 310 and B 320 are input to matrix operation 130, which produces matrix C 350. Similar to matrices 100 and 200, matrix operation 130 and matrices 310, 320, and 350 are used for neural network training, pixel interpolation during video graphics rendering, or one of a variety of other scientific and commercial applications. In various embodiments, matrix operation 130 is a GEMM operation. The allocation 340 of the input data values ​​of matrices 310 and 320 to the computational units of the SIMD processor core is also shown.

[0060] Matrix A 310 is shown as comprising four blocks labeled A32 to A35. In one embodiment, each of the four blocks comprises K data items. In such an embodiment, matrix A 310 has a number of data items equal to four times K. Similarly, matrix B 320 is shown as comprising four blocks labeled B32 to B35. Matrix B 320 has a number of data items equal to four times K. As previously stated, K is a positive non-zero integer. The value K can be any number, so matrices 310 and 320 can be very large. In other embodiments, the value K represents the amount of data, which comprises multiple data items.

[0061] Matrix C 350 is shown as comprising 4x4 blocks. The top row of matrix C 350 comprises blocks labeled (A32, B32), (A32, B33), (A32, B34), and (A32, B35). Each block of matrix C 350 comprises a combination of a block of matrix A310 and a block of matrix B320, executed by matrix operation 130. Therefore, data items in matrices 310 and 320 are combined by matrix operation 130 in the order indicated by the block labels in matrix C 350. In some embodiments, the SIMD processor core executes the steps of matrix operation 130 in a concurrent and parallel manner.

[0062] In one implementation, the SIMD processor core includes 16 compute units (CUs), as shown in compute unit allocation 340 (or allocation 340), but in other implementations, the SIMD processor core includes a different number of compute units. As shown in allocation 340, data reuse is typical in GEMM computation. For example, each compute unit CU0-CU3 loads and uses the same data from block B32 of matrix B320. Similarly, each compute unit CU4-CU7 loads and uses the same data from block B33 of matrix B320, and so on.

[0063] Without modifying the SIMD processor core that performs matrix operations 130, in the illustrated embodiment, the SIMD processor core generates 32 data fetches, with each computation unit having 2 data fetches. For example, computation unit 0 (CU0) uses two data fetches for data in blocks A32 and B32. Similarly, computation unit 3 (CU3) uses two data fetches for data in blocks A35 and the same block B32. However, if the SIMD processor core is modified to include broadcast support for reused data (shared data) in matrix B 320, in one embodiment, the SIMD processor core generates 20 data fetches instead of 32. For example, if the SIMD processor core fetches shared data only once in block B32 and broadcasts the fetched shared data to computation units 0-3, then for the leftmost column of matrix C 350, the software application generates 4 data fetches for data in blocks A32-A35 and a single data fetch for data in block B32. The number of data extractions for the leftmost column of matrix C250 was reduced from 8 to 5.

[0064] If the SIMD processor core fetches the shared data in block B33 only once and broadcasts the fetched data to computation units 4-7, then for the second leftmost column of matrix C350, the SIMD processor core generates a total of 5 data fetches for the data in blocks A32-A35 and the data in block B33. If the SIMD processor core fetches and broadcasts the shared data in blocks B34 and B35 in a similar manner to fetching and broadcasting the shared data in blocks B32 and B33, then the total number of fetches is (4x5), or 20 data fetches, instead of 32 data fetches. If the SIMD processor core extracts the shared data in blocks A32-A35 of matrix A310 only once and broadcasts the extracted data to computing units 0-15, then the SIMD processor core will generate 4 data extractions for the data in blocks A32-A35, a single data extraction for the data in block B32, and a single data extraction for the data in block B33. This will provide (4+1+1) or 6 data extractions for the leftmost two columns of matrix C350.

[0065] For the rightmost two columns of matrix C 350, two additional data fetches are performed to extract data from blocks B34 and B35. Therefore, when the software application includes instructions supporting the broadcasting of data from matrices A 310 and B 320, the total number of data fetches increases from 6 to 8 to generate matrix C 350 from matrices 310 and 320. If the SIMD processor core supports broadcasting only data from matrix B 320, then the number of data fetches is reduced from 32 to 20 when generating matrix C 350 from matrices 310 and 320. In either case, as previously stated for matrices 100 and 200, when the SIMD processor core supports data broadcasting, the memory subsystem will not load as it would before without broadcast support, resulting in reduced latency and improved performance.

[0066] Now go to Figure 4 The diagram illustrates one implementation of matrix 400 used in operation. The control logic and data values ​​described above are numbered the same. The resulting matrix C 350 and input matrices 310 and 320 are shown again. To reduce memory accesses, data in the columns of matrix B 320 is associated with a broadcast identifier (ID). In one implementation, instructions from the software application assign broadcast identifier (ID) 0 to data in the leftmost column of matrix B 320, which is block B32. The thread group (TG) assigned to data in block B32 is also assigned to broadcast identifier 0. Thus, as shown in broadcast assignment 460, thread group 0-3 (TG 0-3) is assigned to broadcast identifier 0.

[0067] Thread groups 0-3 use data from block B32 of matrix B320 to generate the leftmost column of matrix C350. The assignment of broadcast identifier 0 instructs that the data retrieval for the data in block B32 should occur once, not four times. For example, data for a specific data item in block B32 should be retrieved once, instead of each of the four logics in the four compute units 0-3 (CU 0-3) generating a read access request for that specific data item. When executing instructions from the GEMM software application, each of the four CUs 0-3 still generates a read access request for the same data in block B32, but a read access request is not sent to the memory subsystem until each of the four read access requests has been generated. In another example, each compute unit CU8-CU11 generates one read access request for the same data in block B34. However, a read access request is not sent to the memory subsystem until each of the four read access requests has been generated.

[0068] Once each of the four read access requests has been generated, a single read access request is generated and sent to the memory subsystem. This single read access request includes an indication specifying that the read access request is associated with a broadcast of the requested data. In one embodiment, this indication is a bit storing an assertion value to indicate the broadcast of the requested data. The single read access request also includes an indication of which computing units receive the requested data during the broadcast. In some embodiments, the single read access request includes a bitmask specifying which computing units receive the requested data during the broadcast. In other embodiments, the single read access request includes a broadcast identifier, which is subsequently translated or decoded to determine which computing units receive the requested data during the broadcast. Thus, when data for block B32 is retrieved from the memory subsystem at one time, that data is simultaneously provided to computing units CU0-CU3. Similarly, when data for block B34 is retrieved from the memory subsystem at one time, that data is simultaneously provided to computing units CU8-CU11.

[0069] Although only the broadcast identifier for matrix B 320 is shown, it should be noted that in other embodiments, broadcast identifiers exist for blocks A32-A35 of matrix A 310. In one embodiment, the data allocated to block A32, as well as thread groups (TGs) 0, 4, 8, and 12 of compute units CU0, CU4, CU8, and CU12, are also assigned to broadcast identifier 4 (not shown). Similarly, the data allocated to block A33, as well as thread groups (TGs) 1, 5, 9, and 13 of compute units CU1, CU5, CU9, and CU13, are also assigned to broadcast identifier 5 (not shown). In such embodiments, the SIMD processor core fetches data for each thread group before executing any one of them. For example, when the broadcast identifier is used for both matrices 310 and 320, compute unit CU0 generates read access requests for both block A 32 and block B32. However, a single access request for block B32 is generated using broadcast identifier 0 only after read access requests from compute units C1-C3 have been generated. Similarly, a single access request for block A32 using broadcast identifier 4 (not shown) is not generated until read access requests from computing units C4, C8, and C12 are generated.

[0070] Turn now Figure 5This illustrates one implementation of table 500 for supporting data broadcasting for matrix operations. In some implementations, the instructions of a software application, such as a GEMM software application, include instructions executed by a SIMD processor core for setting up table 500. In one implementation, a specific computational kernel includes the instructions. In another implementation, the application includes table setup instructions prior to function calls used to define the computational kernel using GEMM operations. Table 500 includes fields 512-524 for storing information used during the broadcasting of shared data during matrix operations such as GEMM operations. Table 500 is stored in a group of registers, random access memory, content-addressable memory, or other locations. In some implementations, table 500 is set up as shown, with each entry assigned to specific shared data to be broadcast. In other implementations, each entry is a separate table. Table 500 provides a mapping (assignment) from thread groups to computational units.

[0071] Field 512 stores the broadcast identifier for shared data. (As previously stated...) Figure 4 As shown, each block B32-B35 of matrix B 320 has a unique broadcast identifier that distinguishes the shared data of a particular block (e.g., B32) from the shared data of another block (e.g., B35). Field 514 stores an indication of the number of thread groups accessing the shared data identified by the corresponding broadcast identifier. In the earlier example shown for matrix B 320, four of the sixteen computing units share the data for each block B32-B35. For example, computing units CU0-CU3 share the data for block B32, computing units CU4-CU7 share the data for block B33, and so on. Field 516 stores an indication of the iteration size, which is used to determine the amount of shared data to be extracted. For each individual data extraction of shared data, a specific amount of shared data is extracted or retrieved. The indication of the amount of shared data is specified as a portion of the total amount of shared data, the number of data items, the absolute amount of data, or something else. In some implementations, the amount of shared data to be extracted is specified in the original read access request for the shared data.

[0072] Field 518 stores the base address of shared data for a specific block, such as block B32 of matrix B320. This base address specifies the memory location where the beginning of the shared data is stored. As shown, the base address of the shared data corresponding to broadcast identifier 0 is 0x04DE_8F10. Here, the symbol "0x" represents a hexadecimal value. The base address is also a reference address. To obtain the absolute address of other data in the shared data, a specific shift (or offset) value is added to the base address. When the first read access request targets the first quarter of the shared data, the base address and a value indicating the size of the quarter of the shared data are used in the first read access request. When the second read access request targets the second quarter of the shared data, the size of the quarter of the shared data is added as a shift relative to the base address to provide the target address. Additionally, a value indicating the size of the quarter of the shared data is used in the second read access request.

[0073] In one implementation, instructions of a software application executing on the processor core also compare the target address of a read access request with the base address in field 518. When the target address is within the range of the base address in field 518, the corresponding read access request targets shared data, and this read access request should be converted into a single read access request that supports broadcasting of the shared data. Field 520 stores a count of the number of incomplete read access requests for the corresponding shared data. As previously described, field 514 stores an indication of the number of thread groups accessing the shared data identified by the corresponding broadcast identifier.

[0074] Whenever an instruction from the software application causes the logic in the processor core to detect an incomplete read access request for specific shared data, the corresponding count in field 520 is updated. In one embodiment, updating the value stored in field 520 includes incrementing the value. In such an embodiment, the initial value stored in field 520 is zero. In other embodiments, the initial value is equal to the value stored in field 514, and updating the value stored in field 520 includes decrementing the value. When the value stored in field 520 is not equal to a threshold, an instruction from the software application causes the logic in the processor core to prevent sending a read access request for the shared data. When updating the value stored in field 520 includes incrementing the value, the threshold is the value stored in field 514. When updating the value stored in field 520 includes decrementing the value, the threshold is zero.

[0075] In some implementations, when the value stored in field 520 equals a threshold, instructions from the software application cause logic in the processor core to generate a single read access request for broadcasting shared data. In other implementations, circuitry in the processor core determines that the value stored in field 520 equals a threshold and generates a single read access request without executing specific instructions instructing the circuitry to do so. As previously described, in one implementation, the single read access request includes bits storing an assertion value to indicate the broadcast of the requested data. The single read access request also includes an indication of which computing units receive the requested data during the broadcast. In some implementations, the single read access request includes a broadcast identifier stored in field 512, which is subsequently converted or decoded to determine which computing units receive the requested data during the broadcast. In other implementations, the single read access request includes a bitmask specifying which computing units receive the requested data during the broadcast.

[0076] Field 522 stores a bitmask specifying which computing units receive the requested data during the broadcast. As shown, the shared data corresponding to broadcast identifier 0 includes a bitmask of 0xF000, specifying that computing units CU0-CU3 use the shared data. The shared data corresponding to broadcast identifier 1 includes a bitmask of 0x0F00, specifying that computing units CU4-CU7 use the shared data, and so on. In embodiments where the broadcast identifier is decoded later to generate the bitmask, field 522 is not used in table 500. In some embodiments, table 500 includes field 524, which stores the size of the shared data. In this example, the size of the shared data is 2 kilobytes (KB). In another embodiment, field 524 includes the end address of the shared data, thus the addresses in fields 518 and 524 provide the address range of the shared data. In other embodiments, logic uses the iteration size in field 516 and the base address in field 518 to determine the address range and / or size of the shared data.

[0077] Now go to Figure 6This illustrates one implementation of source code 600 that utilizes broadcast support for shared data. In various implementations, the computing system includes system memory for storing the operating system, software applications developed by the designers, and user and result data of the software applications. The computing system also includes a processing unit that accesses the system memory, stores information in a cache memory subsystem, and executes instructions from the operating system and applications. The processing unit includes one or more processor cores with a Single Instruction Multiple Data (SIMD) parallel architecture. In one implementation, an accelerated processing unit on the motherboard includes a SIMD processor core in addition to a general-purpose processor. In another implementation, a video graphics card includes a SIMD processor core. Code 600 represents an example of a software application stored in system memory. In some implementations, portions of code 600 are also instructions for firmware stored in memory other than system memory or a cache memory subsystem (e.g., erasable read-only memory or other random access memory). Code 600 includes instructions for directing the SIMD processor core or other circuitry to perform GEMM operations. Code 600 also includes code segments 610, 620, and 630 for supporting broadcasting of shared data.

[0078] In some implementations, software developers use parallel computing platforms and corresponding application programming interface (API) models to develop software applications using GEMM operations. An example of a parallel computing platform is... The OpenCL framework (Open Computing Language) enhances the computational performance of a wide range of data-parallel applications used in gaming, entertainment, science, and healthcare. The OpenCL framework (generally referred to as "OpenCL" in this document) comprises a C-like language. An example of this language for video graphics applications is GLSL (OpenGL Shading Language). C-like function calls are called OpenCL kernels, software kernels, computation kernels, or simply "kernels."

[0079] In one implementation, code 600 is matched with data items to form work units and thread groups, the data items being assigned to computational units of the SIMD processor core. As previously shown, in one implementation, this is achieved by creating table 500 ( Figure 5 The allocation is performed using code segments 610, 620, and 630. These segments include instructions for supporting the broadcasting of shared data to multiple compute units. However, the compute kernel executing on a compute unit of a SIMD processor core does not receive instructions from any other compute kernel instructing that the shared data be shared with any other compute unit. In various implementations, code 600 is a computer program comprising a compute kernel and a collection of internal functions. Software developers typically define the compute kernel, while the internal functions are typically defined in libraries.

[0080] In some implementations, code 600 operates on specific columns of a matrix (e.g., the previous matrix B 320). The columns of matrix B 320 are specific blocks, such as blocks B32, B33, etc. Each computational unit (e.g., CU 0-15 in the previous example) executes the instructions of code 600, but operates on different data items. The earlier examples shown for matrices 100, 200, 300, and 400 illustrate that computational units are grouped differently when sharing data. Sharing is based on the input matrix. Code segment 610 generates a mapping from thread groups to computational units. For example, code segment 610 constructs a mapping similar to table 500 (… Figure 5 The table (of type B) is used to construct one or more tables to support the broadcasting of shared data, and the table also assigns thread groups to specific computational units. In some implementations, code segment 610 is placed in an outer loop, which includes code 600 within that loop. In one implementation, each iteration of the "for loop" in code 600 operates on a partition or subset of columns of a matrix, such as matrix B350. For example, each iteration operates on a quarter of the data in block B32 of matrix B350. In some implementations, field 516 of table 500 specifies the iteration size.

[0081] In various embodiments, when code segment 620 detects the generation of a read access request for shared data, code segment 620 generates a broadcast start message. In some embodiments, code segment 600 includes a hint from the static compiler indicating which read access requests target shared data and which broadcast identifier corresponds to the shared data. For example, the static compiler compares the target address of the read access request with the address range of the shared data. Therefore, code segment 620 detects the compiler hint. In other embodiments, the compiler hint is not used, and code segment 620 performs the comparison of the target address of the read access request with the address range of the shared data. The message includes an indication specifying the start of broadcast data, the corresponding broadcast identifier, and the target address.

[0082] In one implementation, instructions in code 600 between code segment 620 and code segment 630 fetch data from another matrix (e.g., matrix A 310 in the previous example or another matrix A). These instructions also store the fetched data in the local data store (LDS) of the corresponding computing unit. The instructions perform fetching and storing separately for each data item. Code segment 630 detects the start of a broadcast message for matrix B and updates a count, such as a count of the number of read access requests for the same shared data. Code segment 630 also compares this count to the total number of thread groups sharing the shared data. This threshold is the same as the threshold stored in field 514 of table 500. When the count equals the threshold, code segment 630 generates a single, individual read access request and sends it to the memory subsystem to retrieve the requested data. In various implementations, the hardware of the SIMD processor core executes code segment 630, which generates a start signal. The hardware then performs the following steps: detecting the start of a broadcast message, updating the count, comparing it with the threshold, and generating a single read access request. These and other implementations are possible and predictable.

[0083] The generated single read access request includes an indication that specifies the read access request is associated with a broadcast of the requested data. In one implementation, this indication is a bit that stores an assertion value to indicate the broadcast of the requested data. The single read access request also includes an indication of which computing units receive the requested data during a subsequent broadcast. In some implementations, the single read access request includes a bitmask specifying which computing units receive the requested data during the broadcast, such as a bitmask in field 522 of table 500. In other implementations, the single read access request includes a broadcast identifier, which is then translated or decoded to determine which computing units receive the requested data during the broadcast. Thus, when the memory subsystem provides the requested shared data based on a single memory access, that data is simultaneously provided to the local data store of the specified computing unit. Code following code segment 630 reads data from the local data store and uses that data to perform matrix operations, such as GEMM operations. The "for loop" of code 600 is repeated as needed.

[0084] Now for reference Figure 7This illustration shows one embodiment of a computing system 700 supporting the broadcasting of shared data for a matrix. Processing node 710 includes one or more processing units, such as units 715 and 730. Unit 715 includes one or more processor cores 712 and an associated cache memory subsystem 714. Unit 730 includes one or more processor cores 732 and a data storage buffer 734. Processing node 710 also includes a memory controller 720 and interfaces 740a and 740b. Memory controller 720 includes logic and queues for servicing memory requests and responses and communicating with memory 750 based on a specific communication protocol. Interfaces 740a-740b also include logic and queues for servicing requests and responses and communicating with specific devices based on appropriate communication protocols. In one embodiment, interface 740b communicates with a video processing unit 760. In one embodiment, at least the illustrated functionality of processing node 710 is integrated onto a single integrated circuit. In some embodiments, the functionality of computing system 700 is integrated onto a system-on-a-chip (SoC). In other embodiments, the functionality of the video processing unit 760 is integrated into a video graphics card inserted into a motherboard, which also includes a processing node 710 and is connected to a memory 750.

[0085] For ease of illustration, other components such as phase-locked loops (PLLs) and other clock generation circuitry, power management units, shared cache memory subsystems, etc., are not shown. In one embodiment, processor core 712 utilizes a general-purpose microarchitecture. Processor core 732 is not a mirror silicon image of processor core 712. Instead, processor core 732 has a different microarchitecture than that used by processor core 712. In one embodiment, processor core 732 has a microarchitecture that provides high instruction throughput for computationally intensive tasks, such as a parallel data architecture. In one embodiment, processor core 732 is a Single Instruction Multiple Data (SIMD) core. Examples of SIMD cores include graphics processing units (GPUs), digital signal processing (DSP) cores, or others. Similarly, processor core 762 of video processing unit 760 also has a SIMD architecture. Each processor core 732 and 762 includes multiple compute units 770, each compute unit having multiple parallel execution channels 772. In one embodiment, processor core 762 has 16 compute units, each compute unit having 64 parallel execution channels. In other embodiments, a different number of compute units and parallel execution channels are used.

[0086] Typically, processor core 712 accesses cache memory subsystem 714 for both data and instructions. If the requested block is not found in cache memory subsystem 714, processor core 712 generates a read access request and sends it to memory 750 via memory controller 720. Similarly, when the requested data is not found in buffer 734, processor core 732 generates a memory access request to send it to memory 750. In one embodiment, one or more of cache memory subsystem 714 and buffer 734 include high-speed cache memories configured to store data blocks and are implemented as a cache hierarchy. Both cache memory subsystem 714 and buffer 734 include cache array memory and a corresponding cache controller.

[0087] Typically, packet processing logic 716 responds to control packets received on the link coupled to processing node 710, generates control packets in response to processor core 712 and / or cache memory subsystem 714, generates probe commands and response packets in response to transactions selected by memory controller 720 for service, and routes the packets (for such packets, node 710 is an intermediate node) to other nodes via interface logic 740a.

[0088] In some implementations, processor core 762 executes graphics-intensive applications, such as vertex shaders or pixel shaders, on a large number of objects (vertices or pixels). Since each object is processed independently of other objects but uses the same sequence of operations, the SIMD architecture using compute unit 770 and channel 772 can significantly improve performance. In some implementations, processor core 732 executes applications for training neural networks, which also process objects independently of other objects but use the same sequence of operations. Therefore, these types of applications also benefit from the SIMD architecture using compute unit 770 and channel 772.

[0089] In various embodiments, memory 750 includes one or more of hard disk drives, solid-state drives, other types of flash memory, portable solid-state drives, tape drives, etc. Memory 750 stores operating system 752, applications 754, and data 756. Although a single operating system and a single application are shown, in other embodiments, a greater number of these software components are stored in memory 750. Operating system 752 includes instructions for starting processing node 710, allocating tasks to hardware circuitry, managing resources of computing system 700, and hosting one or more virtual environments.

[0090] In some implementations, the general-purpose processor core 712 executes instructions from the operating system 752. In some implementations, the application program 754 uses GEMM operations for neural network training, pixel interpolation during graphics rendering, or other purposes. In one implementation, the application program 754 matches its software kernel with one or more data records (e.g., data items in data 756) to generate one or more computational work units. The work units are divided into multiple thread groups, each with its own assigned unique identifier (ID). The application program 754 also includes code for supporting the broadcasting of shared data, such as broadcast support code 755. In one implementation, code 755 is similar to code 600 (…). Figure 6 Code segments 610, 620, and 630. When application program 754 and code 755 are executed, one or more of processor cores 732 and 762 include circuitry for supporting the broadcasting of shared data.

[0091] One or more of the compiler, code 755, and firmware allocate thread groups to computing units 770 in one or more of SIMD processor cores 732 and 762, based on the application being processed. In one implementation, this is achieved by creating table 500 (… Figure 5 The allocation is performed by the processor cores 732 and 762. In some implementations, one of the processor cores 732 and 762 generates a table. For example, in one implementation, application 754 allocates a thread group for pixel interpolation to the SIMD processor core 762 of video processing unit 760, and application 754 allocates multiple thread groups for neural network training to the SIMD processor core 732 of unit 730. Other types of allocation for these and other applications are possible and can be expected. When software application 754 includes code 755 executed by one or more of processor cores 732 and 762 to support the broadcasting of shared data, the shared data is fetched only once. When the shared data is fetched only once, the memory subsystem is not loaded without broadcast support, latency is reduced, and system performance is improved.

[0092] Reference Figure 8 This illustration shows one embodiment of a computing system 800 supporting a shared data broadcast protocol. The control logic and circuit numbers described above are identical. The computing system 800 includes a processing node 710, a memory 750, and a video processing unit 860 (or unit 860). In one embodiment, the illustrated functionality of the computing system 800 is integrated onto a single integrated circuit. In some embodiments, the functionality of the computing system 800 is integrated onto a system-on-a-chip (SoC). In other embodiments, the functionality of the video processing unit 860 is integrated onto a video graphics card inserted into a motherboard that also includes the processing node 710 and is connected to the memory 750.

[0093] Unit 860 includes shader engines 870a-870b, controller blocks 880a-880b, matrix data storage devices 882a-882b, cache 884, and memory controller 890. Other components (e.g., PLLs, any power management logic, buses, or architectures, etc.) are not shown for ease of illustration. The interface 892 of unit 860 includes logic and queues for servicing requests and responses and for communicating with processing node 710 based on appropriate communication protocols. Similarly, memory controller 890 includes logic and queues for servicing memory requests and responses and for communicating with memory 750 based on specific communication protocols. In one embodiment, cache 884 is a secondary (L2) cache within a cache memory subsystem. However, in other embodiments, cache 884 is another level within the cache memory subsystem. In some embodiments, matrix data storage devices 882a-882b store only data for matrix operations, and a primary (L1) cache (not shown) stores data for other operations. Therefore, in some implementations, the matrix data memories 882a-882b include equivalents of an L1 cache controller, and the cache 884 includes an L2 cache controller.

[0094] A command processor (not shown) schedules received groups of commands to computing resources such as shader engines 870a-870b based on status information. Examples of status information include process identifiers (IDs), protected / unprotected modes / states, the type of computation or graphics being worked, and so on. Each of the shader engines 870a-870b includes multiple compute units 770, each compute unit having multiple parallel execution channels 872 and a local data storage area 874. The local data storage area 874 is implemented using registers, random access memory, or other means. For ease of illustration, the register file, other buffers, and pipeline control logic of the shader engines 870a-870b are now shown. In one embodiment, unit 860 has four shader engines, each shader engine having 16 compute units, and each compute unit having 64 parallel execution channels. In other embodiments, other numbers of shader engines, compute units, and parallel execution channels are used.

[0095] Unit 860 also includes controller blocks 880a-880b, which are implemented by hardware, such as circuitry, software, such as firmware, or a combination of hardware and software. In some embodiments, controller blocks 880a-880b (or control blocks 880a-880b) stage and organize the loading of shader input data into shader engines 870a-870b, and manage resources used to execute shader programs. In some embodiments, controller blocks 880a-880b create table 500 ( Figure 5 The system receives read access requests from the computing unit of the shader engine 870a-870b, maintains a count of read access requests for specific shared data, generates a single access request for the shared data, and issues the single access request to the cache 884 through the matrix data storage device 882a-882b.

[0096] In one embodiment, matrix data storage devices 882a-882b store received individual access requests and publish them to cache 884 based on arbitration logic. The arbitration logic selects the access request to be published to cache 884 based on one or more of priority, age, quality of service parameters, source identifier, etc. A single access request includes one or more of a broadcast identifier and a bitmask as described above. In one embodiment, matrix data storage devices 882a-882b determine a bitmask for identifying the computing unit receiving shared data by receiving a bitmask from controller blocks 880a-880b. In another embodiment, matrix data storage devices 882a-882b determine the bitmask by transforming the broadcast identifier in a single access request. When matrix data storage devices 882a-882b receive requested data, they send the requested data to multiple computing units instead of a single computing unit. Logic in matrix data storage devices 882a-882b selects multiple computing units based on the bitmask. The selection logic (such as multiplexers or others) in the shader engines 870a-870b uses a bitmask to determine which compute units receive the broadcast response data. In other embodiments, the response data is supplied serially from the matrix data storage devices 882a-882b to the compute units instead of simultaneously. However, the matrix data storage devices 882a-882b still supply the response data to multiple compute units, and the response data is accessed from cache 884 by a single read access request, rather than by multiple read access requests.

[0097] Now for reference Figure 9An embodiment of a method 900 for supporting a shared data broadcast protocol is shown. For purposes of discussion, the steps in this embodiment are shown in sequence. However, it should be noted that in various embodiments of the described method, one or more of the described elements may be performed simultaneously, in a different order than shown, or omitted entirely. Other additional elements may also be performed as needed. Any of the various systems or apparatuses described herein is configured to implement method 900.

[0098] Software developers design applications with GEMM operations. In one implementation, the logic and circuitry of a parallel data processor core (or processor core) execute the application's instructions to perform GEMM operations. In one implementation, the application includes something similar to ( Figure 6 The application generates multiple computing cores. The processor core assigns computing cores that share the same data to computing units within the processor core. The processor core supports broadcasting of shared data by loading broadcast features into a table (box 902). The application's computing cores are matched against one or more data records (e.g., data items) to generate one or more computing units of work. The units of work are divided into thread groups, each with its own assigned unique identifier (ID). The processor core assigns these thread groups to computing units within the processor core. In one implementation, this is achieved by utilizing information in one or more of fields 512-524 to create (… Figure 5 Table 500 is used to perform the allocation.

[0099] In some implementations, the application includes hints from the static compiler indicating which read access requests are for shared data, and additionally, which broadcast identifier corresponds to the shared data. For example, the static compiler compares the target address of the read access request with the address range of the shared data. During the execution of the application, if the processor core does not detect a memory access to the target shared data (the "No" branch of condition box 904), the application processes other instructions of the compute core (box 906). If the end of the compute core is reached (the "Yes" branch of condition box 908), method 900 completes (box 910). However, if the end of the compute core is not reached (the "No" branch of condition box 908), the control flow of method 900 returns to condition box 904, where the processor core determines whether a memory access to the shared data has occurred.

[0100] If the processor core detects a memory access request for shared data, such as when it detects a compiler prompt (the "Yes" branch of condition box 904), then when executing instructions from the application or firmware, the processor core's circuitry increments a count of the number of read requests from the same thread group for the shared data (box 912). In one implementation, the application updates field 520 of table 500. If the threshold number of read requests from the thread group has not yet been reached (the "No" branch of condition box 914), the processor core prevents the extraction of the shared data (box 916). The processor core does not issue a read access request to the storage subsystem.

[0101] In one implementation, the processor core compares the updated value in field 520 of table 500 with a threshold in field 514 of table 500. If the threshold number of read requests from the thread group is reached (the "Yes" branch of condition box 914), the processor core then resets the count (box 918). The processor core uses a single read access request to retrieve shared data (box 920). For example, the processor core sends a single read access request previously described using one or more of a broadcast identifier and a bitmask to the memory subsystem. When the shared data is returned from the memory subsystem, the processor core broadcasts the shared data to the corresponding compute unit based on broadcast characteristics such as the mask (box 922). Afterward, the control flow of method 900 returns to condition box 904, where the processor core determines whether a memory access to the shared data has occurred.

[0102] In various embodiments, the program instructions of the software application are used to implement the previously described methods and / or mechanisms. The program instructions describe the behavior of the hardware using a high-level programming language (e.g., C). Alternatively, a hardware design language (HDL), such as Verilog, may be used. The program instructions are stored on a non-transitory computer-readable storage medium. Various types of storage media can be used. During use, the computing system can access the storage medium to provide the program instructions and accompanying data to the computing system to execute the program. The computing system includes at least one or more memories and one or more processors that execute the program instructions.

[0103] It should be emphasized that the above embodiments are merely non-limiting examples of implementation methods. Many variations and modifications will become apparent to those skilled in the art once the above disclosure is fully understood. It is intended that the following claims be interpreted as encompassing all such variations and modifications.

Claims

1. A computing system comprising: A processor, which includes multiple computing units, each of which is configured to process instructions; as well as Control block, the control block is configured to: The count of access requests for shared data is compared with the number of computing units among the plurality of computing units that require the shared data; In response to: A first access request was detected from one of the plurality of computing units that require the shared data; as well as The count of access requests for the shared data is determined to be equal to the number of computing units that require the shared data; Transmit a single access request for the shared data, and In response to receiving the shared data, the shared data is broadcast to the computing units that require the shared data.

2. The computing system of claim 1, wherein, The control block is also configured to store the size of the shared data.

3. The computing system of claim 1, wherein, The control block is also configured to store the end address of the shared data.

4. The computing system of claim 1, wherein, The control block is configured to increment the count of access requests for the shared data in response to the detection of such access requests.

5. The computing system of claim 1, wherein, The control block is configured to store broadcast characteristics for the shared data, including one or more of the following: A broadcast identifier that identifies the shared data; and The base address specifies the storage location where the beginning of the shared data is stored.

6. The computing system of claim 5, wherein, The control block is configured to store in the broadcast feature a number equal to the sum of the number of computing units that require the shared data.

7. The computing system of claim 5, wherein, The control block is configured to insert one or more of the following into the single access request: Broadcast identifier; and A mask identifying the computing unit that requests the shared data.

8. The computing system of claim 7, wherein, The cache controller is configured as follows: The mask is determined by receiving the mask in the single access request; and The mask is determined by transforming the broadcast identifier in the single access request.

9. The computing system of claim 8, wherein, The cache controller is configured as follows: Store the mask; as well as During the process of broadcasting the shared data to the computing units that need the shared data, the mask is transmitted to the processor.

10. The computing system of claim 9, wherein, The processor uses the mask to select the computing unit that needs the shared data to receive the shared data.

11. A method comprising: The processor's control block compares the count of access requests for shared data with the number of computing units among the multiple computing units that require the shared data. In response to: A first access request was detected from one of the plurality of computing units that require the shared data; as well as The count of access requests for the shared data is determined to be equal to the number of computing units that require the shared data; The control block of the processor transmits a single access request for the shared data to the cache controller; as well as The shared data is broadcast to the computing units that require it.

12. The method of claim 11, wherein, None of the computing units that require the shared data receive an instruction from any of the computing units that require the shared data to share the shared data.

13. The method of claim 11, further comprising, via the control block, incrementing the count of access requests for the shared data for each access request received for the shared data.

14. The method of claim 11, further comprising storing, via the control block, a broadcast feature for the shared data including one or more of the following: A broadcast identifier that identifies the shared data; and The base address specifies the storage location where the beginning of the shared data is stored.

15. The method of claim 14, further comprising inserting one or more of the following into the single access request via the control block: The broadcast identifier; and A mask that identifies the computing unit that requires the shared data.

16. The method of claim 15, further comprising: The control block determines the mask by receiving the mask in the single access request. as well as The mask is determined by the control block via the transformation of the broadcast identifier in the single access request.

17. An apparatus comprising: The control block includes circuitry configured to communicate with a processor comprising multiple computing units, each configured to process instructions. The matrix data memory, including the circuitry, is configured to communicate with the cache controller; as well as The control block is configured to receive access requests generated by the plurality of computing units executing multiple computing cores; The control block is configured as follows: The count of access requests for shared data is compared with the number of computing units among the plurality of computing units that require the shared data; In response to: A first access request was detected from one of the plurality of computing units that require the shared data; and The count of access requests for the shared data is determined to be equal to the number of computing units that require the shared data; A single access request for the shared data is transmitted to the cache controller via the matrix data memory; and In response to receiving the shared data, the shared data is broadcast to the computing units that require the shared data.

18. The apparatus of claim 17, wherein, None of the computing units that require the shared data receive an instruction from any of the computing units that require the shared data to share the shared data.

19. The apparatus of claim 17, wherein, The circuit is configured as follows: For each access request for the shared data received from the processor, increment the count of access requests requiring the shared data.

20. The apparatus of claim 17, wherein the circuitry is configured to store broadcast characteristics of the shared data, the broadcast characteristics including one or more of the following: A broadcast identifier, wherein the broadcast identifier is a unique identifier that identifies the shared data; and The base address specifies the storage location where the beginning of the shared data is stored.

Images