Method and system for operating graphic processor

Abstract

A method for operating a graphics processor is disclosed. The method for operating a graphics processor according to the present invention comprises: a step of rearranging the row order of a matrix based on the degree of similarity with respect to different rows; a step of generating a zero bitmap, which is information distinguishing zero warps and other warps for the rearranged matrix; a step of scheduling at least a portion of the matrix multiplication operation for the rearranged matrix to be executed in parallel on a plurality of cores of the graphics processor; and a step of performing an operation excluding the operation for zero warps when executing the matrix multiplication operation using the zero bitmap.

Description

Technology Field

[0001] The present invention relates to a method and system for operating a graphics processor. Background Technology

[0003] A graphical neural network model is a model that performs information processing to represent connections as graphs, such as in social networks, protein structures, and paper citations.

[0004] Graphics neural network models perform matrix operations on the connectivity of nodes. The connectivity of the i-th node is represented by a 1 or 0 in the i-th row, where 1 signifies a connection. Such a matrix is ​​called an adjacency matrix, and due to its structure containing a large number of zero values, it is also referred to as a sparse matrix. Operations on sparse matrices are allocated to the cores of a graphics processor (GPU) for parallel processing; however, operations involving numerous zeros are also included in threads, causing inefficiency. The problem to be solved

[0006] The technical problem that the present invention aims to solve is to provide a method and system for operating a graphics processor capable of optimizing matrix multiplication operations of a graphics neural network model by rearranging sparse matrices to more clearly distinguish zero and non-zero regions and by thread arrangement considering zero skipping. means of solving the problem

[0008] To solve the above technical problem, a method for operating a graphics processor according to an embodiment of the present invention may be executed by a host processor and a graphics processor and may include: a step of rearranging the row order of a matrix based on the degree of similarity for different rows; a step of generating a zero bitmap, which is information distinguishing zero warps and other warps for the rearranged matrix; a step of scheduling at least a portion of the matrix multiplication operation for the rearranged matrix to be executed in parallel on a plurality of cores of the graphics processor; and a step of performing an operation excluding the operation for zero warps when executing the matrix multiplication operation using the zero bitmap.

[0009] In one embodiment of the present invention, the degree of similarity may be determined by cosine similarity.

[0010] In one embodiment of the present invention, the rearrangement step may include: a step of determining a first pivot row; a step of determining the degree of similarity between the first pivot row and other rows; and a step of arranging the pivot row and a preset number of rows adjacent to each other in order of the degree of similarity with the second pivot row.

[0011] In one embodiment of the present invention, the rearrangement step may further include: a step of determining a second pivot row; and a step of arranging a preset number of rows adjacently to the second pivot row in the same manner as the first pivot row.

[0012] In one embodiment of the present invention, the scheduling step may include: a step of allocating a plurality of thread blocks to each of a plurality of cores; a step of calculating the overall average of the non-zero number; a step of calculating the non-zero number for the additionally placed thread blocks; a step of comparing the average of the non-zero number with the non-zero number for the additionally placed thread blocks; and a step of allocating the additional thread blocks to any one of the plurality of cores based on the comparison result.

[0013] In one embodiment of the present invention, the scheduling step may further include the step of allocating the additional thread block to the core with the smallest average number of non-zeros among the previously allocated thread blocks when the number of non-zeros of the additional thread block is greater than the overall average number of non-zeros.

[0014] In one embodiment of the present invention, the scheduling step may further include the step of allocating the additional thread block to the core with the largest average number of non-zeros among the previously allocated thread blocks when the number of non-zeros of the additional thread block is equal to or smaller than the overall average number of non-zeros.

[0015] In addition, to solve the above technical problem, an operating system of a graphics processor according to an embodiment of the present invention includes a memory in which at least a portion of an operating method of the graphics processor is stored; a host processor that executes the method stored in the memory; and a graphics processor, wherein the host processor rearranges the row order of a matrix based on the degree of similarity for different rows, generates a zero bitmap which is information distinguishing zero warps and other warps for the rearranged matrix, generates a thread for a matrix multiplication operation for the rearranged matrix and transmits it to the graphics processor, and the graphics processor schedules at least a portion of the matrix multiplication operation for the rearranged matrix to be executed in parallel on a plurality of cores of the graphics processor, and can perform operations excluding operations for zero warps when executing the matrix multiplication operation using the zero bitmap. Effects of the invention

[0017] The present invention has the effect of optimizing matrix multiplication operations of a graphical neural network model by rearranging sparse matrices to more clearly distinguish zero and non-zero regions and by thread arrangement considering zero skipping. Brief explanation of the drawing

[0019] FIG. 1 shows an operating system of a graphics processor according to an embodiment of the present invention. Figure 2 shows nodes that are commonly connected at two nodes and nodes that are not. Figures 3 to 5 are intended to explain the process of rearranging a sparse matrix. Figure 6 shows the relationship between the rearranged sparse matrix and the zero bitmap. Figure 7 shows the number of theorems included in the thread blocks allocated to multiple cores. Figure 8 exemplarily shows the average number of thread blocks allocated to multiple cores and the total number of thread blocks. FIG. 9 illustrates a method of operating a graphics processor according to an embodiment of the present invention. FIG. 10 shows in detail a part of the configuration of a method for operating a graphics processor according to an embodiment of the present invention. FIG. 11 shows in detail a part of the configuration of a method for operating a graphics processor according to an embodiment of the present invention. Figure 12 shows standardized performance indices according to the scale of computation. Figure 13 shows the computational compression ratio according to the computational scale. Figure 14 shows the number of thread blocks and standardized performance indices according to the scale of computation. Figure 15 shows the number of average theorems included in thread blocks allocated to multiple cores. Specific details for implementing the invention

[0020] The present invention is capable of various modifications and may have various embodiments, and specific embodiments are illustrated in the drawings and described in detail. However, this is not intended to limit the present invention to specific embodiments, and it should be understood that it includes all modifications, equivalents, and substitutions that fall within the spirit and scope of the invention.

[0021] In describing the present invention, if it is determined that a detailed description of related known technology may obscure the essence of the present invention, such detailed description is omitted.

[0022] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

[0024] FIG. 1 shows an operating system of a graphics processor according to an embodiment of the present invention.

[0025] Referring to FIG. 1, an operating system of a graphics processor according to one embodiment of the present invention includes a host processor (110), a memory (120), and a graphics processor (130).

[0026] The host processor (110) may be a CPU (central processing unit).

[0027] The host processor (110) can write data to memory (120) or execute data stored as a program.

[0028] The host processor (110) can create a thread and pass it to the graphics processor (130).

[0029] The memory (120) includes a relocation unit (121) and a zero bit map generation unit (122).

[0030] The rearrangement unit (121) rearranges the rows of the matrix to be applied to the matrix multiplication.

[0032] To explain the rearrangement of rows, we first examine the relationship between the connectivity characteristics between nodes and the matrix.

[0033] Figure 2 shows nodes that are commonly connected at two nodes and nodes that are not.

[0034] For example, node v i wa v j is common to v k and v l It is connected to. Meanwhile, node v i is vm It is connected to but v j is v m It is not connected to.

[0035] When the connection relationships between nodes are represented as an adjacency matrix, rows with indices for connected nodes are marked as 1, and rows with indices for unconnected nodes are marked as 0.

[0036] For example, node v i wa v k The connection of is matrix element a ik =1 or a ki Represented as =1.

[0037] Considering these relationships, different nodes v i , v j Regarding the commonly connected nodes, it can be seen that in the i-th and j-th rows, the k-th and l-th columns commonly have a value of 1, i.e., non-zero. That is, a ik and a jk is commonly 1, and a il and a jl Figure 1.

[0038] In contrast, node v i is v m Is it connected to v j is v m When not connected to a im =1, a jm It can be seen that it has =0.

[0039] As seen, the degree of similarity in the connection between two nodes can be proportional to the number of columns that simultaneously have a value of 1.

[0040] This is node v i wa v j row vector i and row j When represented as such, it can be expressed as the cosine similarity of the two row vectors. Expressed as a formula, this is the similarity score S. c It can be obtained from the following [Mathematical Formula 1].

[0042]

[0044] FIGS. 3 to 5 are intended to illustrate the process of rearranging a sparse matrix. For brevity, the size of the matrix is ​​reduced, and empty spaces represent parts where the matrix element value is 0, while colored spaces represent parts where the matrix element value is 1. The rearrangement can be performed by the host processor (110) executing a method recorded in the rearrangement unit (121).

[0045] Figure 3 visually represents the matrix before rearrangement. The index of the row before rearrangement is displayed on the left.

[0046] Figure 4 shows a matrix rearranged based on the pivot row.

[0047] The pivot row can be selected from any row. In one embodiment of the present invention, the matrix at index 0 is determined as the 0th pivot row (P0).

[0048] The host processor (110) calculates the cosine similarity between the pivot row (P0) and the other rows.

[0049] The host processor (110) places other rows adjacent to the pivot row (P0) in descending order of cosine similarity. At this time, the number of rows placed adjacent to the pivot row (P0), including the pivot row (P0), corresponds to the height (TB_H) of the thread block. In the example, since the height (TB_H) of the thread block is 4, the three rows (rows 3, 6, and 11) that have high cosine similarity to the pivot row (P0), including the pivot row (P0), are placed adjacent to the pivot row (P0).

[0050] Through the process described above, rows 0, 3, 6, and 11 before relocation are placed in rows 0, 1, 2, and 3, respectively, which are relocated indices 0, 3, 6, and 11.

[0051] In Fig. 4, the right side of the matrix shows the cosine similarity score (S0) for pivot row 0 (P0), which is partially sorted in descending order for rows 0' to 3'.

[0052] Referring to Fig. 5, the first pivot row (P1) is placed again in the relocation index 4' row.

[0053] Pivot row 1 (P1) can be selected as the row with the earliest sequence number among the unrelocated rows. As a variation, pivot row 1 (P1) can be selected in correspondence with the similarity score with pivot row 0 (P0). For example, pivot row 1 (P1) can be selected as the row with the highest similarity to pivot row 0 (P0) or as the row with the lowest similarity among the unrelocated rows.

[0054] Rows to be placed adjacent to pivot row 1 (P1) can also be selected in order of highest cosine similarity. Figure 5 shows rows 16, 4, and 5 selected before rearrangement, respectively. On the right, the cosine similarity scores for pivot row 1 (P1) are shown in descending order.

[0055] The zero bitmap generation unit (122) generates zero bitmap information for the matrix rearranged by the rearrangement unit (121).

[0056] Figure 6 shows the relationship between a rearranged sparse matrix and a zero bitmap.

[0057] The rectangular boxes appended within the matrix represent warps. A warp is the smallest unit of matrix operation and is a size set during the manufacturing of the graphics processor. In the example, a 2x2 warp is shown.

[0058] Zero bitmap information can be stored separately for each thread block. A thread block is the smallest unit of a set of threads allocated to a core (131).

[0059] One thread block is exemplified as containing 8 warps. A thread block may contain 16 warps. This depends on the pre-configured thread block size.

[0060] The zero bitmap contains location information of a warp and information on whether the warp contains non-zero elements. The zero bitmap is recorded as 1 if it contains non-zero elements, and as 0 if all elements within the warp are 0. If all elements within the warp are 0, the warp is called a zero warp.

[0061] In the example of Fig. 6, the thread block of index 0 is represented by the 8-bit information of zero bitmap information Z[0] = [11110101]. This means that the 5th and 7th warps are zero warps and the remaining warps are non-zero warps.

[0062] A zero bitmap can be used as meta-information to omit operations for zero warps when executing matrix multiplication operations.

[0064] The graphics processor (130) includes a core (131), a register (132), and a scheduler (133).

[0065] The core (131) performs a matrix multiplication operation of warp units and outputs the result.

[0066] The core (131) is composed of multiple cores for parallel operation.

[0067] Register (132) can store zero bitmap information for each thread block.

[0068] The scheduler (133) allocates thread blocks to each core (131) using zero bitmap information.

[0069] Figure 7 shows the number of thread blocks included in the multiple cores (131).

[0070] For thread blocks of a rearranged matrix, the distributions of non-zero warps and zero warps, respectively, may become more concentrated compared to before rearrangement. In other words, the variation in the number of non-zero warps contained within a thread block may increase. This is because rearranging rows based on cosine similarity increases the probability that zero warps will be placed adjacent to zero warps and non-zero warps adjacent to non-zero warps.

[0071] For example, core 0 (131) may be assigned warps with an average of 6.6 themes, and core 32 (131) may be assigned warps with 0.52 themes. If the operation of the core (131) is executed only for the warps with the themes, a deviation in the amount of operation of each core (131) may accumulate. This may cause idle cores (131) to occur, leading to computational inefficiency.

[0072] The scheduler (133) allocates thread blocks based on the number of non-zero operations to mitigate variations in the amount of computation and to distribute non-zero warps evenly among the cores (131).

[0074] FIG. 8 illustrates the average number of zeros and the total number of zeros of thread blocks allocated to multiple cores (131) as an example. In the figure, for convenience of explanation, zero bitmap information is represented as 4-bit information. That is, in this example, one thread block contains 4 warps.

[0075] N thread blocks may be allocated to each core (131) first. For example, N may be 4. On the other hand, N may have a number higher than 4. This is because, depending on the number of thread blocks containing zero warps, relevant information may be removed from the register (132), thereby securing space to allocate additional thread blocks. Therefore, the number of thread blocks allocated to each core (131) may differ.

[0076] In the example of Fig. 8, TB slot N represents the slot index of the thread block, tid represents the thread block index, Z[tid] represents the zero bitmap information of the thread block, and nnz represents the number of non-zero warps.

[0077] The average number of nonzero warps (Avg. nnz) assigned to core 0 (131), core 1 (131), core 2 (131) and core 3 (131) are 2.5, 1.3, 2.7, and 3.1, respectively. This represents the average of the number of nonzero warps assigned so far, and the average value may be cumulatively updated for additional thread blocks assigned.

[0078] It is an example that the global nnz is 2.4. The global nnz may be pre-calculated or may be based on the thread blocks allocated so far. Since the global nnz is not a value that fluctuates significantly, it may not be updated.

[0079] When allocating additional thread blocks, the scheduler (133) determines the core (131) to be allocated based on the result of comparing the number of additional thread blocks with the average of the total number of thread blocks.

[0080] Specifically, the scheduler (133) allocates additional thread blocks to the core (131) with the smallest average number of thread blocks allocated so far when the number of additional thread blocks is greater than the average number of all thread blocks.

[0081] The scheduler (133) allocates additional thread blocks to the largest core (131) with the smallest average number of allocated thread blocks when the number of additional thread blocks is equal to or smaller than the average number of all thread blocks.

[0082] The core (131) executes thread blocks allocated by the scheduler (133), but zero warp can be skipped in the calculation.

[0084] Hereinafter, a method for operating a graphics processor according to an embodiment of the present invention is described. The method for operating a graphics processor according to an embodiment of the present invention can be performed by a system for operating a graphics processor according to an embodiment of the present invention.

[0085] FIG. 9 illustrates a method of operating a graphics processor according to an embodiment of the present invention.

[0086] Referring to FIG. 9, the host processor (110) rearranges the sparse matrix (S10).

[0087] As described above, the sparse matrix can be rearranged so that it is placed adjacently in order of highest cosine similarity based on the pivot row at each height of the thread block.

[0088] The host processor (110) generates a zero bitmap separated by thread block for the relocated sparse matrix (S20). The relocated sparse matrix and zero bitmap information can be converted into information constituting a kernel and transmitted to the graphics processor (130).

[0089] The graphics processor (130) performs scheduling for the thread block (S30).

[0091] Figure 10 shows in detail the step (S10) of rearranging a sparse matrix.

[0092] Referring to FIG. 10, the host processor (110) determines one pivot row in the sparse matrix (S11).

[0093] The host processor (110) calculates a similarity score between the pivot row and other rows (S12).

[0094] The similarity score can be calculated as a cosine similarity score according to [Equation 1].

[0095] The host processor (110) rearranges the pivot row and other rows sequentially to the height of the thread block, and the rows rearranged together with the determined pivot row are selected in order of high similarity score (S13).

[0096] The host processor (110) may repeat steps S11 through S13 for the rows that have not been relocated (S14). Step S14 may be repeated until all rows are relocated.

[0098] Figure 11 shows the thread block scheduling step (S13) in detail.

[0099] Referring to FIG. 11, the scheduler (133) first places thread blocks sequentially or randomly among multiple cores (131) (S31). The number of thread blocks allocated to each core (131) may vary.

[0100] The scheduler (133) calculates the average number of non-zero warps and the overall average for each core (131) (S32). The number of non-zero warps refers to the number of non-zero warps.

[0101] The scheduler (133) calculates the number of additional thread blocks to be placed (S33).

[0102] The scheduler (133) determines the number (TB) of thread blocks to be additionally placed. nnz ) is the number average as an overall topic (Global nnz Determine whether it is greater than ) (S34). If it is greater, proceed to step S35, and if it is smaller, proceed to step S36.

[0103] The scheduler (133) is TB nnz Global nnz In this case, an additional thread block is allocated to the core (131) with the smallest average number of allocated thread blocks (S35).

[0104] The scheduler (133) is TB nnz ≤ Global nnzIn this case, additional thread blocks are allocated to the core (131) with the largest average number of allocated thread blocks (S36).

[0105] The scheduler (133) can update the number average per core (131) when an additional thread block is allocated (S37). The update of the number average per core (131) only needs to be calculated for the additional thread block.

[0107] Figure 12 shows standardized performance indices according to the scale of computation.

[0108] The comparison criterion represents the computational performance of the method that does not use matrix rearrangement, scheduling, and zero skip, and is standardized to 1.

[0109] The comparative example is based on the computation acceleration method introduced in Y. Wang, B. Feng, Z. Wang, G. Huang, and Y. Ding, “TC-GNN: Bridging sparse GNN computation and dense tensor cores on GPUs,” USENIX Annual Technical Conference, July 2023.

[0110] The first embodiment is an embodiment of the present invention using only the relocation and zero-warp omission methods, and the second embodiment represents the application of all of the relocation, zero-warp omission, and thread block scheduling techniques based on non-zero numbers.

[0111] Computational performance was evaluated for small (CU, UT, UW), medium (CO, CS, WK, AZ, CA), and large (PM, FL) matrices, respectively.

[0112] For the evaluation, small, medium, and large graph datasets used for evaluating the performance of each graph neural network (GNN) were used. Each dataset refers to Cornell (CU), Texas (UT), Wisconsin (UW), Cora (CO), Citeseer (CS), WikiCS (WK), Amazon (AZ), Coauthor (CA), Pubmed (PM), and Flicker (FL), respectively.

[0113] Since there is limited room for performance improvement for small graph data sets, the comparative example and the embodiment of the present invention showed similar levels of performance.

[0114] In contrast, as the graph size increased, the first and second embodiments showed improved performance compared to the comparison.

[0115] The second embodiment improves not only relocation and zero skip but also load balancing issues, and can further improve performance compared to the first embodiment.

[0117] Figure 13 shows the computational compression ratio according to the computational scale.

[0118] A comparative example is as shown in Fig. 12.

[0119] In the embodiments of the present invention, a large compression ratio means that the number of warps is small for the actual calculated nonzero, and greater computational efficiency can be achieved.

[0120] While the comparative example showed a compression rate of 79.4%, the embodiment of the present invention was able to achieve a compression rate of 85.2%.

[0122] Figure 14 shows the number of thread blocks and standardized performance indices according to the scale of computation.

[0123] The left side of FIG. 14 is the number of thread blocks allocated to the core (131). In the embodiment of the present invention, since zero warps are skipped and removed, their context data is removed from the register (132), making it possible to load additional thread blocks.

[0124] The right side of Fig. 14 compares standardized performance indices.

[0125] In an embodiment of the present invention, increased thread-level parallelism achieved a performance improvement of 1.48x and 1.41x for medium-scale and large-scale data sets, respectively.

[0127] Figure 15 shows the number of average theorems included in thread blocks allocated to multiple cores (131).

[0128] The comparison example at the top of Fig. 15 shows a round-robin thread block scheduling technique, and it can be seen that there is a large variation in the number between cores (131).

[0129] The bottom of FIG. 15 shows the deviation mitigated according to the thread block scheduling technique of an embodiment of the present invention. In the embodiment of the present invention, the standard deviation of the non-zero warp workload could be significantly mitigated from 1.411 to 0.429.

[0131] The terms used in this application are used merely to describe specific embodiments and are not intended to limit the invention. In this application, terms such as “comprising” or “having” are intended to indicate the presence of the features, numbers, steps, actions, components, parts, or combinations thereof described in the specification, and should be understood as not precluding the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof. Explanation of the symbols

[0133] 110: Host processor 120: Memory 121: Relocation Section 122: Zero Bitmap Generation Unit 130: Graphics processor 131: Core 132: Register 133: Scheduler

Claims

Claim 1 A method for operating a graphics processor, wherein a host processor and a graphics processor operate the graphics processor, comprising: a step of rearranging the row order of a matrix based on the degree of similarity to different rows; a step of generating a zero bitmap, which is information distinguishing between zero warps and other warps for the rearranged matrix; a step of scheduling at least a portion of the matrix multiplication operation for the rearranged matrix to be executed in parallel on a plurality of cores of the graphics processor; and a step of performing an operation excluding the operation for zero warps when executing the matrix multiplication operation using the zero bitmap. Claim 2 A method for operating a graphics processor according to claim 1, characterized in that the degree of similarity is determined by cosine similarity. Claim 3 A method for operating a graphics processor according to claim 1, wherein the rearrangement step comprises: a step of determining a first pivot row; a step of determining the degree of similarity between the first pivot row and other rows; and a step of arranging the pivot row and a preset number of rows adjacent to each other in order of the degree of similarity with the second pivot row. Claim 4 A method for operating a graphics processor according to claim 3, wherein the rearrangement step further comprises: a step of determining a second pivot row; and a step of arranging a preset number of rows adjacently to the second pivot row in the same manner as the first pivot row. Claim 5 A method for operating a graphics processor according to claim 1, wherein the scheduling step comprises: a step of allocating a plurality of thread blocks to a plurality of cores; a step of calculating an overall average of the non-zero number; a step of calculating the non-zero number for a thread block to be additionally placed; a step of comparing the average of the non-zero number with the non-zero number for the thread block to be additionally placed; and a step of allocating the additional thread block to any one of the plurality of cores based on the comparison result. Claim 6 A method for operating a graphics processor according to claim 5, wherein the scheduling step further comprises the step of allocating the additional thread block to the core with the smallest average number of non-zeros of the previously allocated thread blocks when the number of non-zeros of the additional thread block is greater than the overall average number of non-zeros. Claim 7 A method for operating a graphics processor according to claim 5, wherein the scheduling step further comprises the step of allocating the additional thread block to the core with the largest average number of non-zeros of the previously allocated thread blocks when the number of non-zeros of the additional thread block is equal to or smaller than the overall average number of non-zeros. Claim 8 An operating system for a graphics processor comprising: a memory in which at least a portion of the operating method of the graphics processor is stored; a host processor that executes the method stored in the memory; and a graphics processor, wherein the host processor rearranges the row order of a matrix based on the degree of similarity to different rows, generates a zero bitmap which is information distinguishing zero warps and other warps for the rearranged matrix, generates a thread for a matrix multiplication operation for the rearranged matrix and transmits it to the graphics processor, and the graphics processor schedules at least a portion of the matrix multiplication operation for the rearranged matrix to be executed in parallel on a plurality of cores of the graphics processor, and performs operations excluding zero warps when executing the matrix multiplication operation using the zero bitmap.

Images