Processing device and parallel processing method for processing device
By introducing scheduling and computation circuits into the processing device, the second kernel function is processed in parallel with the dependent data before the execution of the first kernel function, which solves the problem of excessive execution time in the prior art and achieves a more efficient processing speed.
Patent Information
- Application Number
- CN202410628462.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-05-20
- Publication Date
- 2025-10-17
- Estimated Expiration
- 2044-05-20
AI Technical Summary
Existing processing devices need to execute kernel functions one by one when executing a single function string, resulting in a long processing time that is difficult to shorten.
By introducing a scheduling circuit and multiple computing circuits into the processing device, the scheduling circuit receives function serial assignments and assigns the computing circuits to execute kernel functions. The computing circuits calculate the dependent data before executing the first kernel function and start executing the second kernel function before the first kernel function is completed, thus achieving parallel processing.
This significantly reduces the time required for the processing device to execute a single function string, thus reducing the overhead of parallel processing.
Smart Images
Figure CN118394522B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to an electronic device, and particularly relates to a processing device and a parallel processing method for the processing device. BACKGROUND
[0002] A current processing device (such as a graphics processing unit (GPU)) can receive a function stream and execute a plurality of kernel functions in the function stream. When executing the same function stream, the processing device needs to execute the plurality of kernel functions in the function stream one by one. For example, the function stream includes a first kernel function, a second kernel function, and a third kernel function. The processing device executes the first kernel function first, and then executes the second kernel function. Next, the processing device executes the second kernel function, and then executes the third kernel function. How to shorten the time length of executing a single function stream is one of the research focuses of the person skilled in the art. SUMMARY
[0003] The present application relates to a processing device and a parallel processing method for the processing device. The time length of executing a single function stream by the processing device can be shortened.
[0004] According to an embodiment of the present application, a processing device includes a plurality of computing circuits and a scheduling circuit. The scheduling circuit is coupled to the plurality of computing circuits. The scheduling circuit receives a function stream and obtains a first kernel function, a second kernel function, and at least one thread group of the first kernel function from the function stream. The scheduling circuit assigns a first computing circuit among the plurality of computing circuits to execute the first kernel function, and assigns a second computing circuit among the plurality of computing circuits to execute the second kernel function. The first computing circuit calculates at least one dependent data before completing execution of the first kernel function. The second computing circuit receives the at least one dependent data according to the at least one thread group, and starts to execute the second kernel function according to the at least one dependent data before the first computing circuit completes execution of the first kernel function.
[0005] According to an embodiment of the present invention, a parallel processing method is used for a processing device. The processing device includes a plurality of computing circuits. The parallel processing method includes: receiving a function string, and obtaining a first kernel function, a second kernel function, and at least one thread group collaboration of the first kernel function from the function string; assigning a first computing circuit among the plurality of computing circuits to execute the first kernel function, and assigning a second computing circuit among the plurality of computing circuits to execute the second kernel function; the first computing circuit calculates at least one dependency data before completing the execution of the first kernel function; and the second computing circuit receives the at least one dependency data based on the at least one thread group collaboration, and starts executing the second kernel function based on the at least one dependency data before the first computing circuit completes the execution of the first kernel function.
[0006] Based on the above, the first computing circuit calculates at least one dependency data before completing the execution of the first kernel function. The second computing circuit receives the at least one dependency data in cooperation with the at least one thread group, and starts executing the second kernel function based on the at least one dependency data before the first computing circuit completes the execution of the first kernel function. Therefore, the second computing circuit does not need to wait until the first kernel function is executed before executing the second kernel function. In this way, the present invention can shorten the time required for the processing device to execute a single function string. BRIEF DESCRIPTION OF THE DRAWINGS
[0007] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings of the embodiments will be briefly introduced below. Obviously, the drawings in the following description only relate to some embodiments of the present invention, but are not intended to limit the present invention.
[0008] Figure 1 is a schematic diagram of a processing device according to an embodiment of the present invention.
[0009] Figure 2 FIG. 4 is a schematic diagram illustrating the execution of a function string according to an embodiment of the present invention.
[0010] Figure 3 FIG. 4 is a schematic diagram illustrating the execution of a function string according to an embodiment of the present invention.
[0011] Figure 4 is a schematic diagram of a processing device according to an embodiment of the present invention.
[0012] Figure 5 is a flow chart of a parallel processing method according to an embodiment of the present invention.
[0013] Description of Reference Numerals
[0014] 100: Processing device
[0015] 110_1, 110_2, ..., 110_n: calculation circuit
[0016] 120: scheduling circuit
[0017] 130: memory circuit
[0018] A1-A4, B: parameter
[0019] CU: computing unit
[0020] DD, DD1, DD2, DD3, DD4: dependent data
[0021] HBM1, HBM2: high bandwidth memory
[0022] K1: first kernel function
[0023] K2: second kernel function
[0024] ODR: order
[0025] PTC: thread group cooperation
[0026] S100: parallel processing method
[0027] S110-S140: step
[0028] STM: function string
[0029] TH1_1-TH1_4, TH2_1-TH2_4: thread group DETAILED DESCRIPTION
[0030] Reference will now be made in detail to the exemplary embodiments of the present application, examples of which are illustrated in the accompanying drawings. Wherever possible, the same reference numbers will be used in the different drawings and the description to refer to the same or like parts.
[0031] The term "coupled" (or connected) used in the detailed description of the specification (including the claims) can refer to any direct or indirect connection. For example, if a first device is coupled (or connected) to a second device, it should be interpreted that the first device can be directly connected to the second device, or the first device can be indirectly connected to the second device through other devices or some connection means. The terms "first", "second", etc. mentioned in the detailed description of the specification (including the claims) are used to name components or to distinguish different embodiments or ranges, and are not used to limit the upper or lower limit of the number of components, nor to limit the order of components. In addition, wherever possible, the same reference numbers of components / means / steps are used in the drawings and embodiments to represent the same or similar parts. Components / means / steps with the same reference numbers or the same terms used in different embodiments can be referred to each other for relevant description.
[0032] Reference is made to Figure 1 ,Figure 1 is a schematic diagram of a processing device according to an embodiment of the present application. In this embodiment, the processing device 100 can be a graphics processing unit (GPU) or an artificial intelligence (AI) chip. The processing device 100 includes computing circuits 110_1-110_n and a scheduling circuit 120. The computing circuits 110_1-110_n each execute a different kernel function. In this embodiment, the kernel functions can be any functions applied in the field of machine learning or in the field of AI.
[0033] In this embodiment, the scheduling circuit 120 is coupled to the computing circuits 110_1-110_n. The scheduling circuit receives a function string STM. The scheduling circuit 120 obtains a first kernel function K1, a second kernel function K2, and a thread block cooperation PTC of the first kernel function K1 from the function string STM. The scheduling circuit 120 can dispatch the first kernel function K1 and the second kernel function K2 to the computing circuits 110_1-110_n. In this embodiment, the scheduling circuit 120 assigns the computing circuit 110_1 to execute the first kernel function K1 and assigns the computing circuit 110_2 to execute the second kernel function K2.
[0034] In this embodiment, the computing circuit 110_1 computes the dependent data DD before completing execution of the first kernel function K1. Further, the dependent data DD is data generated during execution of the first kernel function K1. In other words, the dependent data DD is generated before the first kernel function K1 ends. The computing circuit 110_2 receives the dependent data DD according to the thread block cooperation PTC. The computing circuit 110_2 starts to execute the second kernel function K2 according to the dependent data DD before the computing circuit 110_1 completes execution of the first kernel function K1.
[0035] It is worth mentioning that the computing circuit 110_2 does not need to wait until the first kernel function K1 ends before starting to execute the second kernel function K2. In this way, the processing device 100 can greatly shorten the length of time required to execute a single function string STM.
[0036] In hardware form, the computing circuits 110_1-110_n and the scheduling circuit 120 can be implemented as logic circuitry on an integrated circuit. For example, the functions of the computing circuits 110_1-110_n and the scheduling circuit 120 can be implemented in various logic blocks, modules, and circuits in one or more controllers, hardware controllers, microcontrollers (MCUs), hardware processors, microprocessors (MPUs), application-specific integrated circuits (ASICs), digital signal processors (DSPs), field programmable gate arrays (FPGAs), central processing units (CPUs), or other processing units. The functions of the computing circuits 110_1-110_n and the scheduling circuit 120 can be implemented as hardware circuits, such as various logic blocks, modules, and circuits in an integrated circuit, using hardware description languages (e.g., Verilog HDL or VHDL) or other suitable programming languages.
[0037] In software or firmware form, the computing circuits 110_1-110_n can be implemented as programming codes. For example, the computing circuits 110_1-110_n and the scheduling circuit 120 can be implemented using general programming languages (e.g., C, C++, or assembly language) or other suitable programming languages. The programming codes can be recorded or stored in a "non-transitory machine-readable storage medium".
[0038] In this embodiment, the processing device 100 further includes a memory circuit 130. The computing circuit 110_1 computes the dependent data DD in cooperation with the PTC before completing the execution of the first kernel function K1 and stores the dependent data DD in the memory circuit 130. The computing circuit 110_2 receives the dependent data DD from the memory circuit 130 in cooperation with the PTC before completing the execution of the first kernel function K1.
[0039] The memory circuit 130 can be a memory card, a read only memory (ROM), a FLASH memory, a programmable logic circuit, or other semiconductor memory. The storage device includes a hard disk drive (HDD), a solid-state drive (SSD), or other storage device.
[0040] For example, the computing circuit 110_2 learns that the second kernel function K2 needs the dependency data DD according to the thread group cooperative PTC. Therefore, the computing circuit 110_2 polls the memory circuit 130 to determine whether the memory circuit 130 stores the dependency data DD corresponding to the thread group cooperative PTC. When the computing circuit 110_2 determines that the memory circuit 130 stores the dependency data DD, the computing circuit 110_2 receives the dependency data DD from the memory circuit 130 and uses the dependency data DD to execute at least one thread group of the second kernel function K2.
[0041] In some embodiments, the function string STM can include a third kernel function or more kernel functions and a plurality of thread group cooperatives. The computing circuits 110_1-110_n respectively receive dependency data according to at least one of the plurality of thread group cooperatives and execute the kernel functions according to the received dependency data. Therefore, the embodiments are not limited by the number of kernel functions and the number of thread group cooperatives.
[0042] Reference is made to Figure 1 and Figure 2 , Figure 2 is a diagram illustrating execution of a function string according to an embodiment of the present application. In this embodiment, the function string STM includes at least a first kernel function K1 and a second kernel function K2. The first kernel function K1 includes thread groups TH1_1-TH1_4. The second kernel function K2 includes thread groups TH2_1-TH2_4.
[0043] In this embodiment, the computing circuit 110_1 sequentially executes the thread groups TH1_1-TH1_4 in the first kernel function K1 according to the execution order ODR. The computing circuit 110_2 sequentially executes the thread groups TH2_1-TH2_4 in the second kernel function K2 according to the execution order ODR.
[0044] For example, the computing circuit 110_1 generates the dependency data DD after completing the thread group TH1_1, and starts to execute the thread group TH1_2. Based on the thread group cooperation PTC, the thread group TH2_1 is executed with the dependency data DD. Therefore, the computing circuit 110_2 can execute the thread group TH2_1 upon receiving the dependency data DD. The execution period of the thread group TH2_1 partially overlaps with the execution period of the thread group TH1_2. In this way, the computing circuit 110_2 does not need to wait until the first kernel function K1 is completed before executing the second kernel function K2. The execution overhead of the second kernel function K2 can be reduced.
[0045] It should be noted that the computing circuit 110_2 only needs to know the dependency data DD based on the thread group cooperation PTC. Therefore, the computing circuit 110_2 does not need additional parallel processing circuit to know the dependency data DD. Therefore, the parallel processing overhead of the second kernel function K2 can be greatly reduced.
[0046] For example, the start running time point of the thread group TH2_1 is equal to the start running time point of the thread group TH1_2. For another example, the start running time point of the thread group TH2_1 is later than the start running time point of the thread group TH1_2.
[0047] The number of thread groups of the present application is not limited to the present embodiment. The first kernel function K1 of the present application can include a plurality of thread groups. The second kernel function K2 can include at least one thread group.
[0048] In some embodiments, the function string STM includes the first kernel function K1, the second kernel function K2, and a third kernel function (not shown). For example, a third computing circuit other than the computing circuits 110_1, 110_2 is assigned to execute the third kernel function. The third computing circuit can know, according to the thread group cooperation PTC, that a first thread group of the third kernel function is executed according to the dependency data DD. Therefore, the third computing circuit can execute the first thread group of the third kernel function upon receiving the dependency data DD. In other words, in the function string STM, the first thread group of the third kernel function and the thread group TH2_1 of the second kernel function K2 can be executed in parallel. In this way, the third computing circuit does not need to wait until the first kernel function K1 and the second kernel function K2 are completed before executing the third kernel function.
[0049] Please refer to Figure 1 and Figure 3 , Figure 3is an execution diagram of a function string according to an embodiment of the present application. In this embodiment, the function string STM includes at least a first kernel function K1 and a second kernel function K2. In this embodiment, the computing circuit 110_1 sequentially executes the thread groups TH1_1-TH1_4 in the first kernel function K1. The computing circuit 110_2 sequentially executes the thread groups TH2_1-TH2_4 in the second kernel function K2. The first kernel function K1 can be a convolution function. The first kernel function K1 is, for example, "conv(A, B)". Thus, the thread group TH1_1 can perform a convolution operation on the parameters A1, B to generate dependent data DD1. The dependent data DD1 is, for example, a convolution operation result of the parameters A1, B. The second kernel function K2 can be an activation function (e.g., a ReLU function). The second kernel function K2 is, for example, "relu(DD)". Thus, the thread group TH2_1 can perform an operation (e.g., "relu(DD1)") on the dependent data DD1 using the activation function.
[0050] After the completion of the thread group TH1_1, the computing circuit 110_1 then executes the thread group TH1_2. The thread group TH1_2 can perform a convolution operation on the parameters A2, B to generate dependent data DD2. Thus, the thread group TH2_2 can perform an operation (e.g., "relu(DD2)") on the dependent data DD2 using the activation function.
[0051] After the completion of the thread group TH1_2, the computing circuit 110_1 then executes the thread group TH1_3. The thread group TH1_3 can perform a convolution operation on the parameters A3, B to generate dependent data DD3. Thus, the thread group TH2_3 can perform an operation (e.g., "relu(DD3)") on the dependent data DD3 using the activation function.
[0052] After the completion of the thread group TH1_3, the computing circuit 110_1 then executes the thread group TH1_4. The thread group TH1_4 can perform a convolution operation on the parameters A4, B to generate dependent data DD4. Thus, the thread group TH2_4 can perform an operation (e.g., "relu(DD4)") on the dependent data DD4 using the activation function.
[0053] Please refer to Figure 1 and Figure 4 , Figure 4is a schematic diagram of a processing device according to an embodiment of the present application. The processing device 100 is implemented by a graphics processor. In the present embodiment, the graphics processor comprises a plurality of compute units CU, a dispatch circuit 120, and high bandwidth memories HBM1, HBM2. In the present embodiment, the dispatch circuit 120 receives a string of kernel functions STM. The dispatch circuit 120 obtains a first kernel function K1, a second kernel function K2, and a thread group co-schedule PTC of the first kernel function K1 from the string of kernel functions STM. The dispatch circuit 120 can organize one or more compute units CU as a first compute unit group according to the first kernel function K1, and take the first compute unit group as a compute circuit 110_1. The dispatch circuit 120 can organize one or more compute units CU other than the compute circuit 110_1 as a second compute unit group according to the second kernel function K2, and take the second compute unit group as a compute circuit 110_2.
[0054] For example, the number of compute units CU of the compute circuits 110_1, 110_2 can be adjusted based on the dispatch of the dispatch circuit 120. The present application is not limited to the number of compute units CU of the compute circuits 110_1, 110_2.
[0055] In the present embodiment, the compute circuit 110_1 executes the first kernel function K1 using the first compute unit group. The compute circuit 110_2 executes the second kernel function K2 using the second compute unit group. For example, the compute circuit 110_1 executes at least one thread group of the first kernel function K1 using at least one compute unit CU of the first compute unit group. The compute circuit 110_2 executes at least one thread group of the second kernel function K2 using at least one compute unit CU of the second compute unit group.
[0056] For example, each compute unit CU comprises a plurality of execution units. The plurality of execution units respectively execute at least one thread of at least one thread group or a single thread group of a kernel function.
[0057] In the present embodiment, at least one of the high bandwidth memories HBM1, HBM2 can be taken as a memory circuit 130.
[0058] The operations of the compute circuits 110_1, 110_2 and the memory circuit 130 have been clearly described in the embodiments of Figures 1 to 3 and will not be repeated here.
[0059] Please refer to Figure 1 and Figure 5 , Figure 5is a flowchart of a parallel processing method according to an embodiment of the present application. In this embodiment, the parallel processing method S100 is applied to a processing device 100. The parallel processing method S100 includes steps S110-S140. In step S110, the scheduling circuit 120 receives a function string STM. The scheduling circuit 120 obtains a first kernel function K1, a second kernel function K2, and a thread group coordination PTC of the first kernel function K1 from the function string STM.
[0060] In step S120, the scheduling circuit 120 assigns the computing circuit 110_1 (i.e., a first computing circuit) to execute the first kernel function K1, and assigns the computing circuit 110_2 (i.e., a second computing circuit) to execute the second kernel function K2.
[0061] In step S130, the computing circuit 110_1 calculates the dependent data DD before completing execution of the first kernel function K1.
[0062] In step S140, the computing circuit 110_2 receives the dependent data DD according to the thread group coordination PTC. The computing circuit 110_2 starts to execute the second kernel function K2 according to the dependent data DD before the computing circuit 110_1 completes execution of the first kernel function K1 in step S140.
[0063] The operation details of steps S110-S140 have been clearly described in the embodiment of Figures 1 to 3 and will not be repeated here.
[0064] In summary, a first computing circuit of a processing device executes a first kernel function of a function string. A second computing circuit of the processing device executes a second kernel function of the function string. The first computing circuit calculates dependent data before completing execution of the first kernel function. The second computing circuit receives the dependent data according to a thread group coordination, and starts to execute the second kernel function according to the dependent data before the first computing circuit completes execution of the first kernel function. Therefore, the second computing circuit does not need to wait until the first kernel function is completed before executing the second kernel function. In this way, the present application can greatly shorten the length of time required for the processing device to execute a single function string.
[0065] Finally, it should be noted that the above-described embodiments are merely used to illustrate the technical solutions of the present application, rather than limit the present application; although the present application has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can make modifications to the technical solutions recorded in the above embodiments, or replace equivalent replacements for part or all of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.
Claims
1. A processing device, characterized in that: The processing device comprises: a plurality of computing circuits; and a scheduling circuit coupled to the plurality of computing circuits and configured to receive a function string, obtain a first kernel function, a second kernel function, and at least one thread group coordination of the first kernel function from the function string, assign a first computing circuit among the plurality of computing circuits to execute the first kernel function, and assign a second computing circuit among the plurality of computing circuits to execute the second kernel function; The first calculation circuit calculates at least one dependent data before completing the execution of the first kernel function, and The second computing circuit cooperatively receives the at least one dependency data according to the at least one thread group, and starts executing the second kernel function before the first computing circuit completes executing the first kernel function according to the at least one dependency data.
2. The processing device according to claim 1, characterized in that The processing device further includes: a memory circuit coupled to the plurality of computing circuits, The first computing circuit calculates the at least one dependency data in cooperation with the at least one thread group before completing the execution of the first kernel function, and stores the at least one dependency data in the memory circuit.
3. The processing device according to claim 2, characterized in that The second computing circuit receives the at least one dependency data from the memory circuit before completing execution of the first kernel function in cooperation with the at least one thread group.
4. The processing device according to claim 3, characterized in that The second calculation circuit determines in a polling manner whether the memory circuit stores the at least one dependency data corresponding to the at least one thread group collaboration.
5. The processing device according to claim 1, characterized in that: The first kernel function includes a first thread group and a second thread group, The at least one thread group collaboration includes a first thread group collaboration, The first computing circuit generates first dependency data of the at least one dependency data after completing the first thread group and starts executing the second thread group, and The second computing circuit cooperatively receives the first dependency data according to the first thread group.
6. The processing device according to claim 5, characterized in that: The second kernel function includes a third thread group, The second computing circuit executes the third thread group in cooperation with the first thread group, and An execution period of the third thread group partially overlaps with an execution period of the second thread group.
7. A parallel processing method for a processing device, characterized in that: The processing device includes a plurality of computing circuits, wherein the parallel processing method includes: Receive a function string, and obtain a first kernel function, a second kernel function, and at least one thread group collaboration of the first kernel function from the function string; Assigning a first computing circuit among the plurality of computing circuits to execute the first kernel function, and assigning a second computing circuit among the plurality of computing circuits to execute the second kernel function; The first computing circuit calculates at least one dependent data before completing execution of the first kernel function; and The second computing circuit cooperatively receives the at least one dependency data according to the at least one thread group, and starts executing the second kernel function before the first computing circuit completes executing the first kernel function according to the at least one dependency data.
8. The parallel processing method according to claim 7, wherein: The step of calculating the at least one dependent data by the first calculation circuit before completing the execution of the first kernel function includes: The first computing circuit calculates the at least one dependency data in cooperation with the at least one thread group before completing the execution of the first kernel function, and stores the at least one dependency data in the memory circuit.
9. The parallel processing method according to claim 8, wherein: The step of cooperatively receiving the at least one dependent data by the second computing circuit according to the at least one thread group includes: The second computing circuit receives the at least one dependency data from the memory circuit before completing execution of the first kernel function in cooperation with the at least one thread group.
10. The parallel processing method according to claim 9, wherein: The step of cooperatively receiving the at least one dependent data by the second computing circuit according to the at least one thread group further includes: The second calculation circuit determines in a polling manner whether the memory circuit stores the at least one dependency data corresponding to the at least one thread group collaboration.
11. The parallel processing method according to claim 7, wherein: The first kernel function includes a first thread group and a second thread group, the at least one thread group collaboration includes the first thread group collaboration, and the step of calculating the at least one dependent data by the first computing circuit before completing execution of the first kernel function includes: The first computing circuit generates first dependency data of the at least one dependency data after completing the first thread group and starts executing the second thread group.
12. The parallel processing method according to claim 11, wherein: The step of cooperatively receiving the at least one dependency data by the second computing circuit according to the at least one thread group, and starting to execute the second kernel function before the first computing circuit completes executing the first kernel function according to the at least one dependency data further includes: The second computing circuit cooperatively receives the first dependency data according to the first thread group; and The second computing circuit cooperatively executes a third thread group of the second kernel function based on the first thread group, wherein an execution period of the third thread group partially overlaps with an execution period of the second thread group.
Citation Information
Patent Citations
Model iteration implementation method and device based on acceleration chip
CN112101565A
Artificial intelligence chip and data synchronization method thereof
CN117850705A