Operator parallel processing method, electronic equipment, storage medium and program product

Abstract

The invention relates to the technical field of artificial intelligence, and provides an operator parallel processing method, electronic equipment, a storage medium and a program product, and the method comprises the steps: determining a step of data dependence based on each processing step of a first operator and a second operator; and performing parallel execution on each processing step of the first operator and the second operator by applying a synchronization strategy and a segmentation strategy based on the plurality of cooperative thread bundles. According to the method, the parallel fusion processing of operators is realized through the steps of determining data dependence, segmenting the data, realizing parallel execution by utilizing the cooperative thread bundle, inserting synchronization between the data dependence steps and the like, the calculation performance is remarkably improved, and meanwhile, the data blocks after segmentation are multiplexed, so that the data processing efficiency is improved. The data dependence in the operators and between the operators can be reduced, and the overhead of data reading, instruction synchronization and the like can be reduced.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to an operator parallel processing method, electronic equipment, storage medium and program product. Background Art

[0002] In neural network models, operator fusion processing is often used to reduce the overhead of intermediate data reading and writing, thereby accelerating the model. The core idea of ​​operator fusion processing is that the latter operator directly reads the calculation result of the previous operator from the register, without having to write the calculation result of the previous operator to the external memory first, and then the latter operator reads the data from the external memory, thereby greatly reducing the overhead of data reading and writing.

[0003] However, current operator fusion strategies are often implemented in a serial execution mode, that is, in the same thread, the next operator needs to wait for the previous operator to complete the reading and calculation of data before it can perform its own calculation and write operations. Since the previous operator has a large reading and calculation overhead, the waiting time of the next operator is long, which affects the performance of the operator; and the calculation and reading and writing process of the next operator often takes a certain amount of time. Since the serial execution mode cannot effectively delay and hide the calculation and reading and writing process of the next operator, this will cause the reading and writing overhead of the next operator to be particularly obvious when processing large-scale data, resulting in poor performance of the operator and model, affecting the overall computing efficiency. Summary of the invention

[0004] The present invention provides an operator parallel processing method, an electronic device, a storage medium and a program product, which are used to solve the defect that the serial fusion strategy of operators in the related art has large reading and writing overhead when processing large-scale data, which affects the performance of operators and models.

[0005] The present invention provides an operator parallel processing method, comprising: Based on each processing step of the first operator and each processing step of the second operator, determining a step where data dependency exists, the output of the first operator being the input of the second operator; Based on the multiple cooperative thread warps, a synchronization strategy and a segmentation strategy are applied to execute each processing step of the first operator and each processing step of the second operator in parallel; The multiple cooperative thread warps share register resources, the synchronization strategy refers to inserting synchronization operations between the steps with data dependencies, and the segmentation strategy refers to segmenting the input data of the first operator and the input data of the second operator.

[0006] According to an operator parallel processing method provided by the present invention, the output of the first operator is transferred to the second operator in the form of a shared register block.

[0007] According to an operator parallel processing method provided by the present invention, each processing step of the first operator includes a split cache step, a data loading step, a first calculation step and a result storage step; The segmentation and caching step is used to segment the input data of the first operator and store the segmented data blocks in the cache in sequence; The data loading step is used to segment each data block in the cache, load the segmented multiple data slices into the input buffer in sequence, and read and reuse the data slices in the input buffer; The first calculation step is used to perform calculation based on the data slices read from the input buffer to obtain result data; The result storing step is used to write the result data into a shared register block.

[0008] According to an operator parallel processing method provided by the present invention, each processing step of the second operator includes a data reading step, a second calculation step and a result writing step; The data reading step is used to read the result data from the shared register block, and to slice the result data based on the memory arrangement of the second operator to obtain a plurality of result slices; The second calculation step is used to calculate the multiple result slices to obtain calculation results; The result writing step is used to write the calculation result to a specified storage location.

[0009] According to an operator parallel processing method provided by the present invention, there is data dependency between the splitting and caching step, the data loading step and the first calculation step of the first operator, and there is data dependency between the result storage step of the first operator and the data reading step of the second operator.

[0010] An operator parallel processing method provided by the present invention also includes: During execution, the usage of the shared register block is monitored; When it is detected that the usage of the shared register block exceeds the hardware limit, an automatic tuning method is adopted to determine the optimal shared register block size and segmentation strategy.

[0011] According to an operator parallel processing method provided by the present invention, the automatic tuning method is used to determine the optimal shared register block size and segmentation strategy, including: Determine multiple candidate register block sizes and multiple candidate segmentation strategies based on hardware information and shapes of operator data; Determining a plurality of candidate configurations based on the plurality of candidate register block sizes and the plurality of candidate segmentation strategies; The performance of each candidate configuration is evaluated, and an optimal configuration is determined according to the performance evaluation result of each candidate configuration, wherein the optimal configuration includes an optimal shared register block size and a segmentation strategy.

[0012] The present invention also provides an operator parallel processing device, comprising: a step determination unit, configured to determine a step having data dependency based on each processing step of the first operator and each processing step of the second operator, wherein the output of the first operator is the input of the second operator; A parallel execution unit is used to apply a synchronization strategy and a splitting strategy based on multiple collaborative thread bundles to execute each processing step of the first operator and each processing step of the second operator in parallel; wherein the multiple collaborative thread bundles share register resources, the synchronization strategy refers to inserting a synchronization operation between the steps with data dependencies, and the splitting strategy refers to splitting the input data of the first operator and the input data of the second operator.

[0013] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, wherein when the processor executes the computer program, any of the above-mentioned operator parallel processing methods is implemented.

[0014] The present invention also provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the operator parallel processing method described in any one of the above is implemented.

[0015] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements any of the above-mentioned operator parallel processing methods.

[0016] The operator parallel processing method, electronic device, storage medium and program product provided by the present invention can lay a solid foundation for subsequent parallel execution by determining the steps with data dependency based on the processing steps of the first operator and the processing steps of the second operator. By adopting multiple cooperative thread bundles and applying synchronization strategy and segmentation strategy to execute the processing steps of the first operator and the second operator in parallel, the total time of operator processing can be significantly reduced and the overall calculation speed can be improved. In the parallel execution process, by adopting the segmentation strategy to segment the input data of the operator, the large-scale data that originally needs to be processed serially can be segmented into multiple small blocks, and multiple cooperative thread bundles can process different small blocks at the same time, and while a cooperative thread bundle executes an operator step for a small block of data, other cooperative thread bundles can execute other steps of the operator in parallel for other small blocks, thereby realizing delay hiding, that is, reducing the time waiting for a certain step to be completed, thereby shortening the total processing time. At the same time, by adopting the synchronization strategy to insert synchronization operations between the steps with data dependency, it can ensure that data is correctly transmitted between dependent steps, avoiding data competition and inconsistency problems.

[0017] In addition, the present invention uses multiple collaborative thread warps to execute the various processing steps of the first operator and the second operator in parallel, and the collaborative thread warps can share register resources, so that the output of the first operator can be passed to the second operator through the shared register block, thereby realizing the parallel fusion of operators. Compared with the traditional serial fusion method, parallel fusion can significantly improve the performance of operators and models. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the present invention or related technologies, the following briefly introduces the drawings required for use in the embodiments or related technical descriptions. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0019] Figure 1 It is a flowchart of the operator parallel processing method provided by the present invention; Figure 2 It is a schematic diagram of the assembly line arrangement provided by the present invention; Figure 3 It is a structural schematic diagram of the operator parallel processing device provided by the present invention; Figure 4 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0020] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0021] In the calculation process of the neural network model, different operators have different resource consumption patterns according to their characteristics. Specifically, tcore (Tensor Core) operators, such as mma (Matrix Multiply-Accumulate) and conv (convolution), etc., require large-scale numerical calculations, so the amount of calculation of such operators is relatively large. Vcore (Vector Core) operators, such as add (addition) and relu (activation function), etc., are more involved in data reading and storage operations, so the memory access overhead of such operators is relatively large. It should be understood that tcore operators and vcore operators refer to operators designed to accelerate specific types of computing tasks. Among them, tcore operators focus on providing high-performance matrix multiplication and accumulation calculations, while vcore operators focus on optimizing vector computing tasks. The two are mainly used to improve the efficiency of artificial intelligence chips such as GPUs (Graphics Processing Units) in high-performance computing scenarios such as deep learning and scientific computing.

[0022] In order to improve the execution efficiency of the model, an optimization strategy usually adopted is to fuse the tcore operator and the vcore operator. The tcore operator is used as the previous operator and the vcore operator is used as the next operator. The core idea of ​​this fusion processing is to use the result calculated by the tcore operator directly as the input of the vcore operator, that is, the calculation result of the tcore operator is written into the register, and the vcore operator directly reads the calculation result from the register, thereby avoiding the tedious process of writing the intermediate data (i.e. the calculation result of the tcore operator) from the tcore to the external memory and then reading it from the external memory by the vcore. By reading the calculation result of the tcore operator directly from the register, the vcore operator can significantly reduce the read and write overhead.

[0023] Although the fusion processing of tcore operator and vcore operator has shown certain advantages in the optimization of neural network models, the fusion strategy in related technologies still has some obvious defects. These defects are particularly prominent when processing large-scale data (i.e. data with large shapes).

[0024] First, previous operator fusion often adopts a serial execution mode, that is, under the same thread, the vcore operator needs to wait for the tcore operator to read and calculate before it can perform its own calculation and write operations. Due to the high reading and calculation overhead of the tcore operator, the waiting time of the vcore operator will be long, affecting the overall performance of the operator; at the same time, since the calculation and reading and writing of the vcore operator take a certain amount of time, especially when processing large-scale data, that is, data with a larger shape, the reading and writing overhead of the vcore operator is particularly obvious. In the serial execution mode, the various steps of the operator are executed in sequence, and delay hiding cannot be achieved, which leads to poor performance of the operator and model when processing large-scale data.

[0025] Secondly, when processing large-shape data, the read and write overhead of the tcore operator may be greater than the computational overhead. Therefore, when optimizing the model, some large-shape data input to the tcore operator is usually divided into multiple small blocks for block processing. At the same time, the read and write overhead of the vcore operator is also relatively large. In order to achieve acceleration, the writing of the vcore operator is usually implemented using matrix loading instructions. One instruction corresponds to writing a set of data, that is, acceleration is achieved through SIMD (Single Instruction Multiple Data, one instruction corresponds to executing multiple data). Therefore, when processing large-shape data, the vcore operator will also split the data so that one data block can be written out through one instruction each time, and for different data layouts, vcore can have different optimal splitting methods. However, since the fusion of the tcore operator and the vcore operator is executed serially under the same thread, the data splitting method of the vcore operator is often determined by the splitting method of the tcore operator, which may not be the best splitting method for vcore.

[0026] In addition, the data transfer between the tcore operator and the vcore operator is often the value of a single register or the result of a single instruction execution, which leads to low efficiency of data transfer between operators and cannot achieve a good acceleration effect.

[0027] In this regard, the present invention provides an operator parallel processing method, which implements the parallel execution of each step of the first operator and the second operator based on multiple cooperative thread bundles and applies synchronization strategy and segmentation strategy. Compared with serial execution, parallel execution can achieve delay hiding and reduce the waiting time between steps, thereby improving processing efficiency. During the execution process, the data between the first operator and the second operator is transferred in a shared register block instead of the calculation result of a single instruction. Moreover, since each step of the first operator and the second operator is executed in parallel on different cooperative thread bundles, the second operator can determine the optimal segmentation strategy according to the data arrangement, thereby overcoming the above defects.

[0028] It should be noted that the operator parallel processing method provided by the present invention can realize the parallel fusion processing of multiple operators. The following mainly takes the parallel fusion processing of the first operator and the second operator as an example to introduce the technical solution of the present invention. Here, in the field of deep learning and high-performance computing, an operator generally refers to a function or process that performs a specific mathematical operation or logical operation. The first operator and the second operator refer to two different mathematical operations or logical operations. For example, the first operator can be an MMA operator in the Tcore operator type, and the second operator can be a ReLU operator in the Vcore operator type. The present invention does not specifically limit this.

[0029] Figure 1 is a flow chart of the operator parallel processing method provided by the present invention, such as Figure 1 As shown, the method includes: Step 110, based on each processing step of the first operator and each processing step of the second operator, determine the steps having data dependency, where the output of the first operator is the input of the second operator.

[0030] Specifically, each processing step of an operator refers to a series of calculations or operations required to execute the operator. For example, when the first operator is an MMA operator, each processing step of the first operator may include steps such as data segmentation, data loading, data calculation, and result storage; when the second operator is a ReLU operator, each processing step of the second operator may include steps such as data reading, data segmentation, data calculation, and result writing. By analyzing the data flow and dependency between each processing step of the first operator and each processing step of the second operator, steps with data dependency can be identified. Here, steps with data dependency refer to the situation in which the output data of one step is used as input data by another step during the calculation process, and this use requires the latter to wait for the output data of the former before calculation.

[0031] It can be understood that the output of the first operator is the input of the second operator, which means that the result (i.e., output data) generated after the calculation of the first operator is completed is directly used as the input data of the second operator, that is, the second operator must wait for the corresponding output data of the first operator to be ready before reading the data. Therefore, there is data dependency between the result storage step of the first operator and the data reading step of the second operator.

[0032] In addition, by analyzing the data flow and dependency between each processing step of the first operator and each processing step of the second operator, not only the steps with data dependency can be identified, but also the steps without data dependency can be identified. For example, there is no data dependency between the data segmentation and data loading steps of the first operator and the data calculation and result writing steps of the second operator. Therefore, these steps can be executed concurrently, thereby achieving the purpose of delay hiding.

[0033] It should be understood that in order to identify the steps with and without data dependency, the analysis can be implemented by automated means or by manual analysis, and the embodiments of the present invention do not specifically limit this. For example, when analyzing the data dependency between the processing steps of the first operator and the second operator by automated means, a judgment can be made based on the overlap of data indexes between the processing steps. Specifically, for the first operator and the second operator, the index range of the data blocks processed by each processing step can be determined first, and then the index range of the output data block of any processing step can be compared with the index range of the input data block of the other steps. If the two index ranges do not have an intersection, there is no data dependency between the corresponding two processing steps; if there is an intersection, there is data dependency between the corresponding two processing steps.

[0034] Step 120, based on multiple cooperative thread warps, applying a synchronization strategy and a segmentation strategy to execute each processing step of the first operator and each processing step of the second operator in parallel; The multiple cooperative thread warps share register resources, the synchronization strategy refers to inserting synchronization operations between the steps with data dependencies, and the segmentation strategy refers to segmenting the input data of the first operator and the input data of the second operator.

[0035] It should be noted that a cooperative warp (Cwarp for short) is a basic unit for parallel computing in the GPU. It consists of a group of threads that can cooperate with each other during execution, share certain resources (such as registers), and execute the same instructions. Different Cwarps can also share register resources, that is, multiple Cwarps can access and use the same register space to store and read data. Since register access speed is faster than main memory, sharing register resources can significantly reduce data access latency and improve computing efficiency.

[0036] Different Cwarps can execute different program codes in parallel at the same time. That is, when a Cwarp waits for an operation to complete, other Cwarps can continue to execute. This can hide the delay of a single Cwarp and achieve the effect of delay hiding. This mechanism helps to improve the overall computing efficiency of the GPU.

[0037] Specifically, after identifying the steps with data dependency, the processing steps of the first operator and the second operator can be executed in parallel based on multiple Cwarps and by applying synchronization strategy and segmentation strategy. In other words, each processing step of the first operator and each processing step of the second operator are each executed using a different Cwarp, thereby achieving parallel execution of each processing step.

[0038] In the parallel execution process, for steps with data dependencies, a synchronization strategy can be used to insert corresponding synchronization operations between these steps to ensure the production and consumption relationship between these steps, that is, to ensure that the data flow is correctly transmitted between these steps. Here, the synchronization strategy refers to a strategy for inserting synchronization operations between steps with data dependencies in order to maintain data consistency and correctness in parallel computing. Since each processing step is executed on a different Cwarp, the synchronization strategy can also be understood as inserting synchronization operations between different Cwarps.

[0039] Inserting a synchronization operation means forcing the related Cwarp to suspend execution at a certain stage of parallel computing until the Cwarp reaches a certain synchronization point. For example, the input data of the second operator depends on the output data of the first operator. When the first operator has not yet prepared the corresponding output data, a synchronization operation can be inserted between the result storage step of the first operator and the data reading step of the second operator to block the data reading operation of the second operator, so that the second operator waits until the first operator prepares the output data before reading the corresponding data and executing the next step. It should be understood that inserting a synchronization operation can be implemented through some synchronization instructions on hardware (such as GPU), and synchronization instructions can be inserted between steps with data dependence (i.e., different Cwarps that execute data-dependent steps).

[0040] In the embodiment of the present invention, after identifying data dependency, by inserting synchronization operations in a timely manner during the execution process, it is possible to ensure that data is correctly transferred between dependent steps. This synchronization mechanism is the key to maintaining data consistency in parallel processing. Although synchronization will increase some overhead, by precisely controlling the synchronization point, that is, inserting synchronization operations only between steps with data dependency, rather than between all steps, it is possible to minimize unnecessary synchronization waiting time, thereby minimizing synchronization overhead.

[0041] Furthermore, during the execution process, a splitting strategy can be used to split the input data of the first operator and the second operator, thereby decomposing a large data task that originally needs to be processed serially into multiple small tasks that can be processed in parallel. In this way, multiple Cwarps can process different data blocks at the same time, and there is no data dependency between blocks, thereby significantly improving the efficiency of parallel processing, while reducing the data dependency between the processing steps within the operator and between operators. Moreover, the splitting strategy allows data to be more evenly distributed among multiple Cwarps, avoiding the overload of a single thread bundle. Here, the splitting strategy refers to a strategy in parallel computing that divides a computing task or data set into multiple smaller parts in order to make full use of the resources of multiple cooperative thread bundles, and then assigns them to different Cwarps for parallel processing.

[0042] Exemplarily, for the first operator, assuming that the processing steps of the operator include a data segmentation step, a data loading step, a data calculation step, and a result storage step, these steps can be executed in parallel using different Cwarps, such as using Cwarp0 to execute the data segmentation step, using Cwarp1 to execute the data loading step, using Cwarp2 to execute the data calculation step, and using Cwarp3 to execute the result storage step. During the execution process, the input data of the first operator can be segmented by Cwarp0, and the segmented data blocks can be assigned to other Cwarps for parallel processing. For example, the first data block can be loaded by Cwarp1 first, and when the first data block is loaded, Cwarp2 calculates the loaded first data block, and at the same time, Cwarp1 can load the second data block. By analogy, through the parallel execution of these Cwarps, the processes of data loading, data calculation, and result storage can be mutually concealed, thereby reducing the overall delay and improving the operator performance. During the execution of each Cwarp, synchronization operations can be inserted between Cwarps with data dependencies to ensure that data is correctly transmitted. It should be understood that the parallel execution process of each processing step of the second operator can refer to the parallel execution process of each processing step of the first operator, which will not be described in detail here.

[0043] The method provided by the embodiment of the present invention can lay a solid foundation for subsequent parallel execution by determining the steps with data dependency based on the processing steps of the first operator and the processing steps of the second operator. By adopting multiple cooperative thread bundles and applying synchronization strategies and segmentation strategies to execute the processing steps of the first operator and the second operator in parallel, the total time of operator processing can be significantly reduced and the overall calculation speed can be improved. In the parallel execution process, by adopting the segmentation strategy to segment the input data of the operator, the large-scale data that originally needs to be processed serially can be segmented into multiple small blocks, and multiple cooperative thread bundles can process different small blocks at the same time. Moreover, when a cooperative thread bundle executes an operator step for a small block of data, other cooperative thread bundles can execute other steps of the operator in parallel for other small blocks, thereby realizing delay hiding, that is, reducing the time waiting for a step to be completed, thereby shortening the total processing time. At the same time, by adopting a synchronization strategy to insert synchronization operations between steps with data dependency, it can ensure that data is correctly transferred between dependent steps, avoiding data competition and inconsistency problems.

[0044] In addition, the embodiment of the present invention uses multiple collaborative thread warps to execute the various processing steps of the first operator and the second operator in parallel, and the collaborative thread warps can share register resources, so that the output of the first operator can be passed to the second operator through the shared register block, thereby realizing the parallel fusion of operators. Compared with the traditional serial fusion method, parallel fusion can significantly improve the performance of operators and models.

[0045] Based on the above embodiment, the output of the first operator is transferred to the second operator in the form of a shared register block.

[0046] It should be noted that, considering that the data transfer between operators in the related art is often the value of a single register or the result of a single instruction execution, this leads to low efficiency of data transfer between operators. In order to improve the efficiency of data transfer between operators, in the embodiment of the present invention, data transfer between operators is implemented in the form of a shared register block, thereby improving the acceleration effect.

[0047] Specifically, a shared register block is a special memory area that is located inside a processor or computing unit and is designed as small blocks for storing and transferring data. These small blocks usually have a fixed size and shape to suit specific computing needs. Shared register blocks can be accessed by multiple Cwarps simultaneously without the need for complex synchronization mechanisms.

[0048] Specifically, in order to improve the efficiency of data transfer between operators, the output data of the first operator is passed to the second operator in the form of a shared register block, which means that when the first operator (such as the mma operator) completes its calculation task for a data block, the data block it outputs is not directly stored in the ordinary memory or cache, but is organized and stored in a shared register block (i.e., a register shared between multiple Cwarps), so that the second operator (such as the relu operator) can efficiently read the data block output by the first operator from the shared register block and use it as its own input data to perform the next operation. Here, the shared register block can be understood as a data organization structure when transferring data through a shared register, which can be two-dimensional (such as a part of a matrix) or other shapes to adapt to specific computing needs.

[0049] In the embodiment of the present invention, data transfer between the first operator and the second operator is implemented in the form of a shared register block, rather than through the calculation result of a single instruction, which can greatly improve the data transfer efficiency between operators, thereby further improving the calculation efficiency.

[0050] Based on any of the above embodiments, each processing step of the first operator includes a data segmentation and caching step, a data loading step, a first calculation step, and a result storage step; The segmentation and caching step is used to segment the input data of the first operator and store the segmented data blocks in the cache in sequence; The data loading step is used to segment each data block in the cache, load the segmented multiple data slices into the input buffer in sequence, and read and reuse the data slices in the input buffer; The first calculation step is used to perform calculation based on the data slices read from the input buffer to obtain result data; The result storing step is used to write the result data into a shared register block.

[0051] Specifically, in the splitting and caching step, the main purpose of splitting the input data of the first operator is to split it into smaller and more manageable parts for subsequent processing and calculation, and there is no data dependency between the split data blocks, so that parallel processing can be better performed. Here, when the input data is split, it can be performed according to the dimension, size and hardware characteristics of the data. Specifically, when the input data of the first operator is split, it can be determined on which dimensions to split according to the characteristics of the data (such as matrix) and the requirements of the algorithm and the hardware characteristics. For example, for matrix data, it can be selected to split by row or column, or it can be selected to split by row and column at the same time. Then, the split size (that is, the size of each data block) can be calculated according to the available resources on the hardware (such as memory, computing power) and the requirements of the algorithm efficiency. Finally, according to the determined splitting dimensions and sizes, the input data is split into multiple data blocks. Here, multiple data blocks refer to small parts of data obtained after the splitting operation for subsequent processing. These data blocks will be used as inputs for subsequent steps.

[0052] In the split cache step, the multiple data blocks obtained after the split will be read and written to the cache in sequence. Here, the cache refers to an efficient storage area for temporarily storing data, for example, it can be a multi-level cache. The multi-level cache is a hierarchical storage structure designed to improve the speed and efficiency of data access. For example, the multi-level cache may include global memory, shared memory, L1 cache, L2 cache, etc., wherein the L1 cache is the cache closest to the computing core in the GPU, which is located between the core and the memory and is used to speed up data access; the L2 cache is an intermediate layer cache located between the L1 cache and the global memory in the GPU, which is used to store a larger amount of data; the global memory and the shared memory are located outside the processor and have a larger capacity. It should be understood that by splitting the input data of the first operator and reading the split data blocks into the multi-level cache, the computing efficiency of the GPU can be significantly improved.

[0053] In the data loading step, each data block on the cache can be further segmented so that the size of the segmented data slices can adapt to the size of the input buffer, so that the segmented data slices can be loaded into the input buffer, which is convenient for reading the data slices from the input buffer for subsequent calculation. Specifically, when the data blocks on the cache are segmented, the segmentation method of the data blocks can be determined according to the needs of subsequent calculations and the limitations of hardware resources (such as the size limit of the input buffer), which can include determining the dimension of the segmentation, the size of the segmentation, and the order of the segmentation. Subsequently, each data block in the cache is segmented according to the determined segmentation method, so that each data block is divided into smaller, easier to process parts, namely data slices. The segmented data slices can be numbered, sorted or organized so that they can be efficiently accessed and processed in subsequent calculation steps.

[0054] In the data loading step, after completing the segmentation of each data block, each data slice obtained by segmentation can be loaded into the input buffer in a certain order. The data loading process specifically involves reading the data slices from the cache and writing them to the specified position of the input buffer. Here, the input buffer refers to a special storage area located inside or near the processor, which is used to store data to be processed by the processor, and it usually has a faster access speed and a smaller capacity. For example, when the first operator is an mma operator, the input buffer may include A buffer and B buffer, wherein A buffer refers to a buffer for storing data slices of the left matrix, and B buffer refers to a buffer for storing data slices of the right matrix. During the execution of the mma operator, A and B buffers are used as input buffers to temporarily store data slices read from the cache, and these data slices will be sent to the next step for matrix multiplication calculation after being prepared.

[0055] It should be noted that the data slices stored in the input buffer can be reused without repeated reading, which greatly reduces the overhead of data reading. Here, data reuse means using the data that has been loaded into the input buffer as many times as possible when performing calculations to reduce the number of repeated loading and reading of data.

[0056] For example, when the first operator is the mma operator, taking the input matrices A[M, K] and B[K, N] as an example, assuming that the output matrix C[M, N] needs to be calculated, in the conventional data reading and calculation process, the dimensions are traversed in the order of M, N, K. For each output element C[i, j], it is necessary to traverse K times to accumulate the product of A[i, k] and B[k, j]. In this way, every element in A and every element in B will be read. times, resulting in a large data reading overhead. In order to reduce the data reading overhead, the order of the dimension loop can be rearranged so that the data elements in the input matrix A or B can be reused, thereby significantly reducing the number of data reads. For example, the k-dimensional loop can be moved to the outer layer. For each i (row index of A), only K needs to be traversed once to load A[i, k] into the buffer, and then in the inner j loop, for each k, B[k, j] is loaded and calculated. In this way, each element in A only needs to be read times, instead of This achieves the reuse of data elements in A, that is, the data slices after A is split are loaded into the buffer, and then mma calculations are performed on the k-dimensional, and the data slices in the buffer are repeatedly used, thereby greatly reducing the overhead of data reading. Similarly, the data elements in the input matrix B can also be reused.

[0057] It should be understood that when reading the data slice in the input buffer, the corresponding data can be obtained according to the address offset, and the address offset here can be determined according to the starting address of the data slice and the size of the data slice. Specifically, each data element has a fixed position in the buffer, and this position can be determined by a starting address plus an offset. By reading and reusing the data in the input buffer, it is possible to avoid repeatedly reading data from the global memory or cache, which helps to reduce the number of memory accesses.

[0058] Exemplarily, when the first operator is an MMA operator, during the MMA calculation process, due to the inherent characteristics of matrix operations (such as the repetitiveness and pattern of multiplication and addition operations), data in the A and B buffers can be efficiently reused through address offset access. For example, in MMA calculations, elements in the same row may be reused when multiplied with elements in different columns. By storing these elements in the A and B buffers and using address offsets to access them, the calculation efficiency can be significantly improved.

[0059] In the first calculation step, the calculation unit (i.e., a Cwarp) performs corresponding calculation operations on the data slices read from the input buffer. For example, when the first operator is an mma operator, a data slice can be read from each of the A and B buffers, and the two read data slices can be used to perform matrix multiplication calculations to obtain result data. Here, the result data refers to the output data obtained after calculating the data slices in the input buffer in the first calculation step. These data will be used for subsequent storage, transmission, or further calculations.

[0060] In the result storage step, the result data may be written into a shared register block (ie, a register shared among multiple Cwarps) so that the second operator may directly read the corresponding data from the shared register block and process it.

[0061] It can be understood that each of the above steps can be executed in parallel using a Cwarp. For steps with data dependencies, corresponding synchronization operations can be inserted between the Cwarps that execute these steps to ensure the production and consumption relationship between these steps, thereby ensuring that data is correctly transferred between dependent steps.

[0062] In an embodiment of the present invention, by dividing the input data of the first operator into multiple data slices and loading these data slices into the input buffer for reuse, not only the number of data accesses and conflicts can be reduced, but also the data dependency between the processing steps within the first operator can be reduced, as well as the data dependency between the first operator and the second operator can be reduced, thereby reducing the need for synchronization and further reducing the overhead of data reading and synchronization instructions.

[0063] Based on any of the above embodiments, each processing step of the second operator includes a data reading step, a second calculation step and a result writing step; The data reading step is used to read the result data from the shared register block, and to slice the result data based on the memory arrangement of the second operator to obtain a plurality of result slices; The second calculation step is used to calculate the multiple result slices to obtain calculation results; The result writing step is used to write the calculation result to a specified storage location.

[0064] Specifically, for the second operator, the processing steps of the second operator may include a data reading step, a second calculation step, and a result writing step. Each step can also be executed in parallel using a Cwarp. For steps in which data dependencies exist, corresponding synchronization operations can be inserted between the Cwarps that execute these steps to ensure the production and consumption relationship between these steps, thereby ensuring that data is correctly transferred between dependent steps.

[0065] In the data reading step, Cwarp can read the result data output by the first operator from the shared register block, and during the data reading process, the result data can be segmented according to the memory arrangement of the second operator to ensure that the data can be processed by the second operator in the most efficient way. Here, the memory arrangement of the second operator refers to the memory layout or memory access mode followed by the second operator (such as the relu operator) when processing data, which specifically involves the storage order of data in memory, the size of the data block, the dimensional arrangement of the data (such as row priority or column priority), etc.

[0066] It is understandable that in the process of reading the result data, the data is split according to the memory layout, which means that before the second operator calculates the result data of the first operator, it will reorganize the data according to the memory layout inside the second operator. For example, if the second operator expects to access the data in a row-first manner, then when splitting the data, it will ensure that each slice corresponds to a row (or part of a row) of the original data. Since different operators may have different memory access modes and data processing requirements, splitting the data by matching the memory layout of the second operator can reduce unnecessary memory access overhead and optimize the computing performance of the second operator.

[0067] In the second calculation step, the multiple result slices obtained by segmentation can be calculated to obtain the calculation results. Here, the result slice refers to the data subset segmented from the result data of the first operator, and these slices are determined according to the memory arrangement of the second operator so as to be efficiently processed in the second operator. Each result slice can contain a certain number of data elements. It should be understood that when calculating the result slice, each slice can be traversed and specific calculation logic (such as the nonlinear transformation of the relu function) can be applied to each data element in the slice.

[0068] In the result writing step, the calculation results can be written to the specified storage location. Here, the specified storage location refers to the memory address or data structure where the calculation results are stored after the second operator is calculated. This storage location is usually determined before the program is executed. It can correspond to a global memory area, a shared memory area, or a specific data structure (such as an array, vector, or matrix). The calculation results are written to the specified storage location to ensure that these data can be accessed and used by subsequent calculation steps or other parts of the program.

[0069] Based on any of the above embodiments, there is data dependency between the splitting and caching step, the data loading step and the first calculation step of the first operator, and there is data dependency between the result storage step of the first operator and the data reading step of the second operator.

[0070] Specifically, each processing step of the first operator and each processing step of the second operator can be executed in parallel using a Cwarp. For steps with data dependencies, corresponding synchronization operations can be inserted to ensure the correct production and consumption relationship; for most steps without data dependencies, they are allowed to be executed in parallel to achieve the purpose of latency hiding.

[0071] Specifically, in the processing stage of the first operator, there is data dependency between the cache splitting step and the data loading step and the first calculation step, that is, there is dependency between the writing of the cache and the reading of the input buffer data, and there is dependency between the reading of the input buffer data and the data calculation. Therefore, it is necessary to insert corresponding synchronization operations between these dependent steps to ensure the correct production and consumption relationship between these steps.

[0072] Similarly, since the output data of the first operator is the input data of the second operator, that is, the input of the second operator depends on the output of the first operator, there is a data dependency between the result storage step of the first operator and the data reading step of the second operator. In the parallel execution process, corresponding synchronization operations need to be inserted between these two steps to ensure the correct transmission of data.

[0073] For most steps without data dependencies, such as the split cache step of the first operator and the calculation step of the second operator, the data loading step of the first operator and the calculation step of the second operator, the data loading step of the first operator and the result writing step of the second operator, etc., there is no data dependency between these steps and they can be directly assigned to different Cwarps for parallel execution to achieve delay hiding.

[0074] Figure 2 It is a schematic diagram of the pipeline arrangement provided by the present invention, such as Figure 2 As shown, the pipeline ping-pong operation, also known as the ping-pong operation, is a processing technology for data flow control. This operation introduces two or more buffers, so that different functional blocks can take turns to read and write to the buffers, thereby achieving simultaneous operation. Exemplarily, taking the processing stage of the first operator as an example, when Cwarp1 completes the loading of the ping data (i.e., a certain data slice), Cwarp2 can apply the data slice for calculation. At the same time, Cwarp1 can start loading the pong data (i.e., another data slice), that is, start reading the pong data from the input buffer. When Cwarp2 completes the calculation of the ping data, Cwarp3 starts to write the calculation results to the shared register block. At the same time, Cwarp2 can apply the loaded pong data for calculation. And so on, until the calculation of all data slices is completed.

[0075] It can be understood that by adopting a data segmentation strategy and multiple collaborative thread bundles for parallel execution, multiple task pipelines are introduced, so that the read, write and calculation steps of the first operator can be better executed in parallel with the read, write and calculation steps of the second operator, thereby hiding delays and improving operator performance.

[0076] Based on any of the above embodiments, the method further includes: Step 130, during the execution, monitoring the usage of the shared register block; Step 140 : when it is detected that the usage of the shared register block exceeds the hardware limit, an automatic tuning method is used to determine the optimal shared register block size and segmentation strategy.

[0077] Specifically, during the execution process, the usage of the shared register block can be monitored in real time. When the usage of the shared register block exceeds the maximum value (i.e., the hardware limit) of the GPU hardware, it will cause register overflow and affect the performance of the operator. Here, the usage of the shared register block can be tracked in real time through the performance counters provided by the hardware or the monitoring tools at the software level.

[0078] When it is detected that the usage of the shared register block exceeds the hardware limit, an automatic tuning method can be used to select the optimal shared register block size and segmentation strategy. Here, in GPU parallel computing, data is usually divided into smaller blocks for processing, and the shared register block size refers to the size of these small blocks of data. The optimal shared register block size and segmentation strategy refers to the register configuration scheme that can maximize computing efficiency, minimize resource occupancy and conflict rate under a given hardware environment and computing task. It should be understood that the optimal shared register block size and segmentation strategy may depend on multiple factors such as the characteristics of the operator, the shape of the data, the limitations of the hardware, and the complexity of the computing task.

[0079] Automatic tuning is a method of finding the optimal configuration parameters through automated means. These parameters may include shared register block size and segmentation strategy. For example, intelligent optimization algorithms such as genetic algorithms and particle swarm optimization can be used to search for the optimal shared register block size and segmentation strategy in a preset parameter space. For another example, based on machine learning or deep learning models, the performance under different configurations can be predicted based on historical data or real-time monitoring information, so as to select the optimal configuration solution. Based on the determined optimal shared register block size and optimal segmentation strategy, the shared register block size and data segmentation method can be dynamically adjusted to adapt to different computing tasks and hardware environments.

[0080] Based on any of the above embodiments, step 140 specifically includes: Step 141 : determining a plurality of candidate register block sizes and a plurality of candidate segmentation strategies based on the hardware information and the shape of the operator data.

[0081] Specifically, hardware information refers to the relevant parameters and characteristics of the physical device (such as GPU) that performs computing tasks, including but not limited to the processor architecture, the number of registers, memory bandwidth, cache size, the number and type of computing units, etc. These parameters will limit the range of choices for the shared register block size and affect the effectiveness of the segmentation strategy.

[0082] The shape of operator data refers to the dimension and layout of the data processed by the operator (i.e. the basic operation or function in the computing task), which will affect the segmentation strategy. For example, in deep learning, an mma operator processes a two-dimensional matrix (including row and column dimensions), and a conv operator can process a four-dimensional tensor (batch size, number of channels, height, width).

[0083] By analyzing the hardware information and the shape of the operator data, a series of candidate register block sizes and splitting strategies can be generated. Here, the candidate register block size refers to the set of register block sizes that may be adopted under given hardware constraints and operator data shapes. These sizes are derived from the hardware information and data shape analysis, aiming to find a configuration that can balance computational efficiency and resource usage. The candidate splitting strategy refers to a set of methods for splitting data into blocks or slices of different sizes for parallel processing.

[0084] Step 142: Determine multiple candidate configurations based on the multiple candidate register block sizes and the multiple candidate segmentation strategies.

[0085] Specifically, by combining multiple candidate register block sizes and multiple candidate segmentation strategies in pairs, a series of possible configuration combinations can be generated, that is, multiple candidate configurations can be obtained. Here, the candidate configuration is a computing task configuration that includes a specific register block size and segmentation strategy. These configurations will be used in the subsequent performance evaluation and optimization process to find the optimal configuration solution.

[0086] Step 143 , evaluating the performance of each candidate configuration, and determining an optimal configuration according to the performance evaluation result of each candidate configuration, wherein the optimal configuration includes an optimal shared register block size and segmentation strategy.

[0087] Specifically, the performance of each candidate configuration can be evaluated through the following steps: According to the requirements of the computing task and the hardware characteristics, select appropriate evaluation indicators (such as execution time, throughput, resource utilization, etc.), which will be used to quantify the performance differences between different configurations. Then, perform benchmark tests on each candidate configuration using the same test data set under the same hardware and software environment. The benchmark test will simulate the typical execution scenario of the computing task and collect relevant performance data. The collected performance data is analyzed and compared to determine the advantages and disadvantages of each candidate configuration.

[0088] After obtaining the performance evaluation results of each candidate configuration, the candidate configurations can be sorted according to the evaluation indicators, and the configuration set with the best performance can be selected. For example, the candidate configuration with the shortest execution time and the highest throughput can be selected as the optimal configuration. Here, the optimal configuration refers to the computing task configuration that can maximize computing efficiency, minimize resource usage, and meet other related requirements under given hardware constraints, operator data shapes, and performance evaluation indicators.

[0089] In addition, after finding the optimal configuration, it can be stored so that when encountering the same hardware information and operator data shape in the future, the optimal configuration can be directly applied without repeating the tuning selection.

[0090] Based on any of the above embodiments, an embodiment of the present invention provides a method for fine-grained parallel fusion calculation of a tcore operator and a vcore operator, where the tcore operator takes an mma operator as an example, and the vcore operator takes a relu operator as an example, and the method includes: Step S1, mma operator processing stage; Step S11, dividing the input data of the mma operator, reading and writing the divided data blocks into a multi-level cache; Step S12, split the data block in the cache again, and read the split data slices into A and B buffers, preparing for matrix multiplication operation; Step S13, multiplexing the data slices in the A and B buffers, that is, there is no need to repeatedly read the same data slices, and the required data slices are directly obtained through address offset; Step S14, performing MMA calculation using the read data slice, and writing the result to the shared register block; Step S15, loop traversal of the intermediate dimension (ie, K dimension) is performed, and the result of each mma calculation is accumulated to the shared register block.

[0091] Step S2, relu operator processing stage; Step S21, read the result data of MMA calculation from the shared register block, and perform corresponding vcore segmentation on the read data according to the memory arrangement of the relu operator to meet the parallel requirements of the relu calculation; Step S22, performing relu calculation, that is, applying the relu activation function to each element; Step S23, write the result of the relu calculation to the specified storage location through the matrix storage instruction.

[0092] Step S3, Cwarp parallel execution stage; Each step in the mma operator processing stage and the relu operator processing stage (i.e., step S11 to step S15, step S21 to step S23) is assigned to a Cwarp for parallel execution. For steps with data dependencies, corresponding synchronization operations can be inserted to ensure the corresponding production and consumption relationship. Here, the steps with data dependencies mainly include the dependency between cache writing and A and B buffer data reading, the dependency between A and B buffer data reading and mma calculation, and the partial dependency of the shared register block (i.e., the dependency between the storage of mma calculation results and relu data reading). For most steps without data dependencies (for example, there is no dependency between mma data reading and relu calculation and writing), these steps can be executed in parallel to achieve the purpose of delay hiding.

[0093] In addition, for parts with data dependencies, a similar double buffer technology can be used to divide the data into multiple blocks for processing, so as to reduce the data dependencies between blocks and better achieve parallelization. Here, the double buffer technology can make the instruction flow more closely arranged by introducing multiple task pipelines, achieving a certain degree of concurrency. For details, please refer to Figure 2 The process shown.

[0094] Step S4, automatic tuning stage; During the execution process, the usage of the shared register block is monitored. When it is found that the register usage exceeds the hardware limit, automatic tuning is used to select the optimal shared register block size and segmentation strategy.

[0095] The method provided by the embodiment of the present invention has the following advantages: 1. By executing the processing steps of the tcore operator and the vcore operator in parallel, latency hiding can be achieved, greatly improving operator performance; 2. Compared with the traditional serial fusion method, the tcore operator and the vcore operator in the embodiment of the present invention can define different data segmentation methods to better exert the performance of tcore and vcore respectively; 3. It can control the pipeline and synchronization within the tcore operator, the vcore operator, and between the tcore operator and the vcore operator in a more fine-grained manner, so that the reading, writing and calculation of the tcore operator can be better parallelized with the reading, writing and calculation of the vcore operator to hide the delay; 4. For scenarios where the number of available registers is insufficient, automatic tuning can be used to select the optimal shared register block size and segmentation strategy.

[0096] The operator parallel processing device provided by the present invention is described below. The operator parallel processing device described below and the operator parallel processing method described above can be referred to each other.

[0097] Figure 3 Schematic diagram of the structure of the operator parallel processing device provided by the present invention. Figure 3 As shown, the device comprises: A step determination unit 310, configured to determine a step having data dependency based on each processing step of the first operator and each processing step of the second operator, wherein the output of the first operator is the input of the second operator; A parallel execution unit 320 is used to apply a synchronization strategy and a splitting strategy based on multiple collaborative thread bundles to execute each processing step of the first operator and each processing step of the second operator in parallel; wherein the multiple collaborative thread bundles share register resources, the synchronization strategy refers to inserting synchronization operations between the steps with data dependencies, and the splitting strategy refers to splitting the input data of the first operator and the input data of the second operator.

[0098] The device provided by the embodiment of the present invention can lay a solid foundation for subsequent parallel execution by determining the steps with data dependency based on the processing steps of the first operator and the processing steps of the second operator. By adopting multiple cooperative thread bundles and applying synchronization strategies and segmentation strategies to execute the processing steps of the first operator and the second operator in parallel, the total time of operator processing can be significantly reduced and the overall calculation speed can be improved. In the parallel execution process, by adopting the segmentation strategy to segment the input data of the operator, the large-scale data that originally needs to be processed serially can be segmented into multiple small blocks, and multiple cooperative thread bundles can process different small blocks at the same time. Moreover, while a cooperative thread bundle executes an operator step for a small block of data, other cooperative thread bundles can execute other steps of the operator in parallel for other small blocks, thereby realizing delay hiding, that is, reducing the time waiting for a certain step to complete, thereby shortening the total processing time. At the same time, by adopting a synchronization strategy to insert synchronization operations between steps with data dependency, it can ensure that data is correctly transferred between dependent steps, avoiding data competition and inconsistency problems.

[0099] In addition, the embodiment of the present invention uses multiple collaborative thread warps to execute the various processing steps of the first operator and the second operator in parallel, and the collaborative thread warps can share register resources, so that the output of the first operator can be passed to the second operator through the shared register block, thereby realizing the parallel fusion of operators. Compared with the traditional serial fusion method, parallel fusion can significantly improve the performance of operators and models.

[0100] Based on any of the foregoing embodiments, the output of the first operator is transmitted to the second operator in the form of a shared register block.

[0101] Based on any of the above embodiments, each processing step of the first operator includes a data segmentation and caching step, a data loading step, a first calculation step, and a result storage step; The segmentation and caching step is used to segment the input data of the first operator and store the segmented data blocks in the cache in sequence; The data loading step is used to segment each data block in the cache, load the segmented multiple data slices into the input buffer in sequence, and read and reuse the data slices in the input buffer; The first calculation step is used to perform calculation based on the data slices read from the input buffer to obtain result data; The result storing step is used to write the result data into a shared register block.

[0102] Based on any of the above embodiments, each processing step of the second operator includes a data reading step, a second calculation step and a result writing step; The data reading step is used to read the result data from the shared register block, and to slice the result data based on the memory arrangement of the second operator to obtain a plurality of result slices; The second calculation step is used to calculate the multiple result slices to obtain calculation results; The result writing step is used to write the calculation result to a specified storage location.

[0103] Based on any of the above embodiments, there is data dependency between the splitting and caching step, the data loading step and the first calculation step of the first operator, and there is data dependency between the result storage step of the first operator and the data reading step of the second operator.

[0104] Based on any of the above embodiments, the device further includes a tuning unit, and the tuning unit is used to: During execution, the usage of the shared register block is monitored; When it is detected that the usage of the shared register block exceeds the hardware limit, an automatic tuning method is adopted to determine the optimal shared register block size and segmentation strategy.

[0105] Based on any of the above embodiments, the tuning unit is specifically used for: Determine multiple candidate register block sizes and multiple candidate segmentation strategies based on hardware information and shapes of operator data; Determining a plurality of candidate configurations based on the plurality of candidate register block sizes and the plurality of candidate segmentation strategies; The performance of each candidate configuration is evaluated, and an optimal configuration is determined according to the performance evaluation result of each candidate configuration, wherein the optimal configuration includes an optimal shared register block size and a segmentation strategy.

[0106] Figure 4 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 4 As shown, the electronic device may include: a processor 410, a communication interface 420, a memory 430 and a communication bus 440, wherein the processor 410, the communication interface 420 and the memory 430 communicate with each other through the communication bus 440. The processor 410 may call the logic instructions in the memory 430 to execute the operator parallel processing method, the method comprising: based on each processing step of the first operator and each processing step of the second operator, determining the step with data dependency, the output of the first operator being the input of the second operator; based on multiple cooperative thread bundles, applying a synchronization strategy and a splitting strategy, executing each processing step of the first operator and each processing step of the second operator in parallel; wherein the register resources are shared between the multiple cooperative thread bundles, the synchronization strategy refers to inserting a synchronization operation between the steps with data dependency, and the splitting strategy refers to splitting the input data of the first operator and the input data of the second operator.

[0107] In addition, the logic instructions in the above-mentioned memory 430 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the relevant technology or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0108] On the other hand, the present invention also provides a computer program product, which includes a computer program, and the computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the operator parallel processing method provided by the above methods, and the method includes: based on each processing step of a first operator and each processing step of a second operator, determining the steps where data dependencies exist, and the output of the first operator is the input of the second operator; based on multiple collaborative thread warps, applying a synchronization strategy and a splitting strategy to execute each processing step of the first operator and each processing step of the second operator in parallel; wherein register resources are shared between the multiple collaborative thread warps, the synchronization strategy refers to inserting synchronization operations between the steps where data dependencies exist, and the splitting strategy refers to splitting the input data of the first operator and the input data of the second operator.

[0109] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the operator parallel processing method provided by the above-mentioned methods, the method comprising: based on each processing step of a first operator and each processing step of a second operator, determining the steps where data dependencies exist, the output of the first operator being the input of the second operator; based on multiple collaborative thread warps, applying a synchronization strategy and a splitting strategy to execute each processing step of the first operator and each processing step of the second operator in parallel; wherein register resources are shared between the multiple collaborative thread warps, the synchronization strategy refers to inserting synchronization operations between the steps where data dependencies exist, and the splitting strategy refers to splitting the input data of the first operator and the input data of the second operator.

[0110] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0111] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiment.

[0112] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An operator parallel processing method, characterized in that: include: Based on each processing step of the first operator and each processing step of the second operator, determining a step in which data dependency exists, the output of the first operator is the input of the second operator, and the output of the first operator is passed to the second operator in the form of a shared register block; Based on the multiple cooperative thread warps, a synchronization strategy and a segmentation strategy are applied to execute each processing step of the first operator and each processing step of the second operator in parallel; Among them, register resources are shared among the multiple collaborative thread warps, the synchronization strategy refers to inserting synchronization operations between the steps with data dependencies, and the splitting strategy refers to splitting the input data of the first operator and the input data of the second operator, and reusing the split data of the first operator.

2. The operator parallel processing method according to claim 1, characterized in that: The processing steps of the first operator include a splitting and caching step, a data loading step, a first calculation step, and a result storage step; The segmentation and caching step is used to segment the input data of the first operator and store the segmented data blocks in the cache in sequence; The data loading step is used to segment each data block in the cache, load the segmented multiple data slices into the input buffer in sequence, and read and reuse the data slices in the input buffer; The first calculation step is used to perform calculation based on the data slices read from the input buffer to obtain result data; The result storing step is used to write the result data into a shared register block.

3. The operator parallel processing method according to claim 2, characterized in that: Each processing step of the second operator includes a data reading step, a second calculation step and a result writing step; The data reading step is used to read the result data from the shared register block, and to slice the result data based on the memory arrangement of the second operator to obtain a plurality of result slices; The second calculation step is used to calculate the multiple result slices to obtain calculation results; The result writing step is used to write the calculation result to a specified storage location.

4. The operator parallel processing method according to claim 3, characterized in that: There is data dependency between the splitting and caching step, the data loading step and the first calculation step of the first operator, and there is data dependency between the result storage step of the first operator and the data reading step of the second operator.

5. The operator parallel processing method according to any one of claims 1 to 4, characterized in that: Also includes: During execution, the usage of the shared register block is monitored; When it is detected that the usage of the shared register block exceeds the hardware limit, an automatic tuning method is adopted to determine the optimal shared register block size and segmentation strategy.

6. The operator parallel processing method according to claim 5, characterized in that: The automatic tuning method is used to determine the optimal shared register block size and segmentation strategy, including: Determine multiple candidate register block sizes and multiple candidate segmentation strategies based on hardware information and shapes of operator data; Determining a plurality of candidate configurations based on the plurality of candidate register block sizes and the plurality of candidate segmentation strategies; The performance of each candidate configuration is evaluated, and an optimal configuration is determined according to the performance evaluation result of each candidate configuration, wherein the optimal configuration includes an optimal shared register block size and a segmentation strategy.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the operator parallel processing method according to any one of claims 1 to 6 is implemented.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the operator parallel processing method according to any one of claims 1 to 6 is implemented.

9. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the operator parallel processing method according to any one of claims 1 to 6 is implemented.

Images