Convolution operation acceleration method, convolution operation accelerator, computer readable storage medium and program product

Abstract

The application relates to a convolution operation acceleration method, a convolution operation accelerator, a computer readable storage medium and a program product. The method comprises the following steps: acquiring input tensor data and filter weight data; scheduling a general systolic array to perform linear transformation processing on the input tensor data and the filter weight data based on matrix multiplication, so as to obtain an input matrix and a filter matrix in a transformation domain; preloading each weight element in the filter matrix to each processing unit in a special parallel multiply-accumulate array; distributing each input element in the input matrix to each processing unit in parallel, scheduling each processing unit to perform element-by-element multiplication on the weight element and the input element, so as to obtain an intermediate result matrix; and scheduling the general systolic array to perform inverse transformation processing on the intermediate result matrix, so as to obtain a target operation result in a spatial domain. According to the method, optimal hardware can be matched to perform at different algorithm stages, so that operation efficiency and resource utilization rate are improved.

Description

Technical Field

[0001] This application relates to the fields of integrated circuit design and artificial intelligence computing hardware technology, and in particular to a method for accelerating convolution operations, a convolution operation accelerator, a computer-readable storage medium, and a computer program product. Background Technology

[0002] Traditional convolution calculations, which directly perform multiplication and accumulation on the window, typically require a large number of multiplication and accumulation operations, resulting in low computational efficiency. Therefore, a solution has emerged that uses dedicated integrated circuit (ASIC) hardware to perform a linear transformation of the convolution operation using the Winograd algorithm, significantly reducing the number of multiplications. However, this ASIC hardware is only utilized when the Winograd algorithm is executed. The majority of the time it remains idle, leading to significant waste of resources and power. Summary of the Invention

[0003] Therefore, it is necessary to provide a method for accelerating convolution operations, a convolution operation accelerator, a computer-readable storage medium, and a computer program product that can improve the utilization of computing resources, in response to the above-mentioned technical problems.

[0004] In a first aspect, this application provides a method for accelerating convolution operations, applied to a controller in a convolution operation accelerator, wherein the convolution operation accelerator further includes a general-purpose systolic array and a dedicated parallel multiply-accumulate array;

[0005] The method includes:

[0006] Obtain the input tensor data and filter weight data for the convolution operation;

[0007] The general systolic array is scheduled to perform a linear transformation based on matrix multiplication on the input tensor data and the filter weight data to obtain the input matrix and filter matrix in the transformation domain.

[0008] Each weight element in the filter matrix is ​​preloaded into each processing unit in the dedicated parallel multiply-accumulate array;

[0009] Each input element in the input matrix is ​​distributed in parallel to each of the processing units, and each processing unit is scheduled to multiply the weight element with the input element element by element to obtain an intermediate result matrix.

[0010] The general pulsating array is scheduled to perform an inverse transformation on the intermediate result matrix to obtain the target computation result in the spatial domain.

[0011] In one embodiment, the method further includes:

[0012] Determine the parameters required for convolution operations in the current scenario;

[0013] Based on the convolution operation requirement parameters, a partitioning instruction is generated and sent to the dedicated parallel multiply-accumulate array. The partitioning instruction is used to instruct the dedicated parallel multiply-accumulate array to split the multiple processing units according to the convolution operation requirement parameters to form a subarray that matches the convolution operation requirement parameters.

[0014] The steps of preloading each weight element of the filter matrix into each processing unit of the dedicated parallel multiply-accumulate array, distributing each input element of the input matrix to each processing unit in parallel, and scheduling each processing unit to perform element-wise multiplication of the weight elements with the input elements to obtain an intermediate result matrix further include:

[0015] Each of the weight elements is preloaded into each processing unit in the subarray;

[0016] The input elements are distributed in parallel to the processing units in the subarray, and the processing units in the subarray are scheduled to multiply the weight elements and the input elements element by element to obtain the intermediate result matrix.

[0017] In one embodiment, the method further includes:

[0018] Determine the parameters required for convolution operations in the current scenario;

[0019] When the convolution operation requirement parameters match the total number of processing units in the dedicated parallel multiply-accumulate array, each of the processing units is activated;

[0020] The scheduling of each of the processing units involves multiplying the weight elements element-wise with the input elements to obtain an intermediate result matrix, including:

[0021] Each of the processing units is scheduled to perform element-wise multiplication of the weight element and the input element within the current clock cycle to obtain the intermediate result matrix.

[0022] In one embodiment, the method further includes:

[0023] When each of the processing units is scheduled to perform element-wise multiplication, the general systolic array is scheduled in parallel to perform a linear transformation operation based on matrix multiplication on the input tensor data and filter weight data of the next set of convolution operations.

[0024] In one embodiment, after preloading the weight elements of the filter matrix into the processing units of the dedicated parallel multiply-accumulate array, the method further includes:

[0025] Each of the weight elements is statically stored in its corresponding processing unit.

[0026] In one embodiment, distributing the input elements of the input matrix in parallel to the respective processing units includes:

[0027] Based on each of the input elements, a corresponding synchronization instruction is generated and sent to the dedicated parallel multiply-accumulate array. The synchronization instruction is used to open the cross switch in the dedicated parallel multiply-accumulate array so that each of the input elements arrives at each of the processing units synchronously, or to trigger the broadcast bus in the dedicated parallel multiply-accumulate array to synchronously execute the distribution broadcast operation of each of the input elements to each of the processing units.

[0028] Secondly, this application also provides a convolution operation accelerator, including a general-purpose systolic array, a dedicated parallel multiply-accumulate array, and a controller;

[0029] The general-purpose pulsating array is used to perform matrix transformation operations and inverse matrix transformation operations.

[0030] The dedicated parallel multiply-accumulate array includes multiple processing units for performing element-wise multiplication operations;

[0031] The controller is configured to acquire input tensor data and filter weight data for convolution operations; schedule the general-purpose systolic array to perform a linear transformation based on matrix multiplication on the input tensor data and the filter weight data to obtain an input matrix and a filter matrix in the transform domain; preload each weight element in the filter matrix into each processing unit in the dedicated parallel multiply-accumulate array; distribute each input element in the input matrix to each processing unit in parallel, and schedule each processing unit to perform element-wise multiplication of the weight elements with the input elements to obtain an intermediate result matrix; and schedule the general-purpose systolic array to perform an inverse transformation on the intermediate result matrix to obtain the target operation result in the spatial domain.

[0032] In one embodiment, an on-chip buffer unit is also included;

[0033] The controller is further configured to store the input matrix and the filter matrix generated by the general systolic array into the on-chip cache unit;

[0034] The controller is further configured to read the filter matrix from the on-chip cache unit, preload each weight element in the filter matrix into each of the processing units, and read the input matrix from the on-chip cache unit, and distribute each input element in the input matrix to each of the processing units in parallel.

[0035] Thirdly, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, implements the convolution operation acceleration method described in any of the embodiments of the first aspect.

[0036] Fourthly, this application also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, implements the convolution operation acceleration method described in any of the embodiments of the first aspect.

[0037] The aforementioned convolution operation acceleration methods, convolution operation accelerators, computer-readable storage media, and computer program products, by employing general-purpose systolic arrays, dedicated parallel multiply-accumulate arrays, and convolution operation accelerators built with heterogeneous computing architectures, utilize a controller to schedule the general-purpose systolic array for computation during the matrix component-based transformation and inverse transformation stages of the Winograd algorithm. During the element-wise multiplication stage of the Winograd algorithm, the controller preloads each weight element of the filter matrix into each processing unit of the dedicated parallel multiply-accumulate array and distributes each input element of the input matrix to each processing unit in parallel. This allows each processing unit to synchronously execute the multiplication operation between the weight elements and the input elements. This enables the optimal hardware to be matched to perform operations at different stages of the Winograd algorithm, thereby maximizing the overall performance and energy efficiency of the convolution operation accelerator and improving the utilization of computing resources. Attached Figure Description

[0038] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0039] Figure 1 A block diagram of a convolution operation accelerator 100 in one embodiment;

[0040] Figure 2 This is a flowchart illustrating a method for accelerating convolution operations in one embodiment;

[0041] Figure 3 This is a schematic diagram of the element-by-element multiplication step in one embodiment;

[0042] Figure 4 This is a flowchart illustrating the partitioning process of a dedicated parallel multiply-accumulate array in one embodiment;

[0043] Figure 5 This is a flowchart illustrating the time-division processing steps of a dedicated parallel multiply-accumulate array in one embodiment. Detailed Implementation

[0044] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0045] It should be noted that the terms "comprising" and "having," and any variations thereof, used in this application, are intended to cover non-exclusive inclusion. The term "multiple" as used in this application refers to two or more. The term "and / or" as used in this application refers to one of the solutions, or any combination of multiple solutions. The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use, and processing of related data must comply with relevant regulations.

[0046] In one exemplary embodiment, the convolution operation acceleration method provided in this application can be applied to, for example, Figure 1 The controller 102 is shown in the convolution operation accelerator 100. The convolution operation accelerator 100 also includes a general-purpose systolic array 104 and a dedicated parallel multiply-accumulate array 106.

[0047] Optionally, in some embodiments, the general-purpose systolic array 104 can be used to perform transformation and inverse transformation operations in a matrix multiplication-based linear transformation algorithm (hereinafter referred to as the Winograd algorithm). The number of processing units of the dedicated parallel multiply-accumulate array 106 matches the size of the transformation domain matrix in the Winograd algorithm. The dedicated parallel multiply-accumulate array 106 is used to perform element-wise multiplication operations in the Winograd algorithm.

[0048] like Figure 2 As shown, the convolution operation acceleration method provided in this embodiment may include the following steps S202 to S210. Wherein:

[0049] Step S202: Obtain the input tensor data and filter weight data for the convolution operation.

[0050] For example, the controller 102 can issue a read instruction to an external memory to read the input tensor data and filter weight data from the external memory. The input tensor data can be the input feature data of the neural network. The filter weight data can be the network parameters after the neural network has been trained. The neural network uses the filter weight data to perform convolution operations on the input tensor data. The convolution operation accelerator 100 is used to reduce the number of multiplications in the convolution operation performed by the neural network based on the Winograd algorithm, thereby simplifying the execution complexity of the convolution operation and achieving the effect of accelerating the convolution operation.

[0051] Step S204: Schedule the general systolic array to perform a linear transformation based on matrix multiplication on the input tensor data and filter weight data to obtain the input matrix and filter matrix in the transformation domain.

[0052] For example, controller 102 can schedule general-purpose systolic array 104 to perform a linear transformation based on matrix multiplication on the input tensor data to obtain the input matrix of the input tensor data in the transform domain of the Winograd algorithm. It can also schedule general-purpose systolic array 104 to perform a linear transformation based on matrix multiplication on the filter weight data to obtain the filter matrix of the filter weight data in the transform domain of the Winograd algorithm.

[0053] Alternatively, in some implementations, the input matrix can be obtained with reference to the following formula:

[0054] , where V is the input matrix, d is the input tensor data, and B is the predefined input transformation matrix in the Winograd algorithm.

[0055] Alternatively, in other implementations, the filter matrix can be obtained with reference to the following formula:

[0056] Where U is the filter matrix, g is the filter weight data, and G is the predefined weight transformation matrix in the Winograd algorithm.

[0057] Step S206: Preload each weight element in the filter matrix into each processing unit in the dedicated parallel multiply-accumulate array.

[0058] For example, controller 102 can first send a preload start command to dedicated parallel multiply-accumulate array 106 to obtain the storage address of each processing unit in dedicated parallel multiply-accumulate array 106. According to a one-to-one allocation rule, a storage correspondence is established between each weight element in the filter matrix and the storage address of each processing unit. Based on the storage correspondence, each weight element in the filter matrix is ​​preloaded into the corresponding processing unit in dedicated parallel multiply-accumulate array 106.

[0059] Step S208: Distribute each input element in the input matrix to each processing unit in parallel, and schedule each processing unit to multiply the weight elements with the input elements element by element to obtain an intermediate result matrix.

[0060] For example, controller 102 can establish an access correspondence between each input element in the input matrix and the access address of each processing unit according to a one-to-one allocation rule. Based on this correspondence, each input element in the input matrix is ​​distributed in parallel to each processing unit. Each processing unit is scheduled to synchronously perform element-wise multiplication of its pre-loaded weight elements with the currently received input element to obtain the operation result of each processing unit. The operation results of each processing unit in the dedicated parallel multiply-accumulate array 106 are used to construct a matrix, forming an intermediate result matrix within the transform domain of the Winograd algorithm.

[0061] Alternatively, in some implementations, element-wise multiplication can be performed with reference to the following formula:

[0062] , where M is the intermediate result matrix, U is the filter matrix, and V is the input matrix.

[0063] Alternatively, in some implementations, such as Figure 3 The diagram illustrates an element-wise multiplication of processing units. Here, g1, g2, and g3 are the weight elements in the filter matrix U, and d1, d2, and d3 are the input elements in the input matrix V.

[0064] Step S210: Schedule the general pulsating array to perform inverse transformation on the intermediate result matrix to obtain the target operation result in the spatial domain.

[0065] For example, the controller 102 can schedule the general pulsating array 104 to perform inverse transformation processing on the intermediate result matrix in the transformation domain of the Winograd algorithm based on the Winograd algorithm, so as to obtain the corresponding target operation result in the spatial domain.

[0066] Alternatively, in some implementations, the target calculation result can be obtained by referring to the following formula:

[0067] , where Y is the target operation result, M is the intermediate result matrix, and A is the predefined output transformation matrix in the Winograd algorithm.

[0068] In the aforementioned convolution operation acceleration method, by employing a general-purpose systolic array, a dedicated parallel multiply-accumulate array, and a convolution operation accelerator with a heterogeneous computing architecture, the general-purpose systolic array is scheduled for computation by a controller during the matrix component-based transformation and inverse transformation stages of the Winograd algorithm. During the element-wise multiplication stage of the Winograd algorithm, the controller preloads each weight element in the filter matrix into each processing unit of the dedicated parallel multiply-accumulate array and distributes each input element in the input matrix to each processing unit in parallel. This allows each processing unit to synchronously execute the multiplication operation between the weight elements and the input elements. This method can match the optimal hardware to perform the operation at different stages of the Winograd algorithm, thereby maximizing the overall performance and energy efficiency of the convolution operation accelerator and improving the utilization of computing resources.

[0069] In one exemplary embodiment, the dedicated parallel multiply-accumulate array can be designed as a reconfigurable array. For example... Figure 4 As shown, a partitioning method is also provided, including the following steps S402 to S408. Wherein:

[0070] Step S402: Determine the convolution operation requirements parameters for the current scenario.

[0071] Step S404: Generate partitioning instructions based on convolution operation requirements parameters, and send the partitioning instructions to a dedicated parallel multiply-accumulate array.

[0072] The partitioning instruction is used to instruct the dedicated parallel multiply-accumulate array to split multiple processing units according to the convolution operation requirements parameters, forming subarrays that match the convolution operation requirements parameters.

[0073] For example, controller 102 can read convolution operation requirement parameters for the current scenario from external memory. These parameters can characterize the transform domain matrix size parameters of the Winograd algorithm currently being applied. Based on these parameters, the minimum total number of processing units required to execute the Winograd algorithm can be determined. A partitioning instruction for the dedicated parallel multiply-accumulate array 106 is generated based on the minimum total number of processing units, and this instruction is sent to the dedicated parallel multiply-accumulate array 106 to instruct it to split itself into several subarrays that satisfy the minimum total number of processing units.

[0074] Step S406: Preload each weight element into each processing unit in the subarray.

[0075] Step S408: Distribute each input element in parallel to each processing unit in the subarray, and schedule each processing unit in the subarray to multiply the weight element with the input element element by element to obtain an intermediate result matrix.

[0076] For example, controller 102 can determine the subarray required for the current convolution operation. The weight elements transformed by the current convolution operation are preloaded into the processing units within the subarray. The input elements transformed by the current convolution operation are distributed in parallel to the processing units within the subarray, and the processing units within the subarray are scheduled to perform element-wise multiplication of the weight elements corresponding to the current convolution operation with the input elements to form the corresponding intermediate result matrix.

[0077] In this embodiment, a partitioning instruction is generated based on the convolution operation requirements parameters in the current scenario. The partitioning instruction is used to split the dedicated parallel multiply-accumulate array into subarrays. The subarrays are then used to multiply the weight elements corresponding to the current convolution operation with the input elements element by element, which can improve the resource utilization of the dedicated parallel multiply-accumulate array.

[0078] In one exemplary embodiment, such as Figure 5 As shown, a time-sharing processing method is provided, including the following steps S502 to S506. Wherein:

[0079] Step S502: Determine the convolution operation requirements parameters for the current scenario.

[0080] Step S504: When the convolution operation requirement parameters match the total number of processing units in the dedicated parallel multiply-accumulate array, activate each processing unit.

[0081] Step S506: Schedule each processing unit to multiply the weight elements and input elements element by element within the current clock cycle to obtain an intermediate result matrix.

[0082] The parameters required for convolution operations can be represented as F(m,r). Here, m is the size of the input block of the neural network during the convolution operation, and r is the size of the output block of the neural network during the convolution operation.

[0083] Dedicated parallel multiply-accumulate arrays can employ an M×N processing cell grid.

[0084] For example, the controller 102 can read the convolution operation requirement parameters for the current scenario from external memory. These parameters can characterize the transform domain matrix size of the Winograd algorithm currently being applied. When the convolution operation requirement parameters match the total number of processing units in the dedicated parallel multiply-accumulate array 106, it can be determined that all processing units in the dedicated parallel multiply-accumulate array 106 need to participate in the operation within the current clock cycle. Therefore, the controller 102 can send activation commands to each processing unit in the dedicated parallel multiply-accumulate array 106 to activate each processing unit. The weight elements corresponding to the current convolution operation are preloaded into each processing unit, and the input elements corresponding to the current convolution operation are distributed in parallel to each processing unit. Each processing unit is scheduled to perform element-wise multiplication of the weight elements and input elements within the current clock cycle to obtain an intermediate result matrix.

[0085] In this embodiment, when the convolution operation requirement parameters match the total number of processing units in the dedicated parallel multiply-accumulate array, each processing unit is activated and scheduled to perform element-wise multiplication of the weight elements and input elements within the current clock cycle to obtain an intermediate result matrix. This allows all processing units in the dedicated parallel multiply-accumulate array to process the element-wise multiplication operation of the current convolution operation in a time-sharing manner, thereby improving the resource utilization of the dedicated parallel multiply-accumulate array.

[0086] In an exemplary embodiment, the convolution operation acceleration method provided in this application may further include: when scheduling each processing unit to perform element-wise multiplication operations, parallel scheduling a general systolic array to perform matrix multiplication-based linear transformation processing operations on the input tensor data and filter weight data of the next set of convolution operation operations.

[0087] In this embodiment, by scheduling a dedicated parallel multiply-accumulate array and a general systolic array in parallel to perform the operation, the overall efficiency of convolution operation can be improved.

[0088] In an exemplary embodiment, the convolution operation acceleration method provided in this application may further include: statically storing each weight element in its corresponding processing unit.

[0089] For example, controller 102 can set dedicated parallel multiply-accumulate array 106 to weight static mode to preload each weight element one-to-one and statically store it in the local register of each processing unit.

[0090] Optionally, in some implementations, the controller 102 can set the dedicated parallel multiply-accumulate array 106 to a lockstep parallel computing mode to distribute each input element to the corresponding processing unit in parallel at once. This ensures that all processing units synchronously perform element-wise multiplication operations within the same clock cycle to generate an intermediate result matrix.

[0091] In this embodiment, by configuring a weight static mode and a lockstep parallel computing mode for the dedicated parallel multiply-accumulate array, the weight elements are preloaded and statically maintained, and the input elements are distributed in parallel. This enables the entire matrix multiplication to be completed in a single cycle. Furthermore, there is no data flow between processing units within the dedicated parallel multiply-accumulate array, which simplifies the structure of the dedicated parallel multiply-accumulate array and improves its energy efficiency.

[0092] In an exemplary embodiment, step S208 may further include: generating a corresponding synchronization instruction based on each input element, and sending the synchronization instruction to a dedicated parallel multiply-accumulate array.

[0093] The synchronization instruction is used to open the crossbar switch in the dedicated parallel multiply-accumulate array so that each input element arrives at each processing unit synchronously, or to trigger the broadcast bus in the dedicated parallel multiply-accumulate array to synchronously execute the distribution broadcast operation of each input element to each processing unit.

[0094] In this embodiment, by using a crossbar switch or a broadcast bus for parallel distribution, it can be ensured that each processing unit receives the input elements simultaneously, so that all subsequent processing units can complete the calculation within the same clock cycle, thereby improving the synchronization of the generation of the intermediate result matrix.

[0095] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps. It is understood that the steps in different embodiments can be freely combined as needed, and all non-contradictory solutions formed by such combinations are within the scope of protection of this application.

[0096] Based on the same inventive concept, this application also provides a convolution accelerator for implementing the above-mentioned convolution operation acceleration method. The solution provided by this convolution accelerator is similar to the implementation described in the above method; therefore, the specific limitations in one or more convolution accelerator embodiments provided below can be found in the limitations of the convolution operation acceleration method described above, and will not be repeated here.

[0097] In one exemplary embodiment, a convolution operation accelerator is provided, including a general-purpose systolic array, a dedicated parallel multiply-accumulate array, and a controller.

[0098] A general-purpose systolic array is used to perform matrix transformation operations and inverse matrix transformation operations.

[0099] A dedicated parallel multiply-accumulate array includes multiple processing units for performing element-wise multiplication operations.

[0100] The controller acquires the input tensor data and filter weight data for the convolution operation. It schedules a general-purpose systolic array to perform a linear transformation based on matrix multiplication on the input tensor data and filter weight data, obtaining the input matrix and filter matrix in the transform domain. The weight elements of the filter matrix are preloaded into processing units within a dedicated parallel multiply-accumulate array. The input elements of the input matrix are distributed in parallel to the processing units, and each processing unit performs element-wise multiplication of the weight elements with the input elements to obtain an intermediate result matrix. The general-purpose systolic array then performs an inverse transformation on the intermediate result matrix to obtain the target computation result in the spatial domain.

[0101] In one exemplary embodiment, the convolution operation accelerator further includes an on-chip buffer unit.

[0102] The controller is also used to store the input matrix and filter matrix generated by the general systolic array to the on-chip cache unit.

[0103] The controller is also used to read the filter matrix from the on-chip cache unit, preload each weight element in the filter matrix into each processing unit, and read the input matrix from the on-chip cache unit to distribute each input element in the input matrix to each processing unit in parallel.

[0104] In one exemplary embodiment, the controller is also configured to store the intermediate result matrix to an on-chip cache unit.

[0105] In one exemplary embodiment, the controller and the on-chip cache unit can form a storage and control system.

[0106] In one exemplary embodiment, the controller can be used to issue scheduling instructions to a general-purpose systolic array and a dedicated parallel multiply-accumulate array to control the flow and computation order of data in the heterogeneous architecture of the convolution operation accelerator.

[0107] The modules in the aforementioned convolution operation accelerator can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in hardware within or independently of the processor in a computer device, or stored in software within the memory of the computer device, so that the processor can call and execute the operations corresponding to each module.

[0108] Those skilled in the art will understand that the structures shown in the accompanying drawings of this application are merely block diagrams of some structures related to the solution of this application, and do not constitute a limitation on the computer device to which the solution of this application is applied. Specific computer devices may include more or fewer components than shown in the figures, or combine certain components, or have different component arrangements.

[0109] In one exemplary embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0110] In one exemplary embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps in the above-described method embodiments.

[0111] In one exemplary embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0112] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0113] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0114] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for accelerating convolution operations, characterized in that, A controller used in a convolution operation accelerator, the convolution operation accelerator also including a general-purpose systolic array and a dedicated parallel multiply-accumulate array; The method includes: Obtain the input tensor data and filter weight data for the convolution operation; The general systolic array is scheduled to perform a linear transformation based on matrix multiplication on the input tensor data and the filter weight data to obtain the input matrix and filter matrix in the transformation domain. Determine the convolution operation requirements parameters for the current scenario; the convolution operation requirements parameters are used to characterize the transform domain matrix size parameters of the Winograd algorithm currently being applied. Based on the convolution operation requirement parameters, the minimum total number of processing units for executing the Winograd algorithm is determined. A partitioning instruction for the dedicated parallel multiply-accumulate array is generated according to the minimum total number of processing units. The partitioning instruction is sent to the dedicated parallel multiply-accumulate array. The partitioning instruction is used to instruct the dedicated parallel multiply-accumulate array to split multiple processing units according to the convolution operation requirement parameters to form multiple subarrays that satisfy the minimum total number of processing units. Each weight element in the filter matrix is ​​preloaded into each processing unit of the subarray; Each input element in the input matrix is ​​distributed in parallel to each processing unit, and each processing unit in the subarray is scheduled to multiply the weight element with the input element element by element to obtain an intermediate result matrix. The general pulsating array is scheduled to perform an inverse transformation on the intermediate result matrix to obtain the target computation result in the spatial domain.

2. The method according to claim 1, characterized in that, The method further includes: When the size parameter of the transform domain matrix matches the total number of processing units in the dedicated parallel multiply-accumulate array, each of the processing units is activated; The process of scheduling each processing unit in the subarray to perform element-wise multiplication of the weight element and the input element to obtain an intermediate result matrix includes: The processing units in the subarray are scheduled to multiply the weight elements and the input elements element by element within the current clock cycle to obtain the intermediate result matrix.

3. The method according to claim 1, characterized in that, The method further includes: When each of the processing units is scheduled to perform element-wise multiplication, the general systolic array is scheduled in parallel to perform a linear transformation operation based on matrix multiplication on the input tensor data and filter weight data of the next set of convolution operations.

4. The method according to claim 1, characterized in that, After preloading each weight element of the filter matrix into each processing unit of the subarray, the method further includes: Each of the weight elements is statically stored in its corresponding processing unit.

5. The method according to claim 1, characterized in that, The step of distributing each input element in the input matrix to each processing unit in parallel includes: Based on each of the input elements, a corresponding synchronization instruction is generated and sent to the dedicated parallel multiply-accumulate array. The synchronization instruction is used to open the cross switch in the dedicated parallel multiply-accumulate array so that each of the input elements arrives at each of the processing units synchronously, or to trigger the broadcast bus in the dedicated parallel multiply-accumulate array to synchronously execute the distribution broadcast operation of each of the input elements to each of the processing units.

6. A convolution operation accelerator, characterized in that, This includes general-purpose pulsating arrays, dedicated parallel multiply-accumulate arrays, and controllers; The general-purpose pulsating array is used to perform matrix transformation operations and inverse matrix transformation operations. The dedicated parallel multiply-accumulate array includes multiple processing units for performing element-wise multiplication operations; The controller is used to acquire the input tensor data and filter weight data of the convolution operation; The general systolic array is scheduled to perform a linear transformation based on matrix multiplication on the input tensor data and the filter weight data to obtain the input matrix and filter matrix in the transform domain; the convolution operation requirement parameters for the current scenario are determined; the convolution operation requirement parameters are used to characterize the transform domain matrix size parameters of the Winograd algorithm currently being applied; based on the convolution operation requirement parameters, the minimum total number of processing units for executing the Winograd algorithm is determined; a partitioning instruction for the dedicated parallel multiply-accumulate array is generated according to the minimum total number of processing units; the partitioning instruction is sent to the dedicated parallel multiply-accumulate array; the partitioning instruction is used to instruct the dedicated parallel multiply-accumulate array to split multiple processing units according to the convolution operation requirement parameters to form multiple subarrays that satisfy the minimum total number of processing units; Each weight element in the filter matrix is ​​preloaded into each processing unit in the subarray; each input element in the input matrix is ​​distributed in parallel to each processing unit, and each processing unit in the subarray is scheduled to perform element-wise multiplication of the weight elements and the input elements to obtain an intermediate result matrix; the general systolic array is scheduled to perform inverse transformation on the intermediate result matrix to obtain the target operation result in the spatial domain.

7. The convolution operation accelerator according to claim 6, characterized in that, It also includes on-chip cache units; The controller is further configured to store the input matrix and the filter matrix generated by the general systolic array into the on-chip cache unit; The controller is further configured to read the filter matrix from the on-chip cache unit, preload each weight element in the filter matrix into each of the processing units, and read the input matrix from the on-chip cache unit, and distribute each input element in the input matrix to each of the processing units in parallel.

8. The convolution operation accelerator according to claim 6, characterized in that, The controller is used to issue scheduling instructions to the general pulsating array and the dedicated parallel multiply-accumulate array to control the flow and computation order of data in the heterogeneous architecture of the convolution operation accelerator.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 5.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 5.

Images