Convolution operator optimization implementation method, electronic device, and storage medium

Abstract

A convolution operator optimization implementation method, an electronic device and a storage medium. The convolution operator optimization implementation method comprises: obtaining input parameter information of a convolution operator and constraint conditions of a hardware convolution instruction directly supported by first hardware, wherein the constraint conditions comprise constraints on input parameters of the hardware convolution instruction when the hardware convolution instruction directly supported by the first hardware is used to implement the convolution operator; determining whether to decompose the convolution operator according to the input parameter information of the convolution operator and the constraint conditions of the hardware convolution instruction; and in response to a determination that the convolution operator is to be decomposed, decomposing the convolution operator into a plurality of operators to implement the convolution operator by executing the plurality of operators, wherein at least one of the plurality of operators is executed by using the hardware instruction directly supported by the first hardware. The convolution operator optimization implementation method can make full use of the input parameter information of the convolution operator to optimize the selection of an implementation scheme of the convolution operator and improve the effective use of hardware resources of the first hardware.

Description

Technical Field

[0001] Embodiments of this disclosure relate to an optimized implementation method for a convolution operator, an electronic device, and a storage medium. Background Technology

[0002] As a bridge connecting high-level algorithms and low-level hardware, the compiler can transform abstract high-level algorithm descriptions into hardware-executable machine instructions, which are then executed by the corresponding processing units to realize the high-level algorithm.

[0003] Convolution is a crucial operation in deep learning and image processing, and its execution efficiency directly impacts model performance. Operators are the fundamental units of computation in deep learning models. As the complexity of neural networks increases and the amount of data processed grows, convolution operators have become increasingly diverse and complex. Therefore, determining the optimal implementation scheme for convolution operators and improving their execution performance on hardware platforms is a pressing issue for compilers when translating these operators into hardware-executable machine instructions. Summary of the Invention

[0004] This disclosure provides at least one embodiment of a method for optimizing the implementation of a convolution operator. The method includes: obtaining input parameter information of a convolution operator to be compiled and constraints of hardware convolution instructions directly supported by a first hardware, wherein the constraints include constraints on the input parameters of the hardware convolution instructions when the convolution operator is implemented by the hardware convolution instructions directly supported by the first hardware; determining whether to decompose the convolution operator based on the input parameter information and the constraints; decomposing the convolution operator into multiple operators in response to determining to decompose the convolution operator; compiling the multiple operators into machine instructions to implement the convolution operator by executing the machine instructions of the multiple operators, wherein at least one of the multiple operators is executed by hardware instructions directly supported by the first hardware, the hardware instructions including instructions directly supported by the first hardware for implementing functions other than the convolution operator.

[0005] For example, at least one embodiment of the convolution operator optimization implementation method provided in this disclosure further includes: obtaining a first condition determined based on prior knowledge; wherein, determining whether to decompose the convolution operator according to the input parameter information and the constraint condition includes: determining whether to decompose the convolution operator according to the input parameter information, the first condition and the constraint condition.

[0006] For example, in at least one embodiment of the convolution operator optimization implementation method provided in this disclosure, the step of determining whether to decompose the convolution operator based on the input parameter information, the first condition, and the constraint condition includes: determining not to decompose the convolution operator in response to the input parameter information satisfying the constraint condition, or the input parameter information not satisfying the constraint condition but satisfying the first condition; and determining to decompose the convolution operator in response to the input parameter information not satisfying the first condition and not satisfying the constraint condition.

[0007] For example, in the convolution operator optimization implementation method provided in at least one embodiment of this disclosure, the first condition includes at least one of the following: the convolution kernel size of the convolution operator is greater than 7×7, the convolution operator is a depth-separable convolution operator, the convolution operator is a transposed convolution operator, or the convolution kernel size of the convolution operator is equal to 1×1.

[0008] For example, at least one embodiment of the present disclosure provides a method for optimizing the convolution operator, which further includes: in response to determining that the convolution operator will not be decomposed, deleting the reduction function of the convolution operator from the registration list; re-registering a new reduction function in the registration list, and processing the convolution operator according to the new reduction function.

[0009] For example, in at least one embodiment of the convolution operator optimization implementation method provided in this disclosure, the step of processing the convolution operator according to the new reduction function includes: in response to the input parameter information satisfying the constraint condition, calling the hardware convolution instruction directly supported by the first hardware to implement the convolution operator.

[0010] For example, in the convolution operator optimization implementation method provided in at least one embodiment of this disclosure, the step of processing the convolution operator according to the new reduction function includes: in response to the input parameter information not satisfying the constraint condition, searching the kernel configuration parameters through an optimization strategy and determining whether at least one kernel has been generated; in response to determining that multiple kernels have been generated, determining a target kernel with better performance than other kernels from the multiple kernels to implement the convolution operator; or in response to determining that only one kernel has been generated, using the one kernel to implement the convolution operator.

[0011] For example, at least one embodiment of the convolution operator optimization implementation method provided in this disclosure further includes: in response to kernel generation failure, implementing the convolution operator using native operators through a second hardware.

[0012] For example, in the convolution operator optimization implementation method provided in at least one embodiment of this disclosure, the first hardware includes a graphics processor and the second hardware includes a central processing unit.

[0013] For example, in the convolution operator optimization implementation method provided in at least one embodiment of this disclosure, the plurality of operators include an image transposition operator and a matrix multiplication operator, wherein at least the matrix multiplication operator is executed by the hardware instructions directly supported by the first hardware.

[0014] For example, in the convolution operator optimization implementation method provided in at least one embodiment of this disclosure, the input parameter information includes the shape of the input tensor, the shape of the convolution kernel tensor, the stride, padding, and grouping.

[0015] At least one embodiment of this disclosure also provides an electronic device, the electronic device comprising: a memory that non-transitoryly stores computer-executable instructions; and a processor configured to run the computer-executable instructions, wherein the computer-executable instructions are executed by the processor to implement the convolution operator optimization implementation method according to any embodiment.

[0016] At least one embodiment of this disclosure also provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, implement the convolution operator optimization implementation method described in any embodiment.

[0017] The convolution operator optimization implementation method provided in at least one embodiment of this disclosure provides a new implementation scheme for convolution operators. It can implement some convolution operators that are not supported by the operator library by decomposition, thereby enabling more convolution operators to be executed efficiently. Furthermore, the operators obtained after decomposing the convolution operators can be executed by hardware instructions directly supported by the first hardware, thereby realizing the effective utilization of the computing resources of the first hardware. Attached Figure Description

[0018] To more clearly illustrate the technical solutions of the embodiments of this disclosure, the accompanying drawings of the embodiments will be briefly described below. Obviously, the drawings described below only relate to some embodiments of this disclosure and are not intended to limit this disclosure.

[0019] Figure 1 This is a schematic diagram of a general-purpose graphics processing unit (GPGPU).

[0020] Figure 2A A flowchart illustrating an optimized implementation method for a convolution operator provided in at least one embodiment of this disclosure.

[0021] Figure 2B This is a flowchart illustrating an exemplary method for optimizing a convolution operator, provided for at least one embodiment of this disclosure.

[0022] Figure 3 A schematic diagram of a non-transitory computer-readable storage medium provided for at least one embodiment of the present disclosure.

[0023] Figure 4 This is a schematic diagram of the structure of an electronic device provided in at least one embodiment of the present disclosure. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this disclosure. All other embodiments obtained by those skilled in the art based on the described embodiments of this disclosure without creative effort are within the scope of protection of this disclosure.

[0025] Unless otherwise defined, the technical or scientific terms used in this disclosure shall have the ordinary meaning understood by one of ordinary skill in the art to which this disclosure pertains. The terms “first,” “second,” and similar terms used in this disclosure do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as “comprising” or “including” mean that the element or object preceding the word encompasses the element or object listed following the word and its equivalents, without excluding other elements or objects. Terms such as “connected” or “linked” are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. Terms such as “upper,” “lower,” “left,” and “right” are used only to indicate relative positional relationships, which may change accordingly when the absolute position of the described objects changes. It should be understood that the various steps described in the method embodiments of this disclosure may be performed in different orders and / or in parallel. Furthermore, method embodiments may include additional steps and / or omit the steps shown. The scope of this disclosure is not limited in this respect.

[0026] The present disclosure will now be described through several specific embodiments. To keep the following description of the embodiments of the present disclosure clear and concise, detailed descriptions of known functions and known components may be omitted. When any component of an embodiment of the present disclosure appears in more than one drawing, the component is represented by the same or similar reference numerals in each drawing.

[0027] Neural networks can perform deep learning computations on image data, audio data, text data, or other input data that needs processing in application fields such as medical diagnosis and transportation. A Convolutional Neural Network (CNN) is a type of neural network that includes convolutional layers, which are used to extract local features from the input data. The convolution computation task in the convolutional layers is called a convolution operator, and convolution operators need to be compiled into machine instructions before they can be executed by hardware.

[0028] Convolution operators are typically computationally intensive, involving numerous repetitive and independent calculations. Implementing convolution operators using parallel processing capabilities or dedicated acceleration hardware can improve computational efficiency. Some hardware manufacturers, when providing such hardware, design specific machine instructions for common convolution scenarios. In at least one embodiment of this disclosure, this type of hardware is referred to as "first hardware," and the machine instructions specifically designed by the hardware manufacturer for convolution scenarios that can be directly executed by the hardware are referred to as "hardware convolution instructions." These hardware convolution instructions can be directly executed by the hardware units of the first hardware, thereby efficiently implementing specific convolution operators. That is, for some specific convolution operators (e.g., convolution operators in the operator library provided by the hardware manufacturer), it is not necessary to implement them through software simulation or by combining other general instructions; simply calling the hardware convolution instructions for these specific convolution operators is sufficient to implement them.

[0029] For example, the core logic of convolution operations (such as sliding window calculation, matrix multiplication and accumulation, etc.) can be directly embedded in the hardware unit. The hardware unit can implement convolution operations by executing hardware convolution instructions. Compared with the method of combining multiple general instructions (such as addition instructions, multiplication instructions, etc.) through the central processing unit (CPU) to implement convolution operations, directly executing hardware convolution instructions to implement convolution operators can make full use of the computing power resources of the first hardware and improve computing efficiency.

[0030] The first hardware supporting the execution of convolution operators may include, for example, a graphics processing unit (GPU), a general-purpose computing on graphics processing units (GPGPU), a tensor processing unit (TPU), a neural processing unit (NPU), a data processing unit (DPU), an AI accelerator, or other artificial intelligence chips to be developed in the future, and the embodiments disclosed herein are not limited thereto.

[0031] Figure 1 This is a schematic diagram of a general-purpose graphics processing unit (GPGPU).

[0032] like Figure 1 As shown, a general-purpose graphics processor is an array of programmable multiprocessors. For example, a programmable multiprocessor can be a streaming processor cluster (SPC), such as including... Figure 1 The diagram shows streaming processor clusters 1, ..., M, where M is a positive integer greater than 1. In a general-purpose graphics processor, one streaming processor cluster handles one computational task, or multiple streaming processor clusters handle one computational task. Multiple streaming processor clusters share data through a global cache or global memory.

[0033] like Figure 1 As shown, taking streaming processor cluster 1 as an example, one streaming processor cluster includes multiple computing units, such as... Figure 1 The system is structured as Computation Unit 1, Computation Unit 2, ..., Computation Unit N, where N is a positive integer. Each Computation Unit (CU) can perform arithmetic and logical operations, such as accumulation, reduction, and standard addition, subtraction, multiplication, and division. A Computation Unit includes multiple cores, each containing an Arithmetic Logic Unit (ALU), a floating-point unit, etc., which are used to execute specific computational tasks. In addition, a Computation Unit includes a register file and shared memory, used for hierarchical storage of source and destination data related to computational tasks. Shared memory within a Computation Unit is used to share data between cores within that unit. The register file is closer to the computation cores than the shared memory, and while it has a larger bandwidth, its capacity is smaller than that of shared memory.

[0034] like Figure 1 As shown, each computing unit also provides a tensor core for performing tensor computations, such as matrix multiplication. Tensor cores can accelerate matrix multiplication operations. Tensor cores across multiple computing units can be scheduled and controlled uniformly.

[0035] In parallel computing, computational tasks are typically executed through multiple threads. These threads are divided into multiple thread blocks before execution in a general-purpose graphics processor (or parallel computing processor), and then dispatched via a thread block distribution module. Figure 1(Not shown in the image) Multiple thread blocks are distributed to various computation units. All threads in a thread block must be assigned to the same computation unit for execution. Simultaneously, thread blocks are broken down into minimum execution thread bundles (or simply warps), each containing a fixed number (or less than this fixed number) of threads, for example, 32 threads. Multiple thread blocks can execute in the same computation unit or in different computation units.

[0036] In each computing unit, the thread beam scheduling / distribution module ( Figure 1 (Not shown in the diagram) Thread bundles are scheduled and allocated so that multiple computing cores within the computing unit can run thread bundles. Depending on the number of computing cores in the computing unit, multiple thread bundles within a thread block can be executed concurrently or in a time-sharing manner. Multiple threads within each thread bundle execute the same instructions. Memory execution instructions are issued to shared memory within the computing unit or further issued to intermediate-level caches, global caches, or global memory for read and write operations, etc.

[0037] Convolution operators need to be compiled into machine instructions by a compiler before they can be executed by hardware. In some deep learning frameworks (e.g., PyTorch 2.0), high-performance compilers (e.g., Inductor compiler) can be used to generate high-performance code for the backend of the accelerator, and high-performance programming languages ​​(e.g., Triton) can be used to improve the versatility of the design to achieve high-performance computation. After generating the optimal implementation for the operator, the compiler generates machine instructions corresponding to the optimal implementation, which can then be run by calling the processing unit (or processing core) in the hardware. The processing unit can access the corresponding storage device to read and store the tensor data of the operator (e.g., the input and output data of the operator).

[0038] For example, if the first piece of hardware is a graphics processing unit (GPU), and the GPU can support a certain convolution operator, and the compiler determines that using the operator library provided by the GPU manufacturer to implement the convolution operator is the optimal solution, the compiler will call the hardware convolution instruction corresponding to the convolution operator in the operator library, compile the convolution operator into a hardware convolution instruction directly supported by the GPU, and then use the tensor core in the GPU to implement the convolution operator, thereby making full use of the computing power resources of the GPU's tensor core and improving computational efficiency.

[0039] However, the number of convolution operators directly supported by the hardware is usually limited. For some convolution operators in complex scenarios, hardware convolution instructions cannot be directly adapted. To address this, some compilers provide kernel generation capabilities, allowing developers to write code or generate custom implementations of convolution operators based on kernel templates, thus enabling convolution operator implementation without relying on operator libraries. For example, kernel templates can pre-design several selectable parameters (e.g., "kernel configuration parameters" in this disclosure). By combining these parameters, different implementation schemes for the convolution operator can be generated. The parameters of the kernel template have a significant impact on the performance of the final generated code. The number of configuration schemes that can be formed by these parameters is vast; for a simple convolution kernel, the number of configuration schemes may reach tens of millions or even larger. The generated kernel generally can only use vector kernels to implement the convolution calculations corresponding to the convolution operators.

[0040] For example, the Inductor compiler typically offers two approaches when automatically compiling convolution operators into machine instructions: one is to call predefined operators from the operator library, and the other is to automatically generate multiple Triton kernels based on a kernel template. All the obtained implementations are then placed into a search space for optimization to find the optimal solution for executing the convolution operator. If neither approach works, it falls back to PyTorch's native implementation. That is, if the first hardware's operator library does not support the convolution operator and a kernel cannot be generated (e.g., generation fails), the convolution operator is compiled into CPU-executable machine instructions, and the CPU implements the convolution operator.

[0041] The inventors of this disclosure have discovered that current compilers offer limited implementation options for convolution operators, offering only three methods: calling convolution operators from operator libraries, generating kernels, and falling back to native operators. When some convolution operators cannot be implemented by calling convolution operators from operator libraries or generating kernels, the only option is to fall back to using native operators. However, implementations using native operators cannot fully utilize the computing resources of the primary hardware, resulting in low computational efficiency.

[0042] This disclosure provides at least one embodiment of a method for optimizing the implementation of a convolution operator, an electronic device, and a storage medium. The method includes: obtaining input parameter information of a convolution operator to be compiled and constraints of hardware convolution instructions directly supported by first hardware, wherein the constraints include constraints on the input parameters of the hardware convolution instructions when the convolution operator is implemented by the hardware convolution instructions directly supported by the first hardware; determining whether to decompose the convolution operator based on the input parameter information of the convolution operator and the constraints of the hardware convolution instructions; in response to determining to decompose the convolution operator, decomposing the convolution operator into multiple operators; compiling the multiple operators into machine instructions to implement the convolution operator by executing the machine instructions of the multiple operators, wherein at least one of the multiple operators is executed by hardware instructions directly supported by the first hardware, the hardware instructions including instructions directly supported by the first hardware for implementing functions other than the convolution operator.

[0043] This convolution operator optimization implementation method provides a new implementation scheme for convolution operators. It can implement some convolution operators that are not supported by the operator library by decomposition, thereby enabling more convolution operators to be executed efficiently. Furthermore, the operators obtained after decomposition can be executed by hardware instructions directly supported by the first hardware, thus realizing the effective utilization of the computing resources of the first hardware.

[0044] The present disclosure will now be described in detail with reference to the accompanying drawings at least one embodiment, but the present disclosure is not limited to these specific embodiments.

[0045] Figure 2A A flowchart illustrating an optimized implementation method for a convolution operator provided in at least one embodiment of this disclosure.

[0046] like Figure 2A As shown, at least one embodiment of this disclosure provides a method for optimizing the implementation of a convolution operator, which includes steps S100-S400.

[0047] For example, in at least one embodiment of this disclosure, the convolution operator optimization implementation method can be executed during the compilation of the convolution operator. The convolution operator includes a series of computational operations involving local correlation computation via a sliding window. For example, the convolution operator includes sliding a fixed-size convolution kernel (e.g., a weight matrix) across input data (e.g., input elements in an input feature map) with a set stride, calculating the sum of the products of the convolution kernel and the input elements in the corresponding region at each position, and using this sum of products as the output element at the corresponding position in the output feature map.

[0048] like Figure 2AAs shown, in step S100, the input parameter information of the convolution operator to be compiled and the constraints of the hardware convolution instructions directly supported by the first hardware are obtained. The constraints include the constraints on the input parameters of the hardware convolution instructions when the convolution operator is implemented by the hardware convolution instructions directly supported by the first hardware.

[0049] For example, in at least one embodiment of this disclosure, the input parameter information of the obtained convolution operator may include the shape of the input tensor, the shape of the convolution kernel tensor, the stride, padding, and grouping.

[0050] A tensor is a data structure for storing sets of numbers. In neural networks, the data to be processed can be represented as tensors and then substituted into mathematical formulas for calculation. Tensors can have different dimensions, such as zero-dimensional tensors, one-dimensional tensors (e.g., vectors), two-dimensional tensors (e.g., matrices), three-dimensional tensors, and even higher-dimensional tensors. The specific values ​​in a tensor can be application data, such as image data, natural language data, or other types of data. The data structure storing input data can be called an input tensor, and the data structure storing weight data can be called a kernel tensor. For example, the shape of an input tensor can include its dimension and its size within that dimension. Similarly, the shape of a kernel tensor can include its dimension and its size within that dimension.

[0051] A one-dimensional tensor can be understood as a vector. For example, a one-dimensional tensor represented as [1,2,3] has a shape of 3 and stores the data 1, 2, and 3 respectively. A two-dimensional tensor can be understood as a matrix. For example, a 2x2 matrix represented as [[1,2],[3,4]] has a shape of 2×2 and stores the data 1, 2, 3, and 4 respectively. For example, a specific value in a two-dimensional tensor can represent the grayscale value of a pixel in a monochrome image. A three-dimensional tensor can be understood as a copy of multiple two-dimensional matrices. For example, a three-dimensional tensor represented as [[[1,2,3],[4,5,6]],[[7,8,9],[10,11,12]]] has a shape of 2×3×4 and stores the data 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, and 12 respectively. For example, a specific value in a three-dimensional tensor can represent the value of a pixel in an image with a specific color channel. Similarly, two images can be represented by a four-dimensional tensor. Each image has three colors (RGB), meaning it has three channels, and each channel has 4 rows and 4 columns of pixels. The batch size, channels, height, and width of the image can correspond to the four dimensions B, C, H, and W of the four-dimensional input tensor, respectively, with each dimension having a size of 2, 3, 4, and 4. Therefore, the shape of this four-dimensional tensor is 2×3×4×4. Further explanation of higher-dimensional tensors (four dimensions and above) is omitted.

[0052] The size of input tensors varies depending on the task scenario. For example, in image processing tasks, the input tensor typically has 3 channels, and its length (or width) can reach 224, 256, 448, 1024, 1280, 2048, or other sizes. When processing a large batch of images, the total image size can reach tens of megabytes (MB), hundreds of MB, or even larger, thus placing higher demands on the hardware used for image processing. For instance, GPUs with larger video memory are more efficient at processing large images.

[0053] The stride is the amount of space the convolutional kernel moves across the input feature map. It controls computational complexity (e.g., a larger stride results in higher computational efficiency) and the size of the output feature map (e.g., a larger stride results in a smaller output feature map). For example, in 2D convolution, the stride can include a height stride and a width stride. The height stride can be represented as `stride_h`, and the width stride as `stride_w`. Typically, both are equal, often equal to 1, indicating a movement of one pixel at a time. The stride setting varies across different convolutional operations. For instance, in 2D convolution, the height and width strides can be unequal, taking different values ​​such as 1, 2, 3, or others.

[0054] Padding involves adding extra pixels to the edges of the input feature map to control the size of the output feature map or preserve edge information. For example, the padding value can be 0 or other values. When the padding value is 0, it is often referred to as zero-padding. For instance, in one example, pad_h_top=1 and pad_h_bottom=1 could represent padding the top and bottom edges of the input feature map with one row of 0s each, and pad_w_left=1 and pad_w_right=1 could represent padding the left and right edges of the input feature map with one column of 0s each. Padding can also have other representations, such as using "same" or "padding=1" to indicate that the padding, combined with the convolution kernel, makes the size of the output feature map the same as the size of the input feature map, or using "valid" to indicate no padding, etc. The embodiments of this disclosure are not limited in this regard.

[0055] Grouped convolution divides the input feature map channels and the convolution kernel channels into several groups, performs convolution operations independently within each group, and finally concatenates the output results of all groups to form the final output feature map. This can reduce computational cost. For example, "groups=1" represents a regular convolution without grouping, and "groups=3" represents a grouping of 3. The number of groups is not limited to 3; it can also be 2, 4, 5, or other values. Typically, the number of input channels and the number of convolution kernels are divisible by the number of groups. For example, in a case where the input feature map has 12 channels, using 6 3×3 convolution kernels, and the number of groups is 3, it means that the input channels are divided into 3 groups of 4 channels each, and the convolution kernels are divided into 3 groups of 2 kernels each. In this example, the first group uses two convolutional kernels to process the first four channels of the input feature map, outputting two channels; the second group uses two convolutional kernels to process the middle four channels of the input feature map, outputting two channels; the third group uses two convolutional kernels to process the last four channels of the input feature map, outputting two channels. Finally, the outputs of these three groups (two channels each) are concatenated to obtain a six-channel feature map. In this example, the computational cost of grouped convolution is only one-third that of ordinary convolution without grouping, effectively reducing the computational load.

[0056] Since each convolution operator can include various input parameters such as the shape of the input tensor, the shape of the kernel tensor, stride, padding, and grouping, and the settings of each input parameter can vary greatly depending on the task scenario, the operator library provided by the hardware manufacturers of First Hardware usually only supports some of the more commonly used convolution operators, and cannot cover more convolution operators with diverse parameters.

[0057] For example, the first piece of hardware could be a graphics processing unit (GPU), a general-purpose graphics processing unit (GPGPU), a tensor processor (TPU), a neural network processor (NPU), a data processor (DPU), an AI accelerator, or other artificial intelligence chips to be developed in the future.

[0058] The processing unit (or computing unit) of the first hardware can include dedicated computing units (such as the hardware units mentioned above) and general-purpose computing units. For example, a dedicated computing unit may be a computing unit specifically designed for General Matrix Multiplication (GEMM) and can be used as an accelerator for deep learning. A general-purpose computing unit can handle any type of computational task, but is less efficient for specific computationally intensive tasks (e.g., matrix multiplication). For example, dedicated computing units in a GPU include tensor kernel circuits, while general-purpose computing units include vector kernel circuits. Tensor kernel circuits can perform computations in units of "matrix blocks," and the execution efficiency of convolution operators implemented using tensor kernel circuits is higher than that implemented using vector kernel circuits.

[0059] For example, the hardware convolution instructions directly supported by the first hardware are subject to certain constraints imposed by the first hardware. These constraints include, for instance, restrictions on the input parameters of the convolution instructions. Only when these constraints are met can the first hardware correctly and efficiently execute the corresponding hardware convolution instructions. For example, the fixed data width of the computing unit of the first hardware limits the number of input or output channels of the convolution operators it supports. For example, if the fixed data width of the computing unit of the first hardware is 8, then the number of input or output channels of the convolution operators it supports must be a multiple of 8. For example, the access granularity of the memory bus of the first hardware limits the height and width of the feature maps of the convolution operators it supports. For example, if the global memory access of the first hardware is in 32-byte units, then the height and width of the feature maps must be a multiple of 32.

[0060] In at least one embodiment of this disclosure, after obtaining the input parameter information of the convolution operator and the constraints of the hardware convolution instruction directly supported by the first hardware, it can be determined whether the input parameter information of the convolution operator satisfies the constraints of the hardware convolution instruction, thereby determining whether to decompose the convolution operator.

[0061] like Figure 2A As shown, in step S200, it is determined whether to decompose the convolution operator based on the input parameter information of the convolution operator and the constraints of the hardware convolution instruction.

[0062] For example, if the stride in the input parameters of the convolution operator is 1, and the constraint of the hardware convolution instruction is that the stride is less than or equal to 3, then the input parameters of the convolution operator satisfy the constraint of the hardware convolution instruction, and it is determined that the convolution operator should not be decomposed. Otherwise, the convolution operator can be decomposed.

[0063] In other words, after obtaining the input parameter information of the convolution operator and the constraints of the hardware convolution instructions directly supported by the first hardware, it is possible to determine whether to decompose the convolution operator based on the input parameter information of the convolution operator and the constraints of the hardware convolution instructions. This allows for the early determination of whether the convolution operator is one that can be directly executed by the hardware convolution instructions of the first hardware, and provides a new implementation scheme for convolution operators that cannot be directly executed by the hardware convolution instructions of the first hardware.

[0064] When it is determined that the convolution operator can be decomposed, it means that there is no hardware convolution instruction in the first hardware that can directly implement the convolution operator, then steps S300 and S400 can be executed.

[0065] like Figure 2AAs shown, in step S300, in response to determining to decompose the convolution operator, the convolution operator is decomposed into multiple operators. In step S400, the multiple operators are compiled into machine instructions to implement the convolution operator by executing the machine instructions of the multiple operators, wherein at least one of the multiple operators is executed by hardware instructions directly supported by the first hardware, the hardware instructions including instructions directly supported by the first hardware for implementing other functions besides the convolution operator.

[0066] If the first hardware lacks instructions that can directly implement the convolution operator (hereinafter referred to as the "first convolution operator"), the implementation scheme of the first convolution operator can be determined through the compiler's built-in compilation process. This implementation scheme includes which instructions can be combined or used to implement the first convolution operator. However, determining the implementation scheme of the first convolution operator through the compiler's built-in compilation process (e.g., generating a kernel or falling back to the native operator) suffers from low compilation efficiency, high overhead, and the determined implementation scheme cannot utilize the computing resources of the first hardware, resulting in low computational efficiency.

[0067] In at least one embodiment of this disclosure, by adjusting the compiler's compilation process, if it is determined that there is no hardware convolution instruction in the first hardware that can implement the first convolution operator, the first convolution operator is first decomposed into multiple operators that can be executed by the hardware instructions of the first hardware. This can improve compilation efficiency and better utilize the hardware resources of the dedicated computing unit of the first hardware.

[0068] In at least one embodiment of this disclosure, "hardware instructions" can be understood as various instructions directly supported by the first hardware, including instructions for implementing functions other than the first convolution operator. The "hardware convolution instructions" of the first hardware are instructions specifically designed to implement a particular convolution operator. Besides hardware convolution instructions, the "hardware instructions" of the first hardware also include other more general instructions that can be directly executed by the first hardware. That is, although these hardware instructions cannot be directly used to implement the function of the first convolution operator, they can be used to implement various other functions. In at least one embodiment of this disclosure, these hardware instructions include "hardware convolution instructions" corresponding to convolution operators other than the first convolution operator, instructions for implementing other non-convolution operators, or instructions for implementing other auxiliary functions, etc.

[0069] For example, in at least one embodiment of this disclosure, the multiple operators obtained by decomposing the first convolution operator can be a combination of one or more operators such as image transposition operator, matrix multiplication operator, dimension transformation operator, and addition operator.

[0070] Image transcolumn operators, such as the Im2Col operator, can expand a local region of the input feature map (e.g., the region covered by the convolution kernel) into a column vector, thus converting the convolution operation into a matrix multiplication operation. That is, the convolution operation is the inner product of the input feature map and the convolution kernel within a sliding window, while the image transcolumn operator can arrange the elements of each sliding window column-wise, thus obtaining the same result as the convolution operation with a single matrix multiplication. Image transcolumn operators do not involve any computation and do not change the values ​​of the data; they only adjust the arrangement of the data in memory (or establish a mapping relationship) to serve subsequent computations. Essentially, it is a memory operation, so compared to operators involving computational operations, image transcolumn operators have lower search costs when placed in the search space.

[0071] Matrix multiplication operators, such as the Matmul operator, can be used to perform the multiplication of two matrices and are operators that can be directly supported by the dedicated computing unit of the first hardware. For example, matrix multiplication operators can be compiled by a compiler into hardware instructions that are directly supported by the first hardware.

[0072] Dimension transformation operators are operations used to adjust the dimensional structure of tensors. They do not change the element values ​​of the tensor, but only change the shape or dimensional order of the tensor to adapt to the input requirements of subsequent calculations. For example, dimensional transformation operators can include reshape, transpose, squeeze / unsqueeze, and flatten operators. For instance, the reshape operator can change the shape of a tensor but not the number or order of its elements. For instance, the transpose operator can transpose a tensor, that is, swap two dimensions of the tensor; for example, transposing a tensor A with a shape of 2×2×3 to a tensor B with a shape of 2×3×2. For instance, the squeeze operator can be used to delete a dimension of size 1, and the unsqueeze operator can add a dimension of size 1 at a specified position. For instance, the flatten operator can compress multiple dimensions of a tensor into one or two dimensions.

[0073] The addition operator is an operation that adds elements of two tensors, such as adding tensors to scalars or tensors to each other.

[0074] For example, in at least one embodiment of this disclosure, the first convolution operator can be decomposed into multiple operators to implement the first convolution operator by executing multiple operators, wherein at least one of the multiple operators is executed by hardware convolution instructions directly supported by the first hardware.

[0075] That is, in at least one embodiment of this disclosure, in addition to decomposing the first convolution operator into one or more combinations of operators such as image transposition operators, matrix multiplication operators, dimensionality transformation operators, and addition operators, the first convolution operator can also be decomposed into multiple other convolution operators corresponding to the hardware convolution instructions of the first hardware, or into combinations of one or more other convolution operators and other non-convolution operators. For example, the first convolution operator with a stride of 2 can be split into multiple second convolution operators with a stride of 1, and the multiple second convolution operators with a stride of 1 obtained from the decomposition can be directly executed by the hardware convolution instructions of the first hardware, thereby making full use of the hardware resources of the dedicated computing unit of the first hardware and improving computing efficiency.

[0076] For example, in at least one embodiment of this disclosure, step S200, which determines whether to decompose the convolution operator based on the input parameter information of the convolution operator and the constraints of the hardware convolution instruction, may include: in response to the input parameter information of the convolution operator not satisfying the constraints of the hardware convolution instruction, determining to decompose the convolution operator and attempting to decompose the convolution operator; in response to the input parameter information of the convolution operator satisfying the constraints of the hardware convolution instruction, determining not to decompose the convolution operator and attempting to call hardware instructions directly supported by the first hardware or generate a kernel to implement the convolution operator.

[0077] In other words, if the input parameters of the convolution operator do not meet the constraints of the hardware convolution instruction, it means that the first hardware does not have the convolution operator in its operator library, and the compiler cannot generate hardware convolution instructions that directly support the convolution operator. In this case, we can first try to decompose the convolution operator. If the convolution operator can be decomposed into multiple operators, and these multiple operators can be compiled by the compiler into hardware instructions that the first hardware can directly support, then we can use the computing resources of the first hardware to implement the convolution operator. Here, "try" indicates that there are cases where the convolution operator can be decomposed into multiple operators (and at least one of the multiple operators can be executed by the hardware instructions of the first hardware), and there are also cases where the convolution operator cannot be decomposed into multiple operators. When the convolution operator cannot be decomposed into multiple operators, we can enter the compiler's built-in compilation process to continue executing subsequent operations.

[0078] For example, if the input parameter information of the convolution operator satisfies the constraints of the hardware convolution instruction of the first hardware, then the convolution operator can be called from the operator library of the first hardware, and there is no need to decompose the convolution operator.

[0079] For example, after determining that the convolution operator will not be decomposed, subsequent operations can be performed directly through the compiler's built-in compilation optimization process. This optimization process could involve calling hardware convolution instructions directly supported by the first hardware and attempting to generate a kernel, then testing and searching for multiple implementation schemes corresponding to these two methods in the search space, and selecting the optimal implementation scheme to implement the convolution operator. Here, "attempting" implies that there are cases where kernel generation is successful and cases where kernel generation fails. When kernel generation fails, the compiler's optimization process can be skipped (without searching the search space), and the implementation scheme of the convolution operator in the operator library can be used directly to implement the convolution operator.

[0080] For example, the Inductor compiler can automatically generate one or more kernels based on a kernel template. It places all implementations of convolution operators from the operator library and all generated kernel implementations into a search space for automatic tuning. Automatic tuning determines which implementation achieves optimal performance (e.g., the shortest execution time) by running each implementation multiple times (e.g., testing with a small amount of input data) and averaging the execution time of each run. The determined optimal implementation is then compiled into low-level machine instructions, allowing the first hardware to perform convolution operations using the appropriate computational units based on these compiled machine instructions. For example, if the determined optimal implementation calls a convolution operator from the operator library, the compiler generates hardware convolution instructions directly supported by the first hardware, which executes the convolution operator using dedicated computational units (e.g., tensor kernels in a GPU). If the determined optimal implementation is one of the generated kernels, the compiler generates general-purpose instructions supported by the first hardware, which executes the convolution operator using general-purpose computational units (e.g., vector kernels in a GPU).

[0081] To improve the success rate of decomposing a convolution operator into multiple operators, prior knowledge can be used to exclude convolution operators that cannot be decomposed or whose performance is poor after decomposition, thereby avoiding resource waste caused by invalid attempts.

[0082] For example, at least one embodiment of the convolution operator optimization implementation method provided in this disclosure further includes obtaining a first condition determined based on prior knowledge. Here, prior knowledge may include basic decomposition rules and / or decomposition principles customized by developers according to the evolution of hardware architecture.

[0083] For example, basic decomposition rules might include: if the input feature map or kernel size of the convolution operator is already at its minimum size, then the convolution operator should not be decomposed. For example, decomposition principles customized by developers based on hardware architecture evolution might include: if some parameters of the convolution operator are too large or irregular, requiring decomposition into multiple parts for hardware architecture compatibility, but concatenating these parts would lead to lower computational efficiency in the resulting implementation, then such a convolution operator should not be decomposed. For example, in this case, developers can adaptively adjust the first condition based on hardware architecture evolution.

[0084] For example, in at least one embodiment of this disclosure, the first condition may include at least one of the following: the kernel size of the convolution operator is greater than 7×7, the convolution operator is a depth-separable convolution operator, the convolution operator is a transposed convolution operator, or the kernel size of the convolution operator is equal to 1×1; however, embodiments of this disclosure do not limit this.

[0085] For example, in at least one embodiment of this disclosure, step S200, which is to determine whether to decompose the convolution operator based on the input parameter information of the convolution operator and the constraints of the hardware convolution instruction, may include: determining whether to decompose the convolution operator based on the input parameter information of the convolution operator, the first condition, and the constraints of the hardware convolution instruction.

[0086] By determining whether the input parameters of the convolution operator satisfy the first condition determined based on prior knowledge and the constraints of the hardware convolution instruction, it is possible to determine whether to decompose the convolution operator. Prior knowledge can be used to improve the success rate of decomposing the convolution operator and avoid decomposing the convolution operator into a poor-performing operator, thereby improving the processing efficiency of the compiler.

[0087] For example, if the input parameter information of the convolution operator satisfies the constraints of the hardware convolution instruction, or if the input parameter information does not satisfy the constraints but satisfies the first condition, it is determined that the convolution operator will not be decomposed; if the input parameter information of the convolution operator does not satisfy the first condition and does not satisfy the constraints, it is determined that the convolution operator will be decomposed.

[0088] That is, in at least one embodiment of this disclosure, the situation where the convolution operator is not decomposed can include two types. One is that the input parameter information of the convolution operator satisfies the constraints of the hardware convolution instructions directly supported by the first hardware. In this case, the convolution operator in the operator library of the first hardware can be directly called, and the hardware resources of the first hardware can be directly utilized without decomposing the convolution operator. The other is that the input parameter information of the convolution operator does not satisfy the constraints, but satisfies the first condition determined based on prior knowledge. In this case, the convolution operator is also not decomposed. This is because when the input parameter information of the convolution operator satisfies the first condition, it indicates that the convolution operator may not be decomposed, or even if it is decomposed, the efficiency of implementing the decomposed operator is low.

[0089] If the input parameters of the convolution operator do not satisfy either the first condition determined based on prior indications or the constraints of the hardware convolution instructions directly supported by the first hardware, the convolution operator can be decomposed into multiple operators. At least one of these multiple operators can be executed by hardware instructions directly supported by the first hardware. For example, these multiple operators can be a combination of one or more operators such as image transpilation operators, matrix multiplication operators, dimensionality transformation operators, and addition operators; they can also be one or more other convolution operators corresponding to the hardware convolution instructions of the first hardware; or they can be a combination of one or more other convolution operators corresponding to the hardware convolution instructions of the first hardware and non-convolution operators.

[0090] For example, the decomposition of a convolution operator can be registered through a first decorator in the compiler. This first decorator can be a decorator in the compiler used to decompose a high-level operator into a combination of more basic operators. For example, in the case of an Inductor compiler, the first decorator could be `register_decomposition` in the Inductor compiler. This first decorator can also be a decorator in other compilers that has decomposition operator functionality; the embodiments of this disclosure are not limited thereto.

[0091] In at least one embodiment of this disclosure, after determining that the convolution operator will not be decomposed, subsequent operations can be performed using the compiler's built-in compilation optimization process. The compiler's built-in optimization process automatically places multiple implementation schemes corresponding to the two methods—calling hardware convolution instructions directly supported by the first hardware and attempting to generate a kernel—into the search space, tests and searches these multiple implementation schemes, and thus determines the implementation scheme with the best performance. The ultimately determined implementation scheme with the best performance may be to implement the convolution operator by calling hardware convolution instructions directly supported by the first hardware, or it may be to implement the convolution operator using a generated kernel. The inventors of this disclosure have found that placing multiple implementation schemes corresponding to both methods into the search space results in a large number of implementation schemes, each of which needs to be tested and executed once, leading to a significant overhead in model compilation.

[0092] In at least one embodiment of this disclosure, after determining that the convolution operator will not be decomposed, the reduction function of the convolution operator can be removed from the registration list, and a new reduction function can be re-registered in the registration list. The convolution operator can then be processed according to the new reduction function, thereby replacing the compiler's built-in compilation optimization process for the convolution operator with a new compilation optimization process. In the new compilation optimization process, hardware convolution instructions directly supported by the first hardware can be preferentially called to implement the convolution operator. This eliminates the need to test the implementation scheme of calling hardware convolution instructions directly supported by the first hardware in the search space, thereby reducing the size of the search space, reducing the compilation overhead of the model, and making better use of the computing resources of the first hardware.

[0093] For example, a lowering of the convolution operator can be registered through a compiler-defined second decorator. This second decorator can be a decorator in the compiler used to convert a high-level operator into a low-level operator. For example, in the case of an Inductor compiler, the second decorator could be `register_lowering` in the Inductor compiler. The second decorator can also be a decorator with down-assembly capabilities in other compilers; the embodiments of this disclosure are not limited thereto.

[0094] For example, compilers typically have a default reduction function for convolution operators. In at least one embodiment of this disclosure, this default reduction function for convolution operators can be removed from the compiler's registry and a new reduction function can be re-registered in the registry. This new reduction function defines a new processing flow for the convolution operator. Based on this reduction function, the compiler can first determine whether it can call an operator library to implement the convolution operator. If it cannot call an operator library to implement the convolution operator, it will attempt to generate a kernel to implement the convolution operator.

[0095] For example, in at least one embodiment of this disclosure, the convolution operator is processed according to a new reduction function. A specific example may include: in response to the input parameter information of the convolution operator satisfying the constraints of the hardware convolution instruction, calling a hardware convolution instruction directly supported by the first hardware to implement the convolution operator; in response to the input parameter information of the convolution operator not satisfying the constraints, attempting to generate at least one kernel.

[0096] For example, if the input parameter information of the convolution operator meets the constraints of the hardware convolution instruction, the compiler does not need to call the implementation scheme of the convolution operator from the operator library and put it into the search space for automatic tuning. Instead, it can directly compile the convolution operator into the underlying machine instruction, that is, the hardware convolution instruction directly supported by the first hardware corresponding to the convolution operator. The first hardware executes the hardware convolution instruction, so that the dedicated computing unit of the first hardware implements the convolution calculation operation corresponding to the convolution operator, thereby reducing the size of the search space, reducing the compilation overhead of the model, and making better use of the computing resources of the dedicated computing unit of the first hardware.

[0097] For example, if the input parameters of the convolution operator do not meet the constraints of the hardware convolution instruction, the compiler can directly attempt to generate a kernel. For instance, the compiler can use tuning tests to search for kernel configuration parameters in the kernel template to try and generate at least one kernel. By including only the kernel configuration parameters in the search space, the size of the search space can be reduced, thus decreasing the model's compilation time.

[0098] For example, developers can first set several possible kernel configuration parameters, determine which key-value parameters will affect the configuration selection, and then use decorators to mark these settings before the kernel function definition. When the kernel function is called, the automatic parameter tuning tool will compile and run these candidate configuration parameters one by one, test their respective execution times, and then select the configuration parameter with the best performance and pass it to the subsequent execution stage to complete the compilation of the kernel function.

[0099] For example, optimal configuration parameters can be stored in the cache of the auto-tuning instance, establishing a mapping between key-value parameters and configuration parameters. If the kernel function is called again with the same key-value parameters, the auto-tuning tool can directly find the matching configuration parameters from the cache and pass them over, eliminating the need for retuning.

[0100] For example, in at least one embodiment of this disclosure, in response to determining that multiple kernels have been generated, a target kernel with superior performance compared to other kernels is determined from the multiple kernels to implement the convolution operator; or in response to determining that only one kernel has been generated, the generated kernel is used to implement the convolution operator.

[0101] For example, if multiple kernels are generated, their corresponding implementation schemes can be placed in a search space for testing and searching to determine the kernel with the best performance to implement the convolution operator. If only one kernel is generated, that kernel can be used directly to implement the convolution operator. Since using a kernel to implement the convolution operator is more efficient than using the native operator, the performance of the convolution operator can still be guaranteed to be good.

[0102] For example, in at least one embodiment of this disclosure, in response to a kernel generation failure, the convolution operator is implemented using native operators via second hardware, such as a central processing unit (CPU). That is, if there are scenarios where neither the hardware vendor's operator library nor the compiler kernel supports the convolution operator, the compiler's native operators can be used to implement the convolution operator, for example, by using a CPU for computation, thereby ensuring compatibility.

[0103] The convolution operator optimization implementation method provided in at least one embodiment of this disclosure can be used in a hardware platform including a first hardware and a second hardware, wherein the first hardware can be a graphics processing unit (GPU) and the second hardware can be a central processing unit (CPU); however, the embodiments of this disclosure are not limited thereto. One or more steps in the convolution operator optimization implementation method provided in at least one embodiment of this disclosure can be adjusted according to the actual application scenario. For example, one or more steps can be omitted depending on the hardware platform of the application or the compiler.

[0104] Figure 2B A flowchart illustrating an exemplary convolution operator optimization implementation method provided in at least one embodiment of this disclosure is shown below. Figure 2B Taking the Inductor compiler, the GPU as the first hardware, and the CPU as the second hardware as an example, this paper details an exemplary convolution operator optimization implementation method provided by at least one embodiment of the present disclosure.

[0105] like Figure 2B As shown, the convolution operator optimization process provided in at least one embodiment of this disclosure may include two parts: decomposition of the convolution operator and reduction of the convolution operator.

[0106] For example, the decomposition of the `aten.convolution` operator (i.e., the convolution operator) can be registered using the `register_decomposition` decorator in the Inductor compiler. The decomposition function processes the input parameters of the `aten.convolution` operator to obtain information such as the shape of the input tensor, the shape of the kernel tensor, stride, padding, and grouping; and determines whether to decompose the `aten.convolution` operator based on this information and the constraints of the hardware convolution instructions directly supported by the GPU.

[0107] If the `aten.convolution` operator is not a convolution already supported by the GPU's hardware convolution instructions, nor is it a convolution operator with a kernel size greater than 7×7, a depthwise separable convolution operator, a transposed convolution operator, or a convolution operator with a kernel size equal to 1×1, then the `aten.convolution` operator should be decomposed. For example, the `aten.convolution` operator can be decomposed into the `Im2Col` operator and the `Matmul` operator, thereby making full use of the GPU's tensor kernel computational resources.

[0108] For example, if the aten.convolution operator is a convolution that is already supported by the GPU's hardware convolution instructions, then the aten.convolution operator will not be decomposed; if the aten.convolution operator is a convolution with a kernel size greater than 7×7, a depthwise separable convolution operator, a transposed convolution operator, or a convolution with a kernel size equal to 1×1, then the aten.convolution operator will not be decomposed.

[0109] For example, if the aten.convolution operator is not decomposed, aten.convolution can be called directly, the result of the call can be returned, and then the process of reducing the convolution operator can be started.

[0110] For example, the lowering of the aten.convolution operator can be registered using the decorator register_lowering, which maps the higher-order operator aten.convolution to a lower-level implementation.

[0111] For example, you can first remove the default aten.convolution operator's descent function from the registration list in Torch Inductor, and then re-register a new descent function to implement the subsequent functionality.

[0112] For example, it can be determined whether the aten.convolution operator satisfies the constraints of the GPU's hardware convolution instructions, thereby determining whether the GPU's hardware convolution instructions support the aten.convolution operator. If the aten.convolution operator satisfies the constraints of the GPU's hardware convolution instructions, then the hardware vendor's operator library can be used to implement the aten.convolution operator directly. That is, the compiler can directly compile the aten.convolution operator into hardware convolution instructions that the GPU can execute directly. For example, the convolution calculation operation corresponding to the aten.convolution operator can be implemented by the GPU's tensor kernel.

[0113] For example, if the aten.convolution operator does not meet the constraints of the GPU's hardware convolution instructions, it is determined whether a Triton Kernel can be automatically generated. For example, an autotuner can be used to automatically search for the optimal configuration parameters of the Triton Kernel.

[0114] For example, if multiple Triton Kernels are successfully generated, the Triton Kernel with the best performance is selected from among them to implement the convolution operation corresponding to the aten.convolution operator; if only one Triton Kernel is successfully generated, that Triton Kernel is used directly to implement the aten.convolution operator, for example, by using the GPU's vector kernel to implement the calculation corresponding to the aten.convolution operator.

[0115] For example, if the Triton Kernel cannot be generated, a native PyTorch operator implementation is used, such as using the CPU for computation, to ensure compatibility.

[0116] The convolution operator optimization implementation method provided in at least one embodiment of this disclosure can determine whether to decompose the convolution operator after obtaining the input parameter information of the convolution operator and the constraints of the hardware convolution instructions directly supported by the first hardware. This allows for the early determination of whether the convolution operator is a convolution operator that can be directly executed by the hardware convolution instructions of the first hardware, and provides a new implementation scheme for convolution operators that cannot be directly executed by the hardware convolution instructions of the first hardware.

[0117] Furthermore, at least one embodiment of this disclosure can determine whether to decompose the convolution operator by judging whether the input parameter information of the convolution operator satisfies the first condition determined based on prior knowledge and the constraint condition of the hardware convolution instruction. This utilizes prior knowledge to improve the success rate of decomposing the convolution operator, avoids decomposing the convolution operator into a poor-performing operator, and thus improves the processing efficiency of the compiler.

[0118] At least one embodiment of this disclosure can also preferentially call hardware convolution instructions directly supported by the first hardware to implement the convolution operator, thereby eliminating the need to put the implementation scheme of calling hardware convolution instructions directly supported by the first hardware to implement the convolution operator into the search space for testing. Only the kernel configuration parameters are put into the search space, which can reduce the size of the search space, reduce the compilation overhead of the model, and make better use of the computing resources of the first hardware.

[0119] At least one embodiment of this disclosure also provides a non-transitory computer-readable storage medium. Figure 3 A schematic diagram of a non-transitory computer-readable storage medium provided for at least one embodiment of this disclosure. For example, such as Figure 3 As shown, one or more computer-executable instructions 301 can be stored non-transitorily on storage medium 300. For example, when the computer-executable instructions 301 are executed by a processor, one or more steps in the convolution operator optimization implementation method described above can be performed. The technical effects of this non-transitory storage medium are the same as those of the convolution operator optimization implementation method described above, and will not be repeated here.

[0120] For example, the aforementioned non-transiently readable storage medium is implemented as a memory, such as volatile memory and / or non-volatile memory. In the above embodiments, the memory can be volatile memory, such as random access memory (RAM) and / or cache. Non-volatile memory can include, for example, read-only memory (ROM), hard disk, erasable programmable read-only memory (EPROM), portable compact disc read-only memory (CD-ROM), USB memory, flash memory, etc. The memory can also store various applications (code, instructions) and data, as well as various data used and / or generated by the applications.

[0121] The aforementioned computer-readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to perform the methods disclosed herein.

[0122] Computer program code for performing the methods of this disclosure can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0123] At least one embodiment of this disclosure also provides an electronic device, including a memory and a processor. The memory stores computer-executable instructions non-transitoryly, and the processor is configured to run the computer-executable instructions, wherein the computer-executable instructions are executed by the processor to implement the convolution operator optimization implementation method as described in any of the above embodiments.

[0124] For the specific implementation details and related explanations of each step of the convolution operator optimization implementation method, please refer to the above-mentioned embodiment of the convolution operator optimization implementation method. Repeated details will not be repeated here.

[0125] For example, the processor and memory can communicate with each other directly or indirectly. For example, the processor and memory can communicate via a network. The network can include wireless networks, wired networks, and / or any combination of wireless and wired networks. The processor and memory can also communicate with each other via a system bus, which is not limited in this disclosure. Figure 4 This is a schematic diagram of the structure of an electronic device provided in at least one embodiment of the present disclosure. The electronic device in at least one embodiment of the present disclosure may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle terminals (e.g., vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 4 The illustrated electronic device 400 is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0126] For example, such as Figure 4As shown, in some examples, electronic device 400 includes a processing unit (e.g., central processing unit, graphics processor, etc.) 401, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 402 or a program loaded from storage device 408 into random access memory (RAM) 403. RAM 403 also stores various programs and data required for the operation of the computer system. Processing unit 401, ROM 402, and RAM 403 are connected via bus 404. Input / output (I / O) interface 405 is also connected to bus 404.

[0127] For example, the following components can be connected to I / O interface 405: input devices 406 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 407 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 408 including, for example, magnetic tapes, hard disks, etc.; and communication devices 409, such as network interface cards like LAN cards and modems, etc. Communication device 409 allows electronic device 400 to communicate wirelessly or wiredly with other devices to exchange data and perform communication processing via networks such as the Internet. Drive 410 is also connected to I / O interface 405 as needed. Removable storage media 411, such as disks, optical disks, magneto-optical disks, semiconductor memories, etc., are installed on drive 410 as needed so that computer programs read from them can be installed into storage device 408 as needed. Although Figure 4 An electronic device 400 including various devices is shown, but it should be understood that it is not required to implement or include all of the devices shown, and more or fewer devices may be implemented or included alternatively.

[0128] For example, the electronic device 400 may further include a peripheral interface (not shown in the figure). This peripheral interface can be various types of interfaces, such as a USB interface, a Lightning interface, etc. The communication device 409 can communicate wirelessly with a network and other devices, such as the Internet, an intranet, and / or a wireless network such as a cellular telephone network, a wireless local area network (LAN), and / or a metropolitan area network (MAN). Wireless communication can use any of a variety of communication standards, protocols, and technologies, including but not limited to Global System for Mobile Communications (GSM), Enhanced Data GSM Environment (EDGE), Wideband Code Division Multiple Access (W-CDMA), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Bluetooth, Wi-Fi (e.g., based on IEEE 802.11a, IEEE 802.11b, IEEE 802.11g, and / or IEEE 802.11n standards), Voice over Internet Protocol (VoIP), Wi-MAX, protocols for email, instant messaging, and / or Short Message Service (SMS), or any other suitable communication protocol.

[0129] For example, the electronic device 400 can be any device such as a mobile phone, tablet computer, laptop computer, e-book, game console, television, digital photo frame, or navigator, or any combination of data processing devices and hardware. The embodiments disclosed herein do not limit this.

[0130] The technical effect of this electronic device is the same as that of the above-mentioned convolution operator optimization implementation method, and will not be repeated here.

[0131] The convolution operator optimization implementation method provided in at least one embodiment of this disclosure can be applied to different devices or scenarios, such as... Figure 4 This applies to the electronic device 400 shown, or other scenarios involving convolution operations such as CPU, GPU, High Performance Computing (HPC), and Artificial Intelligence (AI). Of course, this disclosure is not limited to this; any scenario, device, or apparatus involving convolution operations can utilize the optimized implementation method of the convolution operator provided in at least one embodiment of this disclosure.

[0132] For example, the convolution operator optimization implementation method provided in at least one embodiment of this disclosure can be applied in fields such as speech processing, image processing, text processing, and video processing.

[0133] For example, in the field of speech processing, convolution operators can be the convolutional computations involved in tasks such as feature extraction, speech enhancement, and speech recognition.

[0134] For example, in the field of image processing, convolution operators can be the convolutional computations involved in tasks such as image recognition, feature extraction, image segmentation, object detection, image classification, and scene reconstruction.

[0135] For example, in the field of text processing, convolution operators can be the convolutional computations involved in tasks such as text classification, sentiment analysis, and text generation.

[0136] Of course, this disclosure is not limited to this. For other application scenarios or fields, as long as convolution calculation is required, the convolution operator optimization implementation method of at least one embodiment of this disclosure can be applied, which will not be described in detail here.

[0137] Although the present disclosure has been described in detail above with general descriptions and specific embodiments, modifications or improvements can be made to the embodiments of the present disclosure, which will be obvious to those skilled in the art. Therefore, all such modifications or improvements made without departing from the spirit of the present disclosure are within the scope of protection claimed by the present disclosure.

[0138] In addition to the exemplary content described above, the following points should be noted regarding this disclosure:

[0139] (1) The accompanying drawings of the embodiments of this disclosure only involve the structures involved in the embodiments of this disclosure. Other structures can be referred to the general design.

[0140] (2) For clarity, the thickness of layers or regions in the drawings used to describe embodiments of the present disclosure is enlarged or reduced, i.e., these drawings are not drawn to actual scale.

[0141] (3) Where there is no conflict, the embodiments of this disclosure and the features in the embodiments can be combined with each other to obtain new embodiments.

[0142] The above description is merely a specific embodiment of this disclosure, but the scope of protection of this disclosure is not limited thereto. The scope of protection of this disclosure should be determined by the scope of protection of the claims.

Claims

1. A method for optimized implementation of a convolution operator, characterized in that, The method comprises: obtaining input parameter information of a convolution operator to be compiled and constraint conditions of a hardware convolution instruction directly supported by first hardware, wherein the constraint conditions comprise constraints on input parameters of the hardware convolution instruction when the hardware convolution instruction directly supported by the first hardware is used to implement the convolution operator, and the first hardware comprises hardware designed with the hardware convolution instruction for a preset convolution scenario; determining whether to decompose the convolution operator according to the input parameter information and the constraint conditions; in response to a determination to decompose the convolution operator, decomposing the convolution operator into a plurality of operators; compiling the plurality of operators into machine instructions to implement the convolution operator by executing the machine instructions of the plurality of operators, wherein at least one of the plurality of operators is executed by a hardware instruction directly supported by the first hardware, and the hardware instruction comprises a hardware instruction directly supported by the first hardware and used to implement a function other than the convolution operator.

2. The method of claim 1, wherein, Further comprising: obtaining a first condition determined based on prior knowledge; wherein the determining whether to decompose the convolution operator according to the input parameter information and the constraint conditions comprises: determining whether to decompose the convolution operator according to the input parameter information, the first condition, and the constraint conditions, wherein the first condition comprises at least one of the following: a kernel size of the convolution operator is greater than 7x7, the convolution operator is a depth separable convolution operator, the convolution operator is a transpose convolution operator, or the kernel size of the convolution operator is equal to 1x1.

3. The method of claim 2, wherein, The determining whether to decompose the convolution operator according to the input parameter information, the first condition, and the constraint conditions comprises: in response to the input parameter information satisfying the constraint conditions, or the input parameter information not satisfying the constraint conditions but satisfying the first condition, determining not to decompose the convolution operator, in response to the input parameter information not satisfying the first condition and not satisfying the constraint conditions, determining to decompose the convolution operator.

4. The method according to claim 2 or 3, characterized in that, Further comprising: in response to a determination not to decompose the convolution operator, deleting a reduction function of the convolution operator from a registration list; re-registering a new reduction function in the registration list and processing the convolution operator according to the new reduction function.

5. The method of claim 4, wherein, The processing the convolution operator according to the new reduction function comprises: in response to the input parameter information satisfying the constraint conditions, invoking the hardware convolution instruction directly supported by the first hardware to implement the convolution operator.

6. The method of claim 4, wherein, The processing the convolution operator according to the new reduction function comprises: in response to the input parameter information not satisfying the constraint conditions, searching for a kernel configuration parameter through an optimization strategy and determining whether at least one kernel is generated; in response to a determination that a plurality of kernels are generated, determining a target kernel with better performance than other kernels from the plurality of kernels to implement the convolution operator; or in response to a determination that only one kernel is generated, using the one kernel to implement the convolution operator.

7. The method of claim 6, wherein, Further comprising: In response to generating a kernel failure, the convolution operator is implemented by the second hardware using native operators.

8. The method of claim 7, wherein, The first hardware comprises a graphics processing unit and the second hardware comprises a central processing unit.

9. The method according to any one of claims 1 to 3, characterized in that, The plurality of operators comprises an image transpose operator and a matrix multiplication operator, wherein at least the matrix multiplication operator is directly supported by the hardware instructions of the first hardware.

10. The method according to any one of claims 1 to 3, characterized in that, The input parameter information comprises a shape of an input tensor, a shape of a convolution kernel tensor, a stride, padding, and grouping.

11. An electronic device, comprising: Comprise: a memory, which non-transitorily stores computer executable instructions; a processor, which is configured to run the computer executable instructions, wherein the computer executable instructions, when run by the processor, implement the convolution operator optimization implementation method according to any one of claims 1-10.

12. A non-transitory computer-readable storage medium, comprising: The non-transitory computer readable storage medium stores computer executable instructions, The computer executable instructions, when executed by the processor, implement the convolution operator optimization implementation method according to any one of claims 1-10.

Images