Efficient methods and devices for accelerating library calls in the PyTorch framework
By introducing the singleton pattern and optimizing handle binding, memory space, and convolution algorithm selection in the PyTorch framework, the problem of efficient PyTorch framework calls on OpenCL devices is solved, significantly improving computing performance.
Patent Information
- Application Number
- CN202311650471.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-12-05
- Publication Date
- 2026-11-17
- Estimated Expiration
- 2043-12-05
AI Technical Summary
The existing PyTorch framework has low efficiency in calling accelerated libraries on OpenCL computing devices, making it difficult to efficiently transcode to domestic high-performance heterogeneous computing platforms.
The singleton pattern is used to optimize handle binding, and the Cache class is used to automatically calculate the video memory space and select the convolution algorithm, reducing redundant calculations and improving call efficiency.
The PyTorch framework's efficient acceleration library calls were implemented on OpenCL-enabled computing devices, improving computing performance, especially with a 3.07x performance improvement when training deep neural networks.
Smart Images

Figure CN117608797B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of efficient library invocation technology, and in particular to a method and apparatus for efficient invocation of PyTorch framework acceleration libraries on OpenCL-enabled computing devices. Background Technology
[0002] With the continuous upgrading of hardware devices, deep learning has developed rapidly, playing a positive role in promoting the global technological revolution and industrial transformation. The application of deep learning relies on the support of deep learning frameworks; therefore, it can be said that deep learning frameworks are the cornerstone of industrial intelligence. Currently, mainstream deep learning frameworks such as PyTorch (a Python-based deep learning framework) only support CUDA (a general-purpose parallel computing architecture that enables GPUs to solve complex computational problems) and CPUs (Central Processing Units), exhibiting poor compatibility with other computing devices that support the OpenCL standard (a framework for writing programs for heterogeneous computing platforms, which can consist of CPUs, GPUs, or other types of processors). The PyTorch framework, benefiting from its flexible, lightweight programming model and rich toolkit, has been widely adopted. To efficiently apply deep learning frameworks and train deep neural networks on domestically produced high-performance heterogeneous computing platforms, transcoding the PyTorch framework to OpenCL-compatible computing devices (also known as OpenCL platforms) is crucial.
[0003] In the CUDA implementation of convolution functions in the PyTorch framework, some operators call cuDNN (a GPU-accelerated library for deep neural networks) for implementation. Existing methods already include implementations using the OpenCL acceleration library. That is, there are existing methods for implementing the PyTorch framework's OpenCL kernel function to call the acceleration library on OpenCL-enabled computing devices. These methods include binding handles, calculating and allocating input and output memory space, checking whether the input data format is consistent with the model input, copying data to memory space, setting specific library configuration parameters, loading data and executing on the GPU, obtaining the calculation results, converting the calculation result data as needed, and releasing OpenCL resources. However, how to efficiently call the acceleration library that supports OpenCL in the PyTorch framework still needs further research.
[0004] Therefore, there is an urgent need for an efficient technology to call PyTorch framework acceleration libraries on OpenCL-enabled computing devices. This is of great significance for transcoding the PyTorch framework to OpenCL-enabled computing devices. Summary of the Invention
[0005] The purpose of this invention is to provide an efficient method and apparatus for calling acceleration libraries in the PyTorch framework. It optimizes existing acceleration library calling methods and enables efficient calling of acceleration libraries in the PyTorch framework on computing devices that support OpenCL, which is beneficial for transcoding the PyTorch framework to computing devices that support OpenCL.
[0006] To achieve the above objectives, the present invention provides the following solution:
[0007] An efficient method for calling acceleration libraries in the PyTorch framework, the efficient calling method includes:
[0008] When binding a handle, the system determines whether the handle has already been bound by using a handle binding class that is pre-created based on the singleton pattern. If it has, the system outputs the handle instance generated when binding the handle using the handle binding class. If it has not, the system binds the handle using the handle binding class, generates and stores the handle instance, and outputs the handle instance.
[0009] In calculating and allocating input and output video memory space, it is determined whether there is a convolution kernel parameter class in the first cache class that is the same as the convolution kernel parameter class to be calculated; if so, the video memory space corresponding to the convolution kernel parameter class to be calculated is output through the first cache class; if not, the video memory space of the convolution kernel parameter class to be calculated is calculated, and the convolution kernel parameter class to be calculated and the video memory space corresponding to the convolution kernel parameter class to be calculated are stored in the first cache class; the convolution kernel parameter class includes convolution kernel parameters;
[0010] During GPU data loading and execution, it is determined whether there is a convolution kernel parameter class in the second cache class that is the same as the convolution kernel parameter class to be selected. If so, the convolution algorithm corresponding to the convolution kernel parameter class to be selected is output through the second cache class. If not, the convolution algorithm corresponding to the convolution kernel parameter class to be selected is selected, and the convolution kernel parameter class to be selected and the convolution algorithm corresponding to the convolution kernel parameter class to be selected are stored in the second cache class.
[0011] In some embodiments, the members of the handle binding class include handle_record, Accelerator_Library_Manager, and get_handle; handle_record is a static private object of the handle binding class, used to store handle instances; Accelerator_Library_Manager is a private constructor of the handle binding class, used to bind handles and generate handle instances; get_handle is used to obtain handle instances.
[0012] In some embodiments, determining whether a handle has been bound using a handle binding class pre-created based on the singleton pattern; if yes, outputting the handle instance generated during handle binding using the handle binding class; if no, binding the handle using the handle binding class, generating and storing the handle instance, and outputting the handle instance, specifically includes:
[0013] The handle instance generated when binding the handle is obtained by using get_handle in the handle binding class that was created in advance based on the singleton pattern, in order to determine whether the handle has been bound;
[0014] If the handle instance can be obtained, the handle has been bound, and the handle instance is output through get_handle in the handle binding class;
[0015] If the handle instance cannot be obtained, the handle is not bound. The handle is bound through Accelerator_Library_Manager in the handle binding class, a handle instance is generated, the handle instance is stored through handle_record in the handle binding class, and the handle instance is output through get_handle in the handle binding class.
[0016] In some embodiments, members of both the first cache class and the second cache class include std::unordered_map and a find method; std::unordered_map includes a hash function used to calculate the hash value of the convolution kernel parameter class; the find method is used to find and output the value corresponding to the convolution kernel parameter class; in the first cache class, the value is video memory space; in the second cache class, the value is the convolution algorithm.
[0017] In some embodiments, determining whether there is a convolution kernel parameter class in the first cache class that is the same as the convolution kernel parameter class to be calculated specifically includes: using the find method in the first cache class to call a hash function to calculate a first hash value of the convolution kernel parameter class to be calculated, and comparing the first hash value with the second hash value of each convolution kernel parameter class in the first cache class. If any second hash value is the same as the first hash value, it is determined that there is a convolution kernel parameter class in the first cache class that is the same as the convolution kernel parameter class to be calculated.
[0018] The step of determining whether there is a convolution kernel parameter class in the second cache class that is the same as the convolution kernel parameter class to be selected specifically includes: using the find method in the second cache class to call a hash function, calculating the third hash value of the convolution kernel parameter class to be selected, and comparing the third hash value with the fourth hash value of each convolution kernel parameter class in the second cache class. If any of the fourth hash values is the same as the third hash value, then it is determined that there is a convolution kernel parameter class in the second cache class that is the same as the convolution kernel parameter class to be selected.
[0019] In some embodiments, the step of outputting the video memory space corresponding to the convolution kernel parameter class to be calculated through the first cache class specifically includes: outputting the video memory space corresponding to the same convolution kernel parameter class as the one to be calculated through the find method in the first cache class.
[0020] The step of outputting the convolution algorithm corresponding to the convolution kernel parameter class to be selected through the second cache class specifically includes: outputting the convolution algorithm corresponding to the same convolution kernel parameter class as the one to be selected through the find method in the second cache class.
[0021] In some embodiments, members of both the first cache class and the second cache class include an insert method; the insert method is used to store the convolution kernel parameter class and the value corresponding to the convolution kernel parameter class; in the first cache class, the value is video memory space; in the second cache class, the value is the convolution algorithm.
[0022] In some embodiments, storing the convolution kernel parameter class to be calculated and the corresponding video memory space to be calculated into the first cache class specifically includes: storing the convolution kernel parameter class to be calculated and the corresponding video memory space to be calculated into the first cache class through the insert method in the first cache class.
[0023] The step of storing the selected convolution kernel parameter class and the corresponding convolution algorithm into the second cache class specifically includes: storing the selected convolution kernel parameter class and the corresponding convolution algorithm into the second cache class through the insert method in the second cache class.
[0024] An efficient calling device for accelerating libraries in the PyTorch framework, the efficient calling device comprising:
[0025] The handle binding optimization module is used to determine whether a handle has been bound beforehand by using a handle binding class created in advance based on the singleton pattern. If so, the handle instance generated when binding the handle is output through the handle binding class. If not, the handle is bound through the handle binding class, a handle instance is generated and stored, and the handle instance is output.
[0026] The video memory space calculation optimization module is used to determine whether a convolution kernel parameter class with the same name as the convolution kernel parameter class to be calculated exists in the first cache class during the calculation and allocation of input and output video memory space; if so, the video memory space corresponding to the convolution kernel parameter class to be calculated is output through the first cache class; if not, the video memory space of the convolution kernel parameter class to be calculated is calculated, and the convolution kernel parameter class to be calculated and the video memory space corresponding to the convolution kernel parameter class to be calculated are stored in the first cache class; the convolution kernel parameter class includes convolution kernel parameters;
[0027] The convolution algorithm selection and optimization module is used to determine whether there is a convolution kernel parameter class in the second cache class that is the same as the convolution kernel parameter class to be selected during GPU data loading and execution; if so, the convolution algorithm corresponding to the convolution kernel parameter class to be selected is output through the second cache class; if not, the convolution algorithm corresponding to the convolution kernel parameter class to be selected is selected, and the convolution kernel parameter class to be selected and the convolution algorithm corresponding to the convolution kernel parameter class to be selected are stored in the second cache class.
[0028] According to specific embodiments provided by the present invention, the present invention discloses the following technical effects:
[0029] This invention provides an efficient method and apparatus for calling acceleration libraries in the PyTorch framework. When binding a handle, the method determines whether a handle has already been bound using a handle binding class pre-created based on the singleton pattern. If so, the handle instance generated during binding is output through the handle binding class; otherwise, the handle is bound through the handle binding class, a handle instance is generated and stored, and then output. During the calculation and allocation of input / output memory space, the method determines whether a convolution kernel parameter class identical to the one to be calculated exists in the first cache class. If so, the memory space corresponding to the convolution kernel parameter class to be calculated is output through the first cache class; otherwise... The process involves calculating the GPU memory space for the convolution kernel parameter class to be calculated and storing the convolution kernel parameter class and its corresponding GPU memory space in a first cache class. During GPU data loading and execution, it is determined whether a convolution kernel parameter class identical to the one to be selected exists in a second cache class. If so, the convolution algorithm corresponding to the selected convolution kernel parameter class is output through the second cache class. If not, the convolution algorithm corresponding to the selected convolution kernel parameter class is selected, and both the selected convolution kernel parameter class and its corresponding convolution algorithm are stored in the second cache class. This invention optimizes existing acceleration library calling methods, enabling efficient calling of acceleration libraries in the PyTorch framework on OpenCL-supported computing devices, thus facilitating the transcoding of the PyTorch framework to OpenCL-supported computing devices. Attached Figure Description
[0030] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0031] Figure 1 This is a schematic diagram of the conventional calling process provided in Embodiment 1 of the present invention;
[0032] Figure 2 This is a flowchart of the efficient calling method provided in Embodiment 1 of the present invention;
[0033] Figure 3 This is a schematic diagram of the class of the handle binding class provided in Embodiment 1 of the present invention;
[0034] Figure 4 This is a class diagram of the Cache class provided in Embodiment 1 of the present invention;
[0035] Figure 5This is a block diagram of the efficient calling device provided in Embodiment 2 of the present invention. Detailed Implementation
[0036] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0037] The purpose of this invention is to provide an efficient method and apparatus for calling acceleration libraries in the PyTorch framework. It optimizes existing acceleration library calling methods and enables efficient calling of acceleration libraries in the PyTorch framework on computing devices that support OpenCL, which is beneficial for supporting the PyTorch framework on domestic heterogeneous platforms.
[0038] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.
[0039] Example 1:
[0040] like Figure 1 As shown, existing methods for implementing OpenCL kernel function call acceleration libraries in the PyTorch framework on OpenCL-enabled computing devices include binding handles, calculating and allocating input and output memory space, checking whether the input data format is consistent with the model input, copying the data to the memory space, setting specific library configuration parameters, loading data and executing it on the GPU, obtaining the calculation results, converting the calculation result data as needed, and releasing OpenCL resources. However, how to efficiently call OpenCL-enabled acceleration libraries in the PyTorch framework still needs further research.
[0041] In view of this, this embodiment analyzes Figure 1 The diagram illustrates the calling flow of the deep learning acceleration library in the OpenCL PyTorch project, and the following two optimization schemes are designed for processes such as handle binding, memory space calculation, and convolution algorithm selection:
[0042] (1) Use the singleton pattern to initialize the acceleration library to avoid the acceleration library initialization operation being executed frequently and repeatedly.
[0043] (2) The memory space is automatically calculated based on the kernel parameters and the optimal convolution algorithm is automatically selected. The calculation results of the same parameter combination are stored in the cache of the computing device.
[0044] More specifically, this embodiment describes... Figure 1Steps 1, 2, and 6 have been optimized; the remaining steps are the same as... Figure 1 The traditional calling process shown is the same.
[0045] (1) Optimization for step 1:
[0046] When binding handles, the traditional calling process requires binding the handle once every time the accelerator library function is called. This embodiment optimizes this process by introducing a singleton pattern. First, it checks whether there is a bound handle in the previous operation of the process. If not, it binds the handle to the device. If it exists, it directly obtains the previously bound handle. The singleton pattern design avoids the need to repeatedly bind the device and handle every time the accelerator library operator is executed.
[0047] (2) Optimization of step 2:
[0048] When calculating and allocating the video memory space for input, output, and intermediate calculation results, the traditional calling process calculates the video memory space every time. This embodiment optimizes this process by checking whether the same scale of calculation has been performed in the previous calculations of the process. If not, the video memory space for input, output, and intermediate calculation results is calculated and allocated. If it exists, the previously calculated video memory space is directly obtained from the cache without recalculation.
[0049] (3) Optimization of step 6:
[0050] In the GPU data loading and execution process, the traditional calling process for calling the convolution operator of the acceleration library is as follows: the fastest convolution acceleration method (i.e., convolution algorithm) is evaluated based on the input size, and the GPU executes the selected optimal algorithm in parallel. However, the convolution algorithm must be selected each time. This embodiment optimizes this process by checking whether the same size operation has been performed in the previous operation of the process. If not, the fastest convolution acceleration method is evaluated based on the input size. If it exists, the previously evaluated result is directly retrieved from the cache without re-evaluation.
[0051] Since the convolution operator may be executed tens of thousands of times during a single model training, the efficiency improvement brought about by optimizing redundant computation in (2) and (3) is very considerable.
[0052] Based on the above, this embodiment optimizes the difficulties encountered by the core of the PyTorch framework—deep learning operators—when calling acceleration libraries. Specifically, it optimizes the acceleration library calls of OpenCL kernel functions in the PyTorch framework and proposes an efficient process for calling acceleration libraries by OpenCL kernel functions of the PyTorch framework on computing devices that support OpenCL. This is of great significance for transcoding the PyTorch framework to computing devices that support OpenCL.
[0053] Specifically, such as Figure 2 As shown in the figure, this embodiment provides an efficient method for calling acceleration libraries in the PyTorch framework. The efficient calling method includes:
[0054] S1: When binding a handle, determine whether the handle has been bound by the handle binding class created in advance based on the singleton pattern; if yes, output the handle instance generated when binding the handle through the handle binding class; if no, bind the handle through the handle binding class, generate and store the handle instance, and output the handle instance.
[0055] Before calling functions and operators in the PyTorch framework's acceleration library, a Handle instance (i.e., a handle instance) needs to be initialized using the corresponding `create` method of the acceleration library to complete the binding between the device and the handle. Device management and computation are then performed through the handle instance. However, in a single model training task, there will be multiple calls to functions and operators in the acceleration library. Traditional calling processes initialize the handle (i.e., bind the handle to the device) every time a function or operator in the acceleration library is called. Since the device remains constant throughout a model training task, the handle instance generated each time the handle is bound is also the same. Therefore, generating the same handle instance for each call to functions and operators in the acceleration library is essentially a redundant operation. This embodiment attempts to initialize only one handle instance at the very beginning of the model training task, so that subsequent calls to functions and operators in the acceleration library can be completed through this handle instance, eliminating the need for repeated handle binding operations.
[0056] Based on this, this embodiment uses the Singleton pattern to encapsulate the binding handle of the acceleration library. This involves encapsulating the objects and methods needed to create a handle instance, such as device information and handle initialization methods, into a single class, generating a handle binding class. The Singleton pattern is a commonly used creational software design pattern. A class created using the Singleton pattern has only one instance in the current process; that is, the Singleton pattern ensures that a class will only have one instance in the entire system. Therefore, the handle binding class is implemented as a Singleton class. When calling functions and operators in the acceleration library, if the handle is not bound, the handle binding process is executed to generate a handle instance; if the handle is already bound, the existing handle instance is returned directly.
[0057] like Figure 3 As shown, this is the design class diagram of the handle binding class in this embodiment. In the class diagram, a "+" before each member indicates that the member is public, and a "-" indicates that the member is private. The members of the handle binding class include:
[0058] (1) device_id: Represents the number of the current computing device.
[0059] (2) cl_commod_queue: Represents the OpenCL context queue. The current computing device number and the OpenCL context queue are both objects required for binding the handle.
[0060] (3) handle_record: is a static private object of the handle binding class, used to store handle instances.
[0061] (4) ~Accelerator_Library_Manager: is the destructor.
[0062] (5) Accelerator_Library_Manager: is the constructor of the handle binding class. The singleton class only provides a private constructor, that is, Accelerator_Library_Manager is the private constructor of the handle binding class, which is used to bind the handle and generate a handle instance.
[0063] (6) get_instance: is a static public function used to create or retrieve its own static private object.
[0064] (7) get_handle: Used to obtain a handle instance.
[0065] (8)Accelerator_Library_Initializer: Used to return whether the instance has been initialized.
[0066] Based on the above members, in S1, it is determined whether the handle has been bound by a handle binding class created in advance based on the singleton pattern. If yes, the handle instance generated when binding the handle is output through the handle binding class. If no, the handle is bound through the handle binding class, a handle instance is generated and stored, and the handle instance is output. This can include: obtaining the handle instance generated when binding the handle through get_handle in the handle binding class created in advance based on the singleton pattern to determine whether the handle has been bound. If the handle instance can be obtained, it means that the handle has been bound, and there is no need to bind the handle again. The handle instance is directly output through get_handle in the handle binding class. If the handle instance cannot be obtained, it means that the handle has not been bound. The handle is bound through Accelerator_Library_Manager in the handle binding class, a handle instance is generated, the handle instance is stored through handle_record in the handle binding class, and the handle instance is output through get_handle in the handle binding class.
[0067] S2: In calculating and allocating input and output video memory space, determine whether there is a convolution kernel parameter class in the first cache class that is the same as the convolution kernel parameter class to be calculated; if yes, output the video memory space corresponding to the convolution kernel parameter class to be calculated through the first cache class; if no, calculate the video memory space of the convolution kernel parameter class to be calculated, and store the convolution kernel parameter class to be calculated and the video memory space corresponding to the convolution kernel parameter class to be calculated into the first cache class; the convolution kernel parameter class includes convolution kernel parameters.
[0068] To invoke convolution operators in the acceleration library, deep learning frameworks call functions within the library to calculate the required parameters. Before invoking a convolution operator, the `workspace_size` parameter needs to be calculated based on kernel parameters such as kernel size, stride, and window size. `workspace_size` refers to the memory space (i.e., GPU memory) allocated by the computing device during computation. In a single deep learning training task (i.e., a single model training task), the model contains only a few fixed combinations of convolution kernel parameters, and these parameters are repeatedly invoked during training. In the traditional calling process, the `workspace_size` calculation is executed every time a convolution operator is invoked. This repeated execution of `workspace_size` calculation within a single model training task results in significant performance overhead, impacting the execution efficiency of OpenCLPyTorch. The convolution kernel parameters are stored in the `ConvolutionParams` class. To reuse the `workspace_size` obtained from the same parameter combination, this embodiment first uses a hash algorithm to map the `ConvolutionParams` class of convolution kernel parameters for a specific parameter combination to a unique value (i.e., a hash value). Then, a high-speed cache class `Cache` is constructed using a structure to store the parameters.<ConvolutionParams,workspace_size> This key-value pair. When calculating workspace_size, first check if there is a mapping in the cache class that is the same as the convolution kernel parameter class to be calculated (i.e., the same convolution kernel parameter class as the convolution kernel parameter class to be calculated). If the mapping exists, it is retrieved and used directly. If the mapping does not exist, the corresponding function is called to calculate the workspace_size corresponding to the convolution kernel parameter class to be calculated, and it and the convolution kernel parameter class ConvolutionParams to be calculated are cached in the form of a key-value pair.
[0069] Using caching appropriately to reduce redundant computations can effectively improve program performance. In the implementation of deep learning frameworks, functions that only depend on the shape of the input tensor of deep learning operators (such as convolution) can be optimized using this method. Combined with the characteristic of deep learning tasks that repeatedly execute operators of specific shapes, computation time can be effectively reduced.
[0070] The class diagram of the Cache class in this embodiment is as follows: Figure 4 As shown, it can associate a ConvolutionParams object with an object of type T. This template class has four members, as follows:
[0071] (1) A std::mutex object is used to synchronize access to the cache and provide thread safety.
[0072] (2) A std::unordered_map object. std::unordered_map is an associative container provided by the C++ standard library. It is a hash table used to store ConvolutionParams objects and objects of type T associated with ConvolutionParams objects. The hashing and comparison of ConvolutionParams objects are implemented through the functions ParamsHash and ParamsEqual.
[0073] std::unordered_map contains the following:
[0074] std::unordered_map<ConvolutionParams,T,ParamsHash,ParamsEqual>
[0075] ParamsHash is a custom hash function used to calculate the hash value of a ConvolutionParams object. Based on the hash value, the ConvolutionParams object and its associated T object are stored as key-value pairs in a hash table. The hash value indicates the specific location within the hash table, allowing the mapping of ConvolutionParams and T objects to their respective storage locations. This hash function is invoked during the `find` and `insert` operations. Specifically, the ParamsHash function converts the ConvolutionParams object into a pointer to a `uint8_t` type and performs bitwise XOR and multiplication operations on each byte at the memory address pointed to by the pointer (i.e., the address where the ConvolutionParams object resides), ultimately yielding the hash value.
[0076] ParamsEqual is a custom equality comparison function used to compare whether two ConvolutionParams objects are equal. It is used to determine whether two keys are equal when handling hash collisions. When comparing two ConvolutionParams objects, the ParamsEqual function first converts each ConvolutionParams object into a pointer to a uint8_t type, and then uses the memcmp function to compare the data stored at the memory addresses pointed to by the two pointers.
[0077] When creating a std::unordered_map, the user needs to define the hash function and comparison function mentioned above.
[0078] (3) find method: a wrapper for map.find, used to check if key-value pairs exist in the cache. Specifically, it calls a hash function to calculate the hash value of the given ConvolutionParams object, and determines whether there is a ConvolutionParams object in the cache with the same hash value as the given ConvolutionParams object based on the comparison of hash values. If it exists, it searches for the corresponding T type object in the cache based on the hash value of the ConvolutionParams object and returns the result through a pointer.
[0079] (4) insert method: Insert the given ConvolutionParams object and T type object into the cache. When inserting, call the hash function to calculate the hash value of the given ConvolutionParams object. If the hash value of the given ConvolutionParams object is equal to the hash value of a certain ConvolutionParams object stored in the cache, then call the comparison function to determine whether the given ConvolutionParams object is equal to a certain ConvolutionParams object. If they are equal, they will not be stored. If they are not equal, they will be stored.
[0080] This template class provides thread-safe insertion and lookup methods, which can be used safely in a multi-threaded environment.
[0081] The first high-speed cache class and Figure 4 The members of the cache class shown are the same, except that for the first cache class, T refers to the video memory space (i.e., workspace_size). Specifically, some members of the first cache class in this embodiment include: std::unordered_map, the find method, and the insert method. std::unordered_map includes a hash function and a comparison function. The hash function is used to calculate the hash value of the convolution kernel parameter class, the comparison function is used to compare the convolution kernel parameter classes, the find method is used to find and output the video memory space corresponding to the convolution kernel parameter class, and the insert method is used to store the convolution kernel parameter class and the video memory space corresponding to the convolution kernel parameter class.
[0082] Based on the members of the first cache class, in S2, determining whether there is a convolution kernel parameter class in the first cache class that is the same as the convolution kernel parameter class to be calculated may include: using the find method in the first cache class to call a hash function, calculating the first hash value of the convolution kernel parameter class to be calculated, and comparing the first hash value with the second hash value of each convolution kernel parameter class in the first cache class. If any second hash value is the same as the first hash value, it is determined that there is a convolution kernel parameter class in the first cache class that is the same as the convolution kernel parameter class to be calculated. The convolution kernel parameter class corresponding to the second hash value that is the same as the first hash value is the convolution kernel parameter class that is the same as the convolution kernel parameter class to be calculated.
[0083] In S2, outputting the video memory space corresponding to the convolution kernel parameter class to be calculated through the first cache class can include: outputting the video memory space corresponding to the convolution kernel parameter class that is the same as the convolution kernel parameter class to be calculated through the find method in the first cache class. Specifically, the storage location of the convolution kernel parameter class that is the same as the convolution kernel parameter class to be calculated is found according to the hash value, and then the corresponding video memory space is output.
[0084] In S2, storing the convolution kernel parameter class to be calculated and the corresponding video memory space to be calculated into the first cache class can include: storing the convolution kernel parameter class to be calculated and the corresponding video memory space to be calculated into the first cache class through the insert method in the first cache class.
[0085] By establishing a first high-speed cache class, when calculating the GPU memory space corresponding to the convolution kernel parameter class, it is only necessary to calculate it once using the existing method and store the calculated GPU memory space. When calculating the GPU memory space corresponding to this convolution kernel parameter class again, the GPU memory space can be directly retrieved from the first high-speed cache class based on the convolution kernel parameter class, without repeating the calculation, which greatly improves the calculation efficiency.
[0086] S3: During GPU data loading and execution, determine whether there is a convolution kernel parameter class in the second cache class that is the same as the convolution kernel parameter class to be selected; if yes, output the convolution algorithm corresponding to the convolution kernel parameter class to be selected through the second cache class; if no, select the convolution algorithm corresponding to the convolution kernel parameter class to be selected, and store the convolution kernel parameter class to be selected and the convolution algorithm corresponding to the convolution kernel parameter class to be selected in the second cache class.
[0087] After obtaining all parameters of the acceleration library function, the deep learning framework calls the interface in the acceleration library to dynamically select the most suitable convolution algorithm based on the existing convolution kernel parameters. This interface requires the user to pass in all parameters of the convolution kernel, evaluates them, and returns the convolution algorithm with the best acceleration effect for the user to call. In the deep learning acceleration library framework, five types of convolution algorithms are usually provided: GEMM variant, Direct convolutions, FastFourier Transform indirect convolutions, Winograd indirect convolutions, and Implicit GEMM convolutions. In a model training task, the convolution kernel parameters contained in the model are only a few fixed combinations. The traditional calling process will repeatedly select the convolution algorithm each time. Since the return value of this function is only related to the convolution kernel parameters, this embodiment also uses a method similar to S2 to accelerate it using cache.
[0088] The second high-speed cache class and Figure 4 The members of the Cache class shown are the same, except that for the second Cache class, T refers to the convolution algorithm. Specifically, some members of the second Cache class in this embodiment include: std::unordered_map, the find method, and the insert method. std::unordered_map includes a hash function and a comparison function. The hash function is used to calculate the hash value of the convolution kernel parameter class, and the comparison function is used to compare the convolution kernel parameter classes. The find method is used to find and output the convolution algorithm corresponding to the convolution kernel parameter class, and the insert method is used to store the convolution kernel parameter class and the convolution algorithm corresponding to the convolution kernel parameter class.
[0089] Based on the members of the second cache class, in S3, determining whether there is a convolution kernel parameter class in the second cache class that is the same as the convolution kernel parameter class to be selected can include: using the find method in the second cache class to call a hash function, calculating the third hash value of the convolution kernel parameter class to be selected, and comparing the third hash value with the fourth hash value of each convolution kernel parameter class in the second cache class. If any fourth hash value is the same as the third hash value, then it is determined that there is a convolution kernel parameter class in the second cache class that is the same as the convolution kernel parameter class to be selected. The convolution kernel parameter class corresponding to the fourth hash value that is equal to the third hash value is the convolution kernel parameter class that is the same as the convolution kernel parameter class to be selected.
[0090] In S3, outputting the convolution algorithm corresponding to the convolution kernel parameter class to be selected through the second cache class can include: outputting the convolution algorithm corresponding to the same convolution kernel parameter class as the one to be selected through the find method in the second cache class.
[0091] In S3, storing the convolution kernel parameter class to be selected and the convolution algorithm corresponding to the convolution kernel parameter class to be selected into the second cache class can include: storing the convolution kernel parameter class to be selected and the convolution algorithm corresponding to the convolution kernel parameter class to be selected into the second cache class through the insert method in the second cache class.
[0092] By establishing a second cache class, when selecting the convolution algorithm corresponding to the convolution kernel parameter class, the existing method is only needed to select it once and store the selected convolution algorithm. When selecting the convolution algorithm corresponding to the same convolution kernel parameter class again, the convolution algorithm can be directly retrieved from the second cache class based on the convolution kernel parameter class without repeated evaluation, which greatly improves the selection efficiency.
[0093] In this embodiment, the convolution kernel parameter class, the corresponding video memory space, and the convolution algorithm are stored in the cache. On the CPU, the memory access speed of the cache is higher than that of DDR memory, which can further improve efficiency.
[0094] In the PyTorch framework on OpenCL-enabled hardware computing devices, the method described in this embodiment is used to implement the invocation of convolution operators in the acceleration library. When training the deep neural network ResNet-50, it brings a 3.07x performance improvement. The performance optimization comes from the singleton pattern design in S1 and the high-speed cache design in S2 and S3. By taking advantage of the characteristic of deep neural networks to repeatedly call a certain type of operator during training, the computational performance is improved by reducing repeated calculations. This enables efficient invocation of the acceleration library in the PyTorch framework on OpenCL-enabled computing devices, improves the computational performance of OpenCL-enabled computing devices, and is beneficial for transcoding the PyTorch framework to OpenCL-enabled computing devices.
[0095] Example 2:
[0096] This embodiment provides an efficient calling device for accelerating libraries in the PyTorch framework, such as... Figure 5 As shown, the efficient calling device includes:
[0097] The handle binding optimization module M1 is used to determine whether a handle has been bound by a handle in advance by using a handle binding class created based on the singleton pattern. If so, the handle instance generated when binding the handle is output through the handle binding class. If not, the handle is bound through the handle binding class, a handle instance is generated and stored, and the handle instance is output.
[0098] The video memory space calculation optimization module M2 is used to determine whether there is a convolution kernel parameter class in the first cache class that is the same as the convolution kernel parameter class to be calculated during the calculation and allocation of input and output video memory space; if so, the video memory space corresponding to the convolution kernel parameter class to be calculated is output through the first cache class; if not, the video memory space of the convolution kernel parameter class to be calculated is calculated, and the convolution kernel parameter class to be calculated and the video memory space corresponding to the convolution kernel parameter class to be calculated are stored in the first cache class; the convolution kernel parameter class includes convolution kernel parameters;
[0099] The convolution algorithm selection optimization module M3 is used to determine whether there is a convolution kernel parameter class in the second cache class that is the same as the convolution kernel parameter class to be selected during GPU data loading and execution; if so, the convolution algorithm corresponding to the convolution kernel parameter class to be selected is output through the second cache class; if not, the convolution algorithm corresponding to the convolution kernel parameter class to be selected is selected, and the convolution kernel parameter class to be selected and the convolution algorithm corresponding to the convolution kernel parameter class to be selected are stored in the second cache class.
[0100] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the systems disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple; relevant parts can be referred to the method section.
[0101] This document uses specific examples to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. Furthermore, those skilled in the art will recognize that, based on the ideas of the present invention, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of the present invention.
Claims
1. A method for efficiently calling acceleration libraries in the PyTorch framework, characterized in that, The efficient invocation method includes: When binding a handle, the system determines whether the handle has already been bound by using a handle binding class that is pre-created based on the singleton pattern. If it has, the system outputs the handle instance generated when binding the handle using the handle binding class. If it has not, the system binds the handle using the handle binding class, generates and stores the handle instance, and outputs the handle instance. In calculating and allocating input and output video memory space, it is determined whether there is a convolution kernel parameter class in the first cache class that is the same as the convolution kernel parameter class to be calculated; if so, the video memory space corresponding to the convolution kernel parameter class to be calculated is output through the first cache class; if not, the video memory space of the convolution kernel parameter class to be calculated is calculated, and the convolution kernel parameter class to be calculated and the video memory space corresponding to the convolution kernel parameter class to be calculated are stored in the first cache class; the convolution kernel parameter class includes convolution kernel parameters; During GPU data loading and execution, it is determined whether there is a convolution kernel parameter class in the second cache class that is the same as the convolution kernel parameter class to be selected. If so, the convolution algorithm corresponding to the convolution kernel parameter class to be selected is output through the second cache class. If not, the convolution algorithm corresponding to the convolution kernel parameter class to be selected is selected, and the convolution kernel parameter class to be selected and the convolution algorithm corresponding to the convolution kernel parameter class to be selected are stored in the second cache class.
2. The efficient invocation method according to claim 1, characterized in that, The members of the handle binding class include handle_record, Accelerator_Library_Manager, and get_handle; handle_record is a static private object of the handle binding class, used to store handle instances; Accelerator_Library_Manager is a private constructor of the handle binding class, used to bind handles and generate handle instances; get_handle is used to retrieve handle instances.
3. The efficient invocation method according to claim 2, characterized in that, The method determines whether a handle has been bound by a handle binding class created in advance based on the singleton pattern; if so, the handle instance generated when binding the handle is output through the handle binding class. If not, then bind the handle through the handle binding class, generate and store the handle instance, and output the handle instance, specifically including: The handle instance generated when binding the handle is obtained by retrieving the handle instance in the handle binding class that was created in advance based on the singleton pattern, in order to determine whether the handle has been bound; If the handle instance can be obtained, the handle has been bound, and the handle instance is output through get_handle in the handle binding class; If the handle instance cannot be obtained, the handle is not bound. The handle is bound through Accelerator_Library_Manager in the handle binding class, a handle instance is generated, the handle instance is stored through handle_record in the handle binding class, and the handle instance is output through get_handle in the handle binding class.
4. The efficient invocation method according to claim 1, characterized in that, Both the first and second cache classes include members of `std::unordered_map` and the `find` method; `std::unordered_map` includes a hash function used to calculate the hash value of the convolution kernel parameter class; the `find` method is used to find and output the value corresponding to the convolution kernel parameter class; in the first cache class, the value is the video memory space; in the second cache class, the value is the convolution algorithm.
5. The efficient invocation method according to claim 4, characterized in that, The step of determining whether there is a convolution kernel parameter class in the first cache class that is the same as the convolution kernel parameter class to be calculated specifically includes: using the find method in the first cache class to call a hash function to calculate the first hash value of the convolution kernel parameter class to be calculated, and comparing the first hash value with the second hash value of each convolution kernel parameter class in the first cache class. If any second hash value is the same as the first hash value, it is determined that there is a convolution kernel parameter class in the first cache class that is the same as the convolution kernel parameter class to be calculated. The step of determining whether there is a convolution kernel parameter class in the second cache class that is the same as the convolution kernel parameter class to be selected specifically includes: using the find method in the second cache class to call a hash function, calculating the third hash value of the convolution kernel parameter class to be selected, and comparing the third hash value with the fourth hash value of each convolution kernel parameter class in the second cache class. If any of the fourth hash values is the same as the third hash value, then it is determined that there is a convolution kernel parameter class in the second cache class that is the same as the convolution kernel parameter class to be selected.
6. The efficient invocation method according to claim 4, characterized in that, The step of outputting the video memory space corresponding to the convolution kernel parameter class to be calculated through the first cache class specifically includes: outputting the video memory space corresponding to the same convolution kernel parameter class as the one to be calculated through the find method in the first cache class. The step of outputting the convolution algorithm corresponding to the convolution kernel parameter class to be selected through the second cache class specifically includes: outputting the convolution algorithm corresponding to the same convolution kernel parameter class as the one to be selected through the find method in the second cache class.
7. The efficient invocation method according to claim 1, characterized in that, Both the first cache class and the second cache class include an insert method as a member; the insert method is used to store the convolution kernel parameter class and the value corresponding to the convolution kernel parameter class; in the first cache class, the value is the video memory space; in the second cache class, the value is the convolution algorithm.
8. The efficient invocation method according to claim 7, characterized in that, The step of storing the convolution kernel parameter class to be calculated and the corresponding video memory space to be calculated into the first cache class specifically includes: storing the convolution kernel parameter class to be calculated and the corresponding video memory space to be calculated into the first cache class through the insert method in the first cache class. The step of storing the selected convolution kernel parameter class and the corresponding convolution algorithm into the second cache class specifically includes: storing the selected convolution kernel parameter class and the corresponding convolution algorithm into the second cache class through the insert method in the second cache class.
9. A high-efficiency calling device for acceleration libraries in the PyTorch framework, characterized in that, The high-efficiency dispatching device includes: The handle binding optimization module is used to determine whether a handle has been bound beforehand by using a handle binding class created in advance based on the singleton pattern. If so, the handle instance generated when binding the handle is output through the handle binding class. If not, the handle is bound through the handle binding class, a handle instance is generated and stored, and the handle instance is output. The video memory space calculation optimization module is used to determine whether a convolution kernel parameter class with the same name as the convolution kernel parameter class to be calculated exists in the first cache class during the calculation and allocation of input and output video memory space; if so, the video memory space corresponding to the convolution kernel parameter class to be calculated is output through the first cache class; if not, the video memory space of the convolution kernel parameter class to be calculated is calculated, and the convolution kernel parameter class to be calculated and the video memory space corresponding to the convolution kernel parameter class to be calculated are stored in the first cache class; the convolution kernel parameter class includes convolution kernel parameters; The convolution algorithm selection optimization module is used to determine whether there is a convolution kernel parameter class that is the same as the convolution kernel parameter class to be selected in the second cache class during GPU data loading and execution; If yes, then the convolution algorithm corresponding to the convolution kernel parameter class to be selected is output through the second high-speed cache Cache class; if no, then the convolution algorithm corresponding to the convolution kernel parameter class to be selected is selected, and the convolution kernel parameter class to be selected and the convolution algorithm corresponding to the convolution kernel parameter class to be selected are stored in the second high-speed cache Cache class.
Citation Information
Patent Citations
Method for deploying Caffe-based convolutional neural network on FPGA
CN111931913A
Implementation method of OpenCL PyTorch
CN116976430A
