Method for loading operator, method for searching for operator, and apparatus, device and medium

Abstract

Provided are a method for loading an operator, a method for searching for an operator, and an apparatus, a device and a medium, which belong to the technical field of deep learning. The method for loading an operator is applied to a host, wherein the host is externally connected to a computing unit. The method comprises: acquiring an identifier of a target operator; acquiring, from the identifier of the target operator, an index of the target operator in service logic of n levels; and on the basis of the index, searching for the service logic corresponding to the target operator at the n levels, and searching for an executable handle bound to service logic of the target operator at an nth level.

Description

Method for loading operators, method for searching operators, apparatus, device and medium

[0001] This application claims priority to Chinese patent application number 202311238703.3, filed on September 22, 2023, entitled “Method for loading operators, method, device, equipment and medium for searching operators”, the entire contents of which are incorporated herein by reference. Technical Field

[0002] The present application relates to the field of deep learning technology, and in particular to a method for loading an operator, a method for searching an operator, an apparatus, a device, and a medium. Background Art

[0003] To improve operator execution efficiency, consider using the host and computing units for collaborative computing and mutual acceleration. This requires the operator's business logic to run on both the host and the computing units. Computing units include external computing units such as neural processor units (NPUs), graphics processing units (GPUs), and digital signal processors (DSPs), which are plugged into the host.

[0004] Summary of the Invention

[0005] The present application provides a method for loading an operator, a method for searching an operator, an apparatus, a device, and a medium, which enable the business logic of the operator to run on a host and a computing unit.

[0006] In a first aspect, the present application provides a method for loading an operator, which is applied to a host connected to an external computing unit. The method includes: loading a first business logic into the memory of the host, wherein the first business logic is the business logic of the operator to be loaded of the deep learning model running on the host, the first business logic is divided into n levels of business logic according to attributes, each business logic of the i+1th level in the n levels has a subordinate relationship with a business logic of the i-th level, the business logic of the nth level is bound one-to-one to an executable handle, and the executable handle is used to execute the business logic of the operator to be loaded running on the external computing unit, i is greater than or equal to 1 and less than or equal to n-1, loading a second business logic into the memory of the host, and loading the second business logic from the memory of the host into the memory of the external computing unit, wherein the second business logic is the business logic of the operator to be loaded running on the external computing unit.

[0007] In the solution shown in this application, the business logic running on the host is hierarchical, and the business logic of the lowest level is bound to an executable handle, so that when the operator is subsequently executed, the business logic running on the operator on the host can be found and executed according to the level, and the executable handle is found using the business logic of the lowest level, so that the business logic running on the operator on the external computing unit can be executed, thereby realizing the execution of the operator on the host and the external computing unit.

[0008] In an optional embodiment, the method further includes assigning consecutive identifiers to all business logic in the first level of the n levels, and assigning consecutive identifiers to business logic in the (i+1)th level that is subordinate to any business logic in the (i)th level. By assigning identifiers in this manner, operators can be searched sequentially, saving operator search time.

[0009] In an optional manner, the method further includes: obtaining the identifier of the target operator in the operator to be loaded, obtaining the index of the target operator in the business logic of the n levels from the identifier of the target operator, and based on the index, searching for the business logic corresponding to the target operator in the n levels, and searching for the executable handle bound to the business logic of the target operator at the nth level.

[0010] In the solution presented in this application, the operator's identifier is used to determine its index within n levels of business logic. Based on this index and the relationship between the business logic at adjacent levels, the corresponding business logic at the operator at n levels is found, and the executable handle bound to the operator's business logic at the nth level is found. This eliminates the need for hash calculations and hash conflict resolution, enabling rapid operator lookup and reducing the overall operator access and execution time.

[0011] In an optional manner, obtaining the identifier of the target operator in the operators to be loaded includes obtaining a cached identifier of the target operator. In this way, when the identifier of the operator is already available, there is no need to determine the identifier of the operator, which can save time in searching for the operator.

[0012] In an optional manner, before obtaining the cached identifier of the target operator, it also includes: based on the parameter information of the target operator, determining the optimal business logic of the target operator step by step in the business logic of the n levels, wherein the parameter information includes one or more specific parameters, data formats or data types used by the target operator, and tensor shapes, and the identifier of the optimal business logic of the target operator is composed of the identifier of the target operator in the hierarchical order of the n levels, and the identifier of the target operator is cached. In the scheme shown in the present application, the parameter information of the operator is used to search and cache the identifier of the operator. In this way, when searching for the operator later, there is no need to determine the identifier of the operator, which can save the time of searching for the operator.

[0013] In an optional manner, based on the index, the business logic corresponding to the target operator at the n levels is searched, and the executable handle bound to the business logic at the n level of the target operator is searched, including: obtaining n information tables corresponding to the business logic at the n levels, wherein the information table corresponding to the business logic of the i-th level includes information of each business logic at the i-th level, for each business logic at the i-th level, the information of the business logic includes storage address information of the business logic and address information of an information subtable, the information subtable includes information of the business logic at the i+1-th level subordinate to the business logic, the information table of the n-th level includes information of each business logic at the n-th level, for each business logic at the n-th level, the information of the business logic includes storage address information of the business logic and the executable handle bound to the business logic, based on the n information tables and the index, the business logic corresponding to the target operator at the n levels is searched, and the executable handle bound to the business logic at the n-th level of the target operator is searched.

[0014] In the solution shown in this application, there is an information table corresponding to the business logic of n levels. Since the information of the business logic is recorded in the information table, the information table and index can be used to quickly find the business logic corresponding to the operator at n levels, and find the executable handle bound to the business logic of the operator at the nth level, saving the time of searching for the operator.

[0015] In the second aspect, the present application provides a method for searching for an operator, which is applied to a host, which is connected to an external computing unit, and the method includes: obtaining the identifier of the target operator; in the identifier of the target operator, obtaining the index of the target operator in the business logic of n levels, wherein the business logic of the n levels runs on the host, and each business logic of the i+1th level in the n levels has a subordinate relationship with a business logic of the i-th level, and the business logic of the n-th level is bound one-to-one to an executable handle, which is used to execute the business logic of the operator running on the external computing unit, i is greater than or equal to 1 and less than or equal to n-1; based on the index, searching for the business logic corresponding to the target operator on the n levels, and searching for the executable handle bound to the business logic of the target operator at the n-level.

[0016] In the solution shown in this application, the business logic running on the host is hierarchical, and the business logic of the lowest level is bound to an executable handle, so that when the operator is subsequently executed, the business logic running on the host by the operator can be found and executed according to the level, and the executable handle is found using the business logic of the lowest level, so that the business logic running on the external computing unit by the operator can be executed, thereby realizing the execution of the operator on the host and the external computing unit. In addition, the identifier of the operator is used to determine the index in the business logic of n levels, and based on the index and the relationship between the business logic of the adjacent levels, the business logic corresponding to the operator at n levels is found, and the executable handle bound to the business logic of the operator at the nth level is found. In this way, there is no need to perform hash calculations and resolve hash conflicts, thereby achieving fast search for operators, thereby reducing the overall time for operator access and execution.

[0017] In an optional manner, obtaining the identifier of the target operator includes obtaining a cached identifier of the target operator. In this way, when the identifier of the operator is already available, there is no need to determine the identifier of the operator, which can save time in searching for the operator.

[0018] In an optional manner, before obtaining the cached identifier of the target operator, it also includes: based on the parameter information of the target operator, determining the optimal business logic of the target operator step by step in the business logic of the n levels, wherein the parameter information includes one or more specific parameters, data formats or data types, and tensor shapes used by the target operator, and the identifier of the optimal business logic of the target operator is composed of the identifier of the target operator in the hierarchical order of the n levels, and the identifier of the target operator is cached.

[0019] In the solution shown in this application, the operator's parameter information is used to search and cache the operator's identifier. In this way, when searching for the operator later, there is no need to determine the operator's identifier again, which can save the time of searching for the operator.

[0020] In an optional manner, based on the index, searching for the business logic corresponding to the target operator at the n levels, and searching for the executable handle bound to the business logic at the n level of the target operator, includes: obtaining n information tables corresponding to the business logic at the n levels, wherein the information table corresponding to the business logic at the i-th level includes information of each business logic at the i-th level, for each business logic at the i-th level, the information of the business logic includes storage address information of the business logic and address information of an information subtable, the information subtable includes information of the business logic at the i+1-th level subordinate to the business logic, the information table at the n-th level includes information of each business logic at the n-th level, for each business logic at the n-th level, the information of the business logic includes storage address information of the business logic and the executable handle bound to the business logic; based on the n information tables and the index, searching for the business logic corresponding to the target operator at the n levels, and searching for the executable handle bound to the business logic at the n-th level of the target operator.

[0021] In the solution shown in this application, there is an information table corresponding to the business logic of n levels. Since the information of the business logic is recorded in the information table, the information table and index can be used to quickly find the business logic corresponding to the operator at n levels, and find the executable handle bound to the business logic of the operator at the nth level, saving the time of searching for the operator.

[0022] In an optional manner, the method further includes: loading the business logic of the operator to be loaded of the deep learning model running on the host into the memory of the host, wherein the business logic of the n levels is obtained by dividing the business logic of the operator to be loaded on the host according to attributes, and the target operator belongs to the operator to be loaded; loading the business logic of the operator to be loaded running on the external computing unit into the memory of the host, and loading the business logic of the operator to be loaded running on the external computing unit from the memory of the host to the external computing unit; assigning continuous identifiers to all business logics of the first level of the n levels; and assigning continuous identifiers to the business logic of the i+1th level that is subordinate to any business logic of the i-th level.

[0023] The solution shown in this application loads the business logic of the operators used by the deep learning model to the running location in advance, and assigns identifiers in this way. When searching for operators, they can be searched in sequence, saving the time of searching for operators.

[0024] In an optional manner, the business logic of the operators to be loaded of the deep learning model running on the host is loaded into the memory of the host, including: when configuring the deep learning model or running the deep learning model, the business logic of all operators included in the deep learning model running on the host is loaded into the memory of the host.

[0025] In a third aspect, the present application provides a device for loading an operator, the device comprising at least one module, and the at least one module is used to implement the method for loading an operator provided in the first aspect or any optional method in the first aspect.

[0026] In some embodiments, the module in the device for loading an operator is implemented by software, and the module in the device for loading an operator is a program module. In other embodiments, the module in the device for loading an operator is implemented by hardware or firmware.

[0027] In a fourth aspect, the present application provides a device for searching operators, the device comprising at least one module, and the at least one module is used to implement the method for searching operators provided in the second aspect or any optional method in the second aspect.

[0028] In some embodiments, the module in the apparatus for searching the operator is implemented by software, and the module in the apparatus for searching the operator is a program module. In other embodiments, the module in the apparatus for searching the operator is implemented by hardware or firmware.

[0029] In a fifth aspect, the present application provides a computing device comprising a memory and a processor, wherein the processor of the computing device is used to execute instructions stored in the memory of the computing device, so that the computing device executes the method of loading an operator provided in the above-mentioned first aspect or any optional manner in the first aspect, or so that the computing device executes the method of searching an operator provided in the above-mentioned second aspect or any optional manner in the second aspect.

[0030] In a sixth aspect, the present application provides a computer-readable storage medium, which includes computer program instructions. When the computer program instructions are executed by a computing device, the computing device executes the method for loading operators provided in the above-mentioned first aspect or any optional manner in the first aspect, or the computing device executes the method for searching operators provided in the above-mentioned second aspect or any optional manner in the second aspect.

[0031] In the seventh aspect, the present application provides a computer program product comprising instructions, which, when executed by a computing device, enables the computing device to execute the method for loading operators provided in the first aspect or any optional method in the first aspect, or enables the computing device to execute the method for searching operators provided in the second aspect or any optional method in the second aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] FIG1 is a schematic diagram of a traditional search operator method;

[0033] FIG2 is a schematic diagram of the structure of a computing device provided by an exemplary embodiment of the present application;

[0034] FIG3 is a schematic diagram of a pre-stage loading operator provided by an exemplary embodiment of the present application;

[0035] FIG4 is a schematic diagram of a delayed hierarchical loading operator provided by an exemplary embodiment of the present application;

[0036] FIG5 is a schematic diagram of a hierarchical loading operator provided by an exemplary embodiment of the present application;

[0037] FIG6 is a schematic diagram of an information table provided by an exemplary embodiment of the present application;

[0038] FIG7 is a schematic diagram of calling a target operator for the first time provided by an exemplary embodiment of the present application;

[0039] FIG8 is a schematic diagram of a flow chart of a search operator provided by an exemplary embodiment of the present application;

[0040] FIG9 is a schematic diagram of a flow chart of a search operator provided by an exemplary embodiment of the present application;

[0041] FIG10 is a schematic structural diagram of an apparatus for loading an operator provided by an exemplary embodiment of the present application;

[0042] FIG11 is a schematic structural diagram of a search operator apparatus provided by an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0043] In order to make the objectives, technical solutions and advantages of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0044] The following is an explanation of some terminology concepts involved in the embodiments of this application.

[0045] 1. Operators are functions used to perform various mathematical operations and calculations in deep learning models. When the host and computing units collaborate and accelerate each other (that is, in heterogeneous programming), they typically include business logic on the host side and business logic on the external device side. The host-side business logic runs on the host, while the device-side business logic runs on an external computing unit such as an NPU, GPU, or DSP. The external computing unit is connected to the host via a bus. The host-side business logic is the binary code that the operator runs on the host, while the device-side business logic is the binary code that the operator runs on the external computing unit.

[0046] 2. The executable handle of the operator is the handle used by the device to execute the operator kernel function. The kernel function can be found and executed through this handle.

[0047] The background of the present application and conventional technologies are described below.

[0048] As the bottom layer of the entire application software stack, operators are used by various accelerator cards, reasoning engines, and training frameworks in the upper layers. Operators need to be quickly found and run, especially in certain reasoning scenarios where the search time for operators is strictly required. In traditional technology, referring to Figure 1, when searching for an operator, a hash calculation is performed using the global identifier of the operator. Based on the hash value after the hash calculation, the operator is found in the hash table. Since it is necessary to search through the hash table, there is the overhead of hash calculation and hash conflict resolution. When the number of operators is large, the time-consuming overhead of resolving hash conflicts increases rapidly. Moreover, hash search is not a stable constant time, which brings about time uncertainty. Based on this, in an embodiment of the present application, the identifier of the operator is used to split out the index of the operator in the business logic of n levels, and the index at each level is used to find the executable handle of the operator, thereby executing the operator. Since there is no need to perform hash calculation or resolve hash conflicts, it is possible to save the search time of the operator.

[0049] The following describes the execution body of the method for loading an operator and the execution body of the method for searching an operator.

[0050] The method for loading an operator is performed by a device that loads the operator. This device is referred to as a host, which is a computing device such as a terminal or server. Alternatively, this device is a software device, such as a software program running on the host.

[0051] The execution subject of the method for searching an operator is a device for searching an operator, which is a host mentioned later, and the host is a computing device such as a terminal or server. Alternatively, the device is a software device, such as a software program running on the host.

[0052] As shown in FIG2 , a computing device 200 is provided, which may be implemented by a general bus architecture. The computing device 200 includes at least one processor 201 , a communication bus 202 , a memory 203 , and at least one network interface 204 .

[0053] The processor 201 is, for example, a general-purpose central processing unit (CPU), a network processor (NP), a graphics processing unit (GPU), a neural-network processing unit (NPU), a data processing unit (DPU), a microprocessor, or one or more integrated circuits for implementing the solution of the present application. For example, the processor 201 includes an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The PLD is, for example, a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0054] Communication bus 202 is used to transmit information between the above components. Communication bus 202 can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used in Figure 2, but this does not mean that there is only one bus or one type of bus.

[0055] The memory 203 is, for example, a read-only memory (ROM) or other type of static storage device that can store static information and instructions, a random access memory (RAM) or other type of dynamic storage device that can store information and instructions, an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, an optical disc storage (including a compact disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 203 is, for example, independent and connected to the processor 201 via the communication bus 202. The memory 203 can also be integrated with the processor 201.

[0056] Optionally, the memory 203 is used to store the information table mentioned below. When the computing device 200 needs to use the information table, the processor 201 accesses the memory 203 to obtain the information table stored in the memory 203.

[0057] The network interface 204 uses any transceiver-like device for communicating with other devices or communication networks. The network interface 204 includes a wired network interface and may also include a wireless network interface. For example, the wired network interface may be an Ethernet interface. The Ethernet interface may be an optical interface, an electrical interface, or a combination thereof. The wireless network interface may be a wireless local area network (WLAN) interface, a cellular network interface, or a combination thereof.

[0058] In a specific implementation, as an example, the processor 201 may include one or more CPUs.

[0059] In a specific implementation, as an example, the computing device 200 may include multiple processors. Each of these processors may be a single-core processor (single-CPU) or a multi-core processor (multi-CPU). A processor herein may refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions).

[0060] In a specific implementation, as an example, the computing device 200 may further include an output device and an input device. The output device communicates with the processor 201 and can display information in various ways. For example, the output device can be a liquid crystal display (LCD), a light emitting diode (LED) display device, a cathode ray tube (CRT) display device, or a projector. The input device communicates with the processor 201 and receives user input in various ways. For example, the input device can be a mouse, a keyboard, a touch screen device, or a sensor device.

[0061] In some embodiments, the memory 203 is used to store program code 2031 for executing the search operator or the load operator in this application, and the processor 201 executes the program code 2031 stored in the memory 203. That is, the computing device 200 can implement the method of searching for an operator or loading an operator provided in the method embodiment through the processor 201 and the program code 2031 in the memory 203.

[0062] The following describes the application scenarios of the operator loading method and the operator search method.

[0063] In the embodiment of the present application, the application scenarios include scenario one and scenario two. Scenario one is a scenario of preloading operators, that is, loading operators when configuring the deep learning model, and scenario two is a scenario of delayed loading, that is, configuring the deep learning model first and loading operators when executing the operators. In scenarios one and two, the interactive process involving the application, deep neural network (DNN) acceleration library and operator library is involved. The application, DNN acceleration library and operator library are all software programs, all deployed on the host, and the application is a program that uses the deep learning model to perform calculations. The DNN acceleration library can provide highly optimized implementations of common operations such as convolution, pooling, normalization and activation, which can accelerate calculations. The host is connected to one or more external computing units, and the types of the multiple external computing units are exactly the same. For example, the multiple external computing units are all GPUs, or there are external computing units of different types in the multiple external computing units. For example, the multiple external computing units include GPUs and NPUs.

[0064] Scenario 1: Preloading an operator. See Figure 3 for the preloading operator and the operator execution process.

[0065] In offline inference or training scenarios, load a deep learning model (any inference model, such as an image classification model or a speech recognition model) and start loading the operators included in the deep learning model.

[0066] Step 301: The application loads a deep learning model file. The deep learning model file is stored in the Open Neural Network Exchange (ONNX) format.

[0067] Among them, the application is a program that uses a deep learning model to perform calculations in offline inference scenarios or training scenarios.

[0068] Step 302: The application calls the Setup interface to instruct the DNN acceleration library to configure the deep learning model.

[0069] In step 303, the DNN acceleration library creates a context (create Context), sets a stream (set Stream), and sets an allocator (set Allocator).

[0070] Stream is the task queue of the external computing unit. Tasks are executed in the order in Stream. Here, tasks can be kernel functions of operators. Set Allocator is used to specify the memory allocation strategy.

[0071] In step 304, the DNN acceleration library calls the loading interface to instruct the operator library to preload the operators used in the operator library.

[0072] The operator library has an operator list that records information about all operators included in the operator library. The operators used in the operator library include all operators used by the deep learning model, but may exceed the total number of operators used by the deep learning model.

[0073] Step 305: The operator library loads operators in a hierarchical manner.

[0074] In this embodiment, the operator library loads the full amount of host-side and device-side business logic into the host's memory. The host-side business logic mainly includes the logic of operator execution preparation and the logic of the operator's executable handle search strategy. For example, operator execution preparation includes the derivation of the shape of the tensor and data segmentation. The operator library then loads the device-side business logic into the memory of each external computing unit. For detailed loading process, please refer to the description below. Among them, the host-side business logic refers to the business logic of the operator running on the host, and the device-side business logic refers to the business logic of the operator running on the external computing unit.

[0075] It should be noted that in the scenario of preloading operators, the business logic of the device side is loaded into each external computing unit. The reason is: it is currently unknown on which external computing unit the subsequent device-side business logic will run, so each external computing unit has all the device-side business logic. In this way, when running the device-side business logic subsequently, there is no need to reload it.

[0076] In step 306 , the operator library returns a return value to the DNN acceleration library indicating that loading is complete.

[0077] In step 307 , the DNN acceleration library returns a return value to the application program indicating that the configuration is complete.

[0078] In step 308, the application provides the DNN acceleration library with computation-related description information, such as input / output description (input / output desc), convolution description (conv desc), and input / output memory (i / o mem).

[0079] The input / output description includes the format of the input / output data, etc., and the input / output memory refers to the memory occupied by the input / output data.

[0080] In step 309, the DNN acceleration library calls the Get Kernel interface to obtain the kernel function of the target operator from the operator library.

[0081] The kernel function acquisition interface is used to acquire the kernel function of the target operator, which is any operator of the index. The above description information is carried when the kernel function acquisition interface is called.

[0082] In step 310 , the operator library determines the optimal business logic of the target operator among the business logics at each level, determines the identifier of the target operator, and returns the identifier of the target operator to the DNN acceleration library.

[0083] In this embodiment, the operator library uses the description information to determine the optimal business logic of the target operator at each level of the business logic. The detailed process is described later. The operator library combines the identifier of the optimal business logic in the hierarchical order to form the identifier of the target operator.

[0084] It's important to note that the operator library determines the business logic at each level running on the host and then executes it. For example, the business logic on the host is divided into three levels: the operation level, the tactic level, and the kernel level. The operator library first determines the business logic at the operation level and then executes the business logic at the operation level. After determining the business logic at the tactic level, it then executes the business logic at the tactic level. After determining the business logic at the kernel level, it then executes the business logic at the kernel level.

[0085] Step 311: The DNN acceleration library caches the identifier of the target operator.

[0086] In step 311, after the DNN acceleration library caches the identifier of the target operator to the host's memory, when the target operator is accessed again later, the operator can be indexed in constant time based on the identifier of the target operator.

[0087] In step 312, the DNN acceleration library calls the interface for obtaining the kernel, and requests the operator library to obtain the executable handle of the kernel. When calling the interface, the identifier of the target operator is carried.

[0088] In step 313, the operator library uses the identifier of the target operator to determine the executable handle of the kernel and returns the executable handle to the DNN acceleration library.

[0089] In step 314, the DNN acceleration library calls the get workspace size interface to obtain the workspace size from the operator library.

[0090] In step 315 , the operator library determines the amount of data when the kernel function is launched (kernel.Get Launch Data Size), and determines the amount of intermediate data when the kernel function is running (kernel.Get Scratch Size).

[0091] In step 316, the operator library returns the amount of data when the kernel function is started and the amount of intermediate data when the kernel function is running to the DNN acceleration library.

[0092] In step 317 , the DNN acceleration library allocates runtime memory (Allocate Workspace Mem) based on the amount of data when the kernel function is started and the amount of intermediate data when the kernel function is running.

[0093] In step 318, the DNN acceleration library calls the Setup Run Info interface to configure the run information for the operator library.

[0094] In step 319 , the operator library initializes the kernel function startup data (kernel.Initial Launch Data) and returns a return value indicating that initialization is complete to the DNN acceleration library.

[0095] In step 320, the DNN acceleration library calls the launch kernel interface to instruct the operator library to launch the kernel.

[0096] Step 321: The operator library runs the kernel function (Kernel.Run).

[0097] The operator library instructs the external computing unit to run the kernel function. For example, the operator library determines the load of each external computing unit and uses the external computing unit with the smallest load to run the kernel function.

[0098] Step 322: If the kernel function fails to execute correctly, the operator library returns error (Error Handling) information to the DNN acceleration library. If the kernel function completes its execution correctly, the operator library returns the calculation result to the DNN acceleration library.

[0099] In step 323, the DNN acceleration library returns a check result (Check Result) to the application.

[0100] Scenario 2: Delayed Loading Operator Scenario. See Figure 4 for the delayed loading operator and the operator execution process.

[0101] Step 401: The application loads a deep learning model file. The deep learning model file is stored in the ONNX format.

[0102] Step 402: The application calls the Setup interface to instruct the DNN acceleration library to configure the deep learning model.

[0103] In step 403, the DNN acceleration library creates a Context, sets a Stream, and sets an Allocator.

[0104] In step 404 , the DNN acceleration library returns a return value to the application program indicating that the configuration is complete.

[0105] In step 405, the application provides the DNN acceleration library with computation-related description information such as input / output desc, conv desc, and i / o mem.

[0106] In step 406 , the DNN acceleration library instructs the operator library to load the required operators based on the description information provided by the application.

[0107] The required operators are all operators included in the deep learning model.

[0108] Step 407: The operator library loads operators in a hierarchical manner.

[0109] In this embodiment, the operator library loads the business logic of all operators on the host side and the device side into the host's memory, and then loads the business logic on the device side into the external computing unit running based on the external computing unit running the business logic on the device side. For the detailed loading process, please refer to the description below.

[0110] In step 408 , the operator library returns a return value to the DNN acceleration library indicating that loading is complete.

[0111] In step 409, the DNN acceleration library calls the Get Kernel interface to obtain the kernel function of the target operator from the operator library.

[0112] The kernel function acquisition interface is used to acquire the kernel function of the target operator, which is any operator of the index. The above description information is carried when the kernel function acquisition interface is called.

[0113] In step 410 , the operator library determines the optimal business logic of the target operator among the business logics at each level, determines the identifier of the target operator, and returns the identifier of the target operator to the DNN acceleration library.

[0114] In this embodiment, the operator library uses the description information to determine the optimal business logic of the target operator at each level of the business logic. The detailed process is described later. The operator library combines the identifier of the optimal business logic in the hierarchical order to form the identifier of the target operator.

[0115] Step 411: The DNN acceleration library caches the identifier of the target operator.

[0116] In step 412, the DNN acceleration library calls the interface for obtaining the kernel, and requests the operator library to obtain the executable handle of the kernel. When calling the interface, the identifier of the target operator is carried.

[0117] In step 413, the operator library uses the identifier of the target operator to determine the executable handle of the kernel and returns the executable handle to the DNN acceleration library.

[0118] In step 414, the DNN acceleration library calls the get workspace size interface to obtain the operating space size from the operator library.

[0119] In step 415 , the operator library determines kernel.Get Launch Data Size and kernel.Get Scratch Size.

[0120] In step 416, the operator library returns the amount of data when the kernel function is started and the amount of intermediate data when the kernel function is running to the DNN acceleration library.

[0121] In step 417 , the DNN acceleration library allocates running memory based on the amount of data when the kernel function is started and the amount of intermediate data when the kernel function is running.

[0122] In step 418, the DNN acceleration library calls the Setup Run Info interface to configure the run information for the operator library.

[0123] In step 419 , the operator library initializes the startup data of the kernel function and returns a return value indicating that the initialization is complete to the DNN acceleration library.

[0124] In step 420 , the DNN acceleration library calls the interface for starting the kernel function, instructing the operator library to start the kernel function.

[0125] Step 421: The operator library runs the kernel function.

[0126] Step 422: If the kernel function fails to execute correctly, the operator library returns the error information to the DNN acceleration library. If the kernel function completes the execution correctly, the operator library returns the calculation result to the DNN acceleration library.

[0127] In step 423, the DNN acceleration library returns the verification result to the application.

[0128] The difference between the processes shown in Figure 3 and Figure 4 is that Figure 3 preloads all operators, while Figure 4 determines the external computing unit of the operator to be executed before execution, and then loads the business logic of the host side and device side of the required execution operator into the corresponding memory of the external computing unit.

[0129] Furthermore, in the process shown in Figure 4, when the host is connected to multiple external computing units, it's unclear which external computing unit an operator will ultimately run on during deep learning model configuration. However, during execution, the external computing unit on which the operator will run is already determined, allowing the operator's device-side business logic to be directly loaded into the memory of that external computing unit. Furthermore, operators can be loaded only as needed, avoiding the time-consuming initial loading of all operators.

[0130] The following describes the hierarchical loading process, the information table generation process, the process of first determining the target operator's identifier and calling the target operator, and the operator search process.

[0131] 1. Hierarchical loading process.

[0132] When programming an operator, programmers classify the business logic running on the host based on different attributes, resulting in n levels of business logic. These attributes include, but are not limited to, the number of input and output tensors, the shape of the tensors, the data type of the tensors, or the data layout format of the tensors. Within the n levels, except for the topmost level (the first level), the business logic of each level is subordinate to a business logic in the level above it. The business logic at the bottom level is bound to an executable handle one-to-one. That is, a business logic at the bottom level corresponds to the executable handle of an operator on the device side, and different business logics at the bottom level correspond to the executable handles of different operators on the device side. Operators under the same business logic at level i share one or more of the same attributes. For example, operators under the same business logic at level i share the same number of input and output tensors. For another example, the data type of the input tensors of operators under the same business logic at level i is floating-point. Operators under different business logics at level i differ in at least one of the aforementioned attributes. For example, the data types of the input tensors of the two operators under the business logic of the i-th level are floating-point type and integer type respectively.

[0133] The operators to be loaded are the operators used by the deep learning model in the operator library. The host loads the business logic of the operators to be loaded into the host's memory. The host assigns consecutive identifiers to all business logics in the first layer, records the subordinate relationships between the business logics at each layer and the business logics of the upper layer to which they are subordinate. During the process of establishing subordinate relationships, the host assigns consecutive identifiers to each business logic that is subordinate to the same business logic in the upper layer. For example, the first layer includes three business logics, and the identifiers of the three business logics are 0, 1, and 2. In the second layer, the business logic subordinate to the business logic with identifier 0 includes three business logics, and the identifiers of the three business logics are 0, 1, and 2. In the second layer, the business logic subordinate to the business logic with identifier 1 includes four business logics, and the identifiers of the four business logics are 0, 1, 2, and 3.

[0134] The host then loads the device-side business logic of the operator to be loaded into the host's memory, which is then further loaded into the memory of the external computing unit. During the loading process, the lowest-level business logic (i.e., the nth level) is bound to an executable handle. This allows subsequent searches for the lowest-level business logic on the host to access the bound executable handle and execute the operator.

[0135] For example, see Figure 5. The business logic of the host side of the operator to be loaded is compiled and encapsulated in the host binary (Host Bin) file, and the business logic of the operator to be loaded on the device side is compiled and encapsulated in the Device Bin file. Both the Host Bin file and the Device Bin file belong to the host's file system. The business logic of the operator to be loaded on the host side is divided into three levels, from top to bottom, namely the operation level, the Tactic level, and the kernel level. An operation is shown in the operation level, and two Tactics are shown in the Tactic level as belonging to the operation. Two kernels in the kernel level are subordinate to one Tactic, and the business logic of another Tactic in the kernel level is not shown. When assigning identifiers, the identifier assigned to the one operation is 0, the identifiers assigned to the two Tactics are 0 and 1, and the identifiers assigned to the two kernels are 0 and 1.

[0136] The Device Bin file includes Tactic0.0 and Tactic1.0. Tactic0.0 is loaded onto the external computing unit Device0, and Tactic1.0 is loaded onto the external computing unit Device1. Tactic0.0 includes kernel0 and kernel1 functions, and Tactic1.0 includes kernel0 and kernel1 functions. In Tactic0.0, kernel0 functions on the device side are associated with kernel0 on the host side, and kernel1 functions on the device side are associated with kernel1 on the host side.

[0137] Optionally, when loading the operator to be loaded, all the host-side business logic in the Host Bin file is loaded into the host's memory at one time, and all the device-side business logic in the Device Bin file is loaded into the host's memory at one time, and then all the device-side business logic is further loaded into the memory of the external computing unit.

[0138] In addition, a hierarchical loading mechanism is adopted. When different operators in the operators to be loaded have the same business logic, the business logic is only loaded once, which can save memory storage space.

[0139] 2. The process of generating the information table.

[0140] During the loading process of the operators to be loaded, information tables are generated for each level. The number of information tables is equal to the number of levels. For example, if there are n levels, the number of information tables is also n.

[0141] First, an information table for the top-level business logic is generated. This table contains information about all the top-level business logic. The index of each business logic's information serves as its identifier. After the top-level business logic information table is generated, it is stored in a sequential table format. Specifically, the information for each top-level business logic is stored sequentially in a range of consecutive physical address locations in the host's memory.

[0142] Next, an information table for the second-level business logic is generated. This information table includes multiple information sub-tables. Each information sub-table corresponds to one or more business logics. The business logic corresponding to each information sub-table is subordinate to a business logic in the first level. Different information sub-tables correspond to different business logics in the first level. Each information sub-table contains information about each corresponding business logic. The index of each business logic's information is the identifier of the business logic.

[0143] Next, an information table for the third-level business logic is generated. This information table includes multiple information sub-tables. Each information sub-table corresponds to one or more business logics. The business logic corresponding to each information sub-table is subordinate to a business logic at the second level. Different information sub-tables correspond to different business logics at the second level. Each information sub-table contains information about each corresponding business logic. The index of each business logic's information is the identifier of the business logic.

[0144] Similarly, information tables for business logic at each level are generated step by step. Each information sub-table included in the information table is stored in the host's memory in the form of a sequential table.

[0145] Assuming there are n levels, for each business logic from the first level to the n-1th level, the business logic information includes the name of the business logic, its storage address information in host memory, and the address information of an information subtable. The information subtable includes information about the business logic of the next layer subordinate to the business logic, and the address information includes the first address of the information subtable in host memory. For the business logic at the nth level, the business logic information includes the name of the business logic, its storage address information in host memory, and kernel function information. After obtaining the business logic based on the storage address information, the executable handle bound to the business logic can be obtained through the business logic. For example, the business logic includes the executable handle.

[0146] Figure 6 shows the information table of the business logic of the operator to be loaded shown in Figure 5. Among them, the information table of the Operation level includes the information of all business logics of the Operation level, and the information of each business logic includes the Operation name, Operation pointer (handle) and the address information of the Tactic level information subtable. The information subtable of the Tactic level includes the information of all business logics of the Operation with a subordinate identifier of 0, and the information of each business logic includes the Tactic name, Tactic handle and the address information of the Kernel level information subtable. The information subtable of the Kernel level includes the information of all business logics of the Tactic with a subordinate identifier of 0, and the information of each business logic includes the Kernel name, Kernel handle and Kernel information. Operation name, Tactic name and Kernel name respectively represent the names of the business logic of each level. Operation handle, Tactic handle and Kernel handle respectively represent the storage address information of the business logic of each level. Kernel information includes relevant information such as the space and parameters required to run the kernel function.

[0147] It should be noted that, from the above description, the identifier of the target operator is divided into data domains of multiple lengths according to the number of levels. The number of data domains is the same as the number of levels. The length of the data domain corresponding to each level can be set according to actual needs. It can be the same or different. The value of each data domain is the identifier of the operator's non-business logic at that level. According to this strategy, each operator has a unique identifier.

[0148] In addition, a reserved field may be set in the identifier of the target operator, and the reserved field may be set at the end of the identifier of the target operator.

[0149] 3. The process of first determining the target operator's identity and calling the target operator.

[0150] Users building deep learning models configure the specific parameters of each operator based on the operator list in the operator library. These specific parameters include the identifier of the operator's first-level business logic. When calling the target operator, the target operator's specific parameters and parameter information are obtained. This parameter information includes one or more of the specific parameters used by the target operator, the data layout format of the input tensor, the shape of the input tensor, or the data type of the input tensor. These specific parameters are unique to the target operator.

[0151] When calling the target operator, the first-level index of the target operator's business logic is obtained from the target operator's specified parameters. Based on this index, the information of the target operator's first-level business logic is determined from the first-level information table. From this information, the address information of the target operator's first-level business logic is obtained. This address information is used to read the first-level business logic of the target operator from memory. Furthermore, from this information, the address information of the second-level information subtable is obtained. This information subtable is an information subtable of the business logic subordinate to the first business logic. This address information is used to determine the second-level business logic subordinate to the first business logic.

[0152] The host then obtains the target operator's parameter information. Using this information, the host determines the optimal business logic for the target operator at each subsequent level of business logic. The identifiers of the optimal business logic for the target operator determined at each level are combined in order from the highest level to the lowest level to obtain the target operator's identifier. In other words, the identifiers for the target operator are arranged from the first level to the nth level, from the front to the back.

[0153] For example, referring to FIG7 , n levels include 3 levels, including the Operation level, the Tactic level, and the Kernel level. The specified parameters of the target operator are used at the Operation level to determine the first business logic of the target operator at the Operation level. The business logic of the Tactic level subordinate to the first business logic is determined. In the business logic of the Tactic level, a second business logic that conforms to one or more of the specific parameters, data formats, or data types is determined. The second business logic is the optimal business logic of the target operator at the Tactic level. Then, the business logic of the Kernel level subordinate to the second business logic is determined. In the business logic of the Kernel level, a third business logic that conforms to the tensor shape is determined. The third business logic is the optimal business logic of the target operator at the Kernel level. The identifier of the target operator includes the identifier of the first business logic, the identifier of the second business logic, and the identifier of the third business logic. The identifiers of the first business logic, the second business logic, and the third business logic are arranged from front to back.

[0154] In addition, after determining the optimal business logic of the target operator in the lowest level, the executable handle bound to the optimal business logic is determined. This executable handle is then used to start the launch process and execute the target operator.

[0155] The above process is the process of calling the target operator for the first time. During the first call of the target operator, after the identifier of the target operator is determined, the identifier of the target operator is cached. The identifier of the target operator can be directly used to search for the target operator later.

[0156] 4. The process of searching for operators.

[0157] After determining the identifier of the target operator, the identifier of the target operator is cached. When the target operator is used again, the cached identifier of the target operator is obtained. The process of searching for an operator is shown in steps 801 to 803 in FIG8 .

[0158] Step 801: Obtain the identifier of the target operator.

[0159] In this embodiment, the host obtains the identifier of the cached target operator.

[0160] Step 802: Obtain the index of the target operator in the business logic of n levels from the identifier of the target operator, wherein the business logic of the n levels runs on the host, each business logic of the i+1th level in the n levels has a subordinate relationship with a business logic of the i-th level, and the business logic of the nth level is bound one-to-one to the executable handle, which is used to execute the business logic of the operator running on the external computing unit, and i is greater than or equal to 1 and less than or equal to n-1.

[0161] In this embodiment, the host splits the target operator's identifier according to the length of the data fields corresponding to the n levels, obtaining n identifiers. These n identifiers are the indexes corresponding to the target operator in the business logic of the n levels. For example, if the n levels include three levels and the data fields of each of the n levels are 1 bit long, the 1-bit data obtained continuously from the beginning of the target operator's identifier is the index corresponding to the target operator in the business logic of the n levels.

[0162] Step 803 : Based on the index, search for the business logic corresponding to the target operator at the n levels, and search for the executable handle bound to the business logic of the target operator at the nth level.

[0163] In this embodiment, the host uses the determined index to determine the index of the target operator at n levels in the business logic of n levels step by step, and obtains the executable handle bound to the target operator at the business logic of the nth level.

[0164] In an alternative approach, an information table is used to determine the executable handle. The process is as follows:

[0165] Using the first-level index, the first-level information table is used to determine the information of the target operator's first business logic at the first level. From this information, the host's memory address for the first business logic and the address of the information subtable corresponding to the target operator's second-level business logic are retrieved. The host uses this address information to locate the first business logic in its memory. Using the address information of this information subtable, the host finds the information subtable. Within this information subtable, the second-level index is used to locate the second business logic of the target operator at the second level. From this information, the host's memory address for the second business logic and the address of the information subtable corresponding to the target operator's third business logic at the third level are retrieved. This process continues until the target operator's nth business logic at the nth level is located. From this information, the host's memory address for the nth business logic is retrieved. Using this address information, the host finds the nth business logic in its memory and retrieves the executable handle bound to the nth business logic.

[0166] In this way, the target operator runs the business logic of the first to nth levels on the host, and the external computing unit uses the executable handle to find the kernel function, execute the kernel function, and realize the execution of the target operator.

[0167] For example, see Figure 9. Taking the information table shown in Figure 6 as an example, the target operator is identified as 000, and its indexes at the Operation, Tactic, and Kernel levels are 0, 0, and 0, respectively. Information about the business logic identified as 0 is obtained at the Operation level. The address information of the Tactic level information subtable recorded in this information is used to obtain information about the business logic identified as 0 in this Tactic level information subtable. The address information of the Kernel level information subtable recorded in this information is used to obtain information about the business logic identified as 0 in this Kernel level information subtable.

[0168] By using the method for searching operators in the embodiment of the present application, there is no need to perform hash calculations and resolve hash conflicts, thereby achieving fast operator search and reducing the overall time for operator access and execution. Moreover, starting from the first-level information table, the address of the next-level information table is quickly obtained according to the index, and the executable handle of the kernel function on the external computing unit side can be accessed in a constant time. In this way, when the operator is called frequently in inference or training scenarios, the overall computing efficiency can be improved, and the performance stability can be improved.

[0169] In addition, during the hierarchical loading process of operators, the identifiers of each level of operators are allocated synchronously, which can achieve fast loading and subsequent indexing of operators.

[0170] In addition, this application is used for the underlying operator-level software stack of the heterogeneous development platform. The operator's calling interface can be obtained from the software development kit (SDK) calling interface header file provided by the chip, and infringement evidence can be collected based on the operator calling method described in the SDK calling interface header file.

[0171] The device provided in the embodiments of the present application is described below.

[0172] Figure 10 is a block diagram of a device for loading an operator provided in an embodiment of the present application. The device is applied to a host computer, which is externally connected to a computing unit. The device can be implemented as part or all of the device through software, hardware, or a combination of both. The device provided in an embodiment of the present application can implement the process described in Figure 5 of the embodiment of the present application. The device includes: a loading module 1010 and a search module 1020, wherein:

[0173] Loading module 1010, used to:

[0174] Loading a first business logic into the memory of the host, wherein the first business logic is the business logic of the operator to be loaded of the deep learning model running on the host, the first business logic is divided into n levels of business logic according to attributes, each business logic of the i+1th level in the n levels has a subordinate relationship with a business logic of the i-th level, the business logic of the nth level is bound to an executable handle one by one, and the executable handle is used to execute the business logic of the operator to be loaded running on the external computing unit, i is greater than or equal to 1 and less than or equal to n-1, which can be specifically used to implement the aforementioned process of loading the business logic running on the host;

[0175] The second business logic is loaded into the memory of the host, and the second business logic is loaded from the memory of the host into the memory of the external computing unit, wherein the second business logic is the business logic of the operator to be loaded running on the external computing unit, which can be specifically used to implement the aforementioned process of loading the business logic running on the external computing unit.

[0176] In an optional manner, the loading module 1010 is further configured to:

[0177] Assigning consecutive identifiers to all business logics of the first level among the n levels;

[0178] A continuous identifier is assigned to the business logic of the (i+1)th level that is subordinate to any business logic of the (i)th level.

[0179] In an optional manner, the search module 1020 is configured to:

[0180] Obtaining the identifier of the target operator in the operators to be loaded;

[0181] From the identifier of the target operator, obtain the index of the target operator in the business logic of the n levels;

[0182] Based on the index, the business logic corresponding to the target operator at the n levels is searched, and the executable handle bound to the business logic of the target operator at the nth level is searched.

[0183] In an optional manner, the search module 1020 is configured to obtain the cached identifier of the target operator.

[0184] In an optional manner, the search module 1020 is further used to determine the optimal business logic of the target operator step by step in the business logic of the n levels based on the parameter information of the target operator before obtaining the cached identifier of the target operator, wherein the parameter information includes one or more specific parameters, data formats or data types, and tensor shapes used by the target operator, and the identifier of the optimal business logic of the target operator is composed of the identifier of the target operator in the hierarchical order of the n levels; and the identifier of the target operator is cached.

[0185] In an optional manner, the search module 1020 is configured to:

[0186] Obtaining n information tables corresponding to the n levels of business logic, wherein the information table corresponding to the i-th level of business logic includes information about each business logic of the i-th level. For each business logic of the i-th level, the business logic information includes storage address information of the business logic and address information of an information subtable. The information subtable includes information about business logic of the (i+1)th level subordinate to the business logic. The information table of the n-th level includes information about each business logic of the n-th level. For each business logic of the n-th level, the business logic information includes storage address information of the business logic and an executable handle bound to the business logic.

[0187] Based on the n information tables and the index, the business logic corresponding to the target operator at the n levels is searched, and the executable handle bound to the business logic of the target operator at the nth level is searched.

[0188] Figure 11 is a block diagram of a device for searching an operator provided in an embodiment of the present application. The device is applied to a host computer, which is externally connected to a computing unit. The device can be implemented as part or all of the device through software, hardware, or a combination of both. The device provided in an embodiment of the present application can implement the process described in Figure 8 of the embodiment of the present application. The device includes: an acquisition module 1110 and a determination module 1120, wherein:

[0189] The acquisition module 1110 is used to: acquire the identifier of the target operator;

[0190] From the identifier of the target operator, obtain the index of the target operator in the business logic of n levels, wherein the business logic of the n levels runs on the host, each business logic of the i+1th level in the n levels has a subordinate relationship with a business logic of the ith level, the business logic of the nth level is bound one-to-one to the executable handle, and the executable handle is used to execute the business logic of the operator running on the external computing unit, i is greater than or equal to 1 and less than or equal to n-1, and can be specifically used to implement the acquisition functions of steps 801 and 802 and execute the implicit steps included in steps 801 and 802;

[0191] Determination module 1120 is used to find the business logic corresponding to the target operator at the n levels based on the index, and to find the executable handle bound to the business logic of the target operator at the nth level. It can be specifically used to implement the determination function of step 803 and execute the implicit steps contained in step 803.

[0192] In an optional manner, the obtaining module 1110 is configured to:

[0193] Get the identifier of the cached target operator.

[0194] In an optional manner, the acquisition module 1110 is further configured to:

[0195] Before obtaining the cached identifier of the target operator, based on parameter information of the target operator, determine the optimal business logic of the target operator step by step in the business logic of the n levels, wherein the parameter information includes one or more specific parameters, data formats or data types, and tensor shapes used by the target operator, and the identifiers of the optimal business logic of the target operator constitute the identifier of the target operator in the hierarchical order of the n levels; and cache the identifier of the target operator.

[0196] In an optional manner, the determining module 1120 is configured to:

[0197] Obtaining n information tables corresponding to the n levels of business logic, wherein the information table corresponding to the i-th level of business logic includes information about each business logic of the i-th level. For each business logic of the i-th level, the business logic information includes storage address information of the business logic and address information of an information subtable. The information subtable includes information about business logic of the (i+1)th level subordinate to the business logic. The information table of the n-th level includes information about each business logic of the n-th level. For each business logic of the n-th level, the business logic information includes storage address information of the business logic and an executable handle bound to the business logic.

[0198] Based on the n information tables and the index, the business logic corresponding to the target operator at the n levels is searched, and the executable handle bound to the business logic of the target operator at the nth level is searched.

[0199] In an optional manner, the device further includes:

[0200] A loading module is used to load the business logic of the operator to be loaded of the deep learning model running on the host into the memory of the host, wherein the business logic of the n levels is obtained by dividing the business logic of the operator to be loaded running on the host according to attributes, and the target operator belongs to the operator to be loaded; the business logic running on the operator to be loaded on the external computing unit is loaded into the memory of the host, and the business logic running on the operator to be loaded on the external computing unit is loaded from the memory of the host to the external computing unit; a continuous identifier is assigned to all business logics of the first level of the n levels; and a continuous identifier is assigned to the business logic of the i+1th level that is subordinate to any business logic of the i-th level.

[0201] In an optional manner, the loading module is used to load the business logic of all operators included in the deep learning model running on the host into the memory of the host when configuring the deep learning model or running the deep learning model.

[0202] The detailed process of loading an operator by the apparatus for loading an operator shown in FIG10 and the detailed process of searching an operator by the apparatus for searching an operator shown in FIG11 are described in the previous embodiments and will not be repeated here. The apparatus for loading an operator shown in FIG10 and the apparatus for searching an operator shown in FIG11 are the computing device 200 in FIG2 .

[0203] In some embodiments, embodiments of the present application provide a computer program product comprising computer instructions stored in a computer-readable storage medium. A processor of a computing device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computing device to execute the processes shown in Figures 3, 4, and 8.

[0204] Those skilled in the art will appreciate that the various method steps and units described in the embodiments disclosed in this application can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the steps and components of each embodiment have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0205] In the several embodiments provided in this application, it should be understood that the disclosed system architecture, device and method can be implemented in other ways. For example, the device embodiment described above is only schematic. For example, the division of the module is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or modules, or it can be an electrical, mechanical or other form of connection.

[0206] The modules described as separate components may or may not be physically separate, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the embodiments of the present application.

[0207] In addition, each module in each embodiment of the present application can be integrated into a processing module, each module can exist physically separately, or two or more modules can be integrated into a module. The above-mentioned integrated modules can be implemented in the form of hardware or software modules.

[0208] If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a ROM, a RAM, a magnetic disk, or an optical disk.

[0209] The above description is merely an exemplary embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and such modifications or substitutions should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. A method for loading an operator, characterized in that: The method is applied to a host, the host is connected to an external computing unit, and the method comprises: Loading a first business logic into the memory of the host, wherein the first business logic is the business logic of the operator to be loaded of the deep learning model running on the host, the first business logic is divided into n levels of business logic according to attributes, each business logic of the i+1th level in the n levels is subordinate to a business logic of the ith level, the business logic of the nth level is bound to an executable handle one by one, and the executable handle is used to execute the business logic of the operator to be loaded running on the external computing unit, i is greater than or equal to 1 and less than or equal to n-1; The second business logic is loaded into the memory of the host, and the second business logic is loaded from the memory of the host into the memory of the external computing unit, wherein the second business logic is the business logic of the operator to be loaded running on the external computing unit.

2. The method according to claim 1, characterized in that The method further comprises: Allocating continuous identifiers to all business logics of the first level among the n levels; A continuous identifier is allocated to the business logic of the (i+1)th level that is subordinate to any business logic of the (i)th level.

3. The method according to claim 1 or 2, characterized in that: The method further comprises: Obtaining an identifier of a target operator in the operators to be loaded; From the identifier of the target operator, obtain the index of the target operator in the business logic of the n levels; Based on the index, the business logic corresponding to the target operator at the n levels is searched, and the executable handle bound to the business logic of the target operator at the nth level is searched.

4. The method according to claim 3, characterized in that The obtaining the identifier of the target operator in the operator to be loaded includes: Get the identifier of the cached target operator.

5. The method according to claim 4, characterized in that Before obtaining the identifier of the cached target operator, the method further includes: Based on the parameter information of the target operator, determine the optimal business logic of the target operator in the n-level business logic step by step, wherein the parameter information includes one or more specific parameters, data formats or data types, and tensor shapes used by the target operator, and the identifier of the optimal business logic of the target operator constitutes the identifier of the target operator according to the hierarchical order of the n levels; The identifier of the target operator is cached.

6. The method according to any one of claims 3 to 5, characterized in that: The searching, based on the index, for the business logic corresponding to the target operator at the n levels, and searching for the executable handle bound to the business logic of the target operator at the nth level, includes: Obtain n information tables corresponding to the business logics of the n levels, wherein the information table corresponding to the business logic of the i-th level includes information of each business logic of the i-th level, and for each business logic of the i-th level, the information of the business logic includes storage address information of the business logic and address information of an information subtable, and the information subtable includes information of business logic of the i+1th level subordinate to the business logic, and the information table of the n-th level includes information of each business logic of the n-th level, and for each business logic of the n-th level, the information of the business logic includes storage address information of the business logic and an executable handle bound to the business logic; Based on the n information tables and the index, the business logic corresponding to the target operator at the n levels is searched, and the executable handle bound to the business logic of the target operator at the nth level is searched.

7. A device for loading operators, characterized in that: The device is applied to a host, the host is connected to an external computing unit, and the device includes: Load modules for: Loading a first business logic into the memory of the host, wherein the first business logic is the business logic of the operator to be loaded of the deep learning model running on the host, the first business logic is divided into n levels of business logic according to attributes, each business logic of the i+1th level in the n levels is subordinate to a business logic of the ith level, the business logic of the nth level is bound to an executable handle one by one, and the executable handle is used to execute the business logic of the operator to be loaded running on the external computing unit, i is greater than or equal to 1 and less than or equal to n-1; The second business logic is loaded into the memory of the host, and the second business logic is loaded from the memory of the host into the memory of the external computing unit, wherein the second business logic is the business logic of the operator to be loaded running on the external computing unit.

8. The device according to claim 7, characterized in that The loading module is further used for: Allocating continuous identifiers to all business logics of the first level among the n levels; A continuous identifier is allocated to the business logic of the (i+1)th level that is subordinate to any business logic of the (i)th level.

9. The device according to claim 7 or 8, characterized in that The device also includes a search module, which is used to: Obtaining an identifier of a target operator in the operators to be loaded; From the identifier of the target operator, obtain the index of the target operator in the business logic of the n levels; Based on the index, the business logic corresponding to the target operator at the n levels is searched, and the executable handle bound to the business logic of the target operator at the nth level is searched.

10. The device according to claim 9, characterized in that The search module is used to obtain the identifier of the cached target operator.

11. The device according to claim 10, characterized in that The search module is further used to determine the optimal business logic of the target operator step by step in the business logic of the n levels based on the parameter information of the target operator before obtaining the identifier of the cached target operator, wherein the parameter information includes one or more of the specific parameters, data formats or data types, and tensor shapes used by the target operator, and the identifier of the optimal business logic of the target operator is composed of the identifier of the target operator in the hierarchical order of the n levels; and the identifier of the target operator is cached.

12. The device according to any one of claims 9 to 11, characterized in that The search module is used to: Obtain n information tables corresponding to the business logics of the n levels, wherein the information table corresponding to the business logic of the i-th level includes information of each business logic of the i-th level, and for each business logic of the i-th level, the information of the business logic includes storage address information of the business logic and address information of an information subtable, and the information subtable includes information of business logic of the i+1th level subordinate to the business logic, and the information table of the n-th level includes information of each business logic of the n-th level, and for each business logic of the n-th level, the information of the business logic includes storage address information of the business logic and an executable handle bound to the business logic; Based on the n information tables and the index, the business logic corresponding to the target operator at the n levels is searched, and the executable handle bound to the business logic of the target operator at the nth level is searched.

13. A method for searching an operator, characterized in that: The method is applied to a host, the host is connected to an external computing unit, and the method comprises: Get the target operator's ID; In the identifier of the target operator, an index of the target operator in the business logic of n levels is obtained, wherein the business logic of the n levels is run on the host, each business logic of the i+1th level in the n levels is subordinate to a business logic of the ith level, the business logic of the nth level is bound to an executable handle one by one, and the executable handle is used to execute the business logic of the operator running on the external computing unit, and i is greater than or equal to 1 and less than or equal to n-1; Based on the index, the business logic corresponding to the target operator at the n levels is searched, and the executable handle bound to the business logic of the target operator at the nth level is searched.

14. The method according to claim 13, characterized in that The obtaining the identifier of the target operator includes: Get the identifier of the cached target operator.

15. The method according to claim 14, characterized in that Before obtaining the identifier of the cached target operator, the method further includes: Based on the parameter information of the target operator, determine the optimal business logic of the target operator in the n-level business logic step by step, wherein the parameter information includes one or more specific parameters, data formats or data types, and tensor shapes used by the target operator, and the identifier of the optimal business logic of the target operator constitutes the identifier of the target operator according to the hierarchical order of the n levels; The identifier of the target operator is cached.

16. The method according to any one of claims 13 to 15, characterized in that The searching, based on the index, for the business logic corresponding to the target operator at the n levels, and searching for the executable handle bound to the business logic of the target operator at the nth level, includes: Obtain n information tables corresponding to the business logics of the n levels, wherein the information table corresponding to the business logic of the i-th level includes information of each business logic of the i-th level, and for each business logic of the i-th level, the information of the business logic includes storage address information of the business logic and address information of an information subtable, and the information subtable includes information of business logic of the i+1th level subordinate to the business logic, and the information table of the n-th level includes information of each business logic of the n-th level, and for each business logic of the n-th level, the information of the business logic includes storage address information of the business logic and an executable handle bound to the business logic; Based on the n information tables and the index, find the business logic corresponding to the target operator at the n levels, and find the target The executable handle of the operator bound to the business logic at the nth level.

17. A device for searching an operator, characterized in that: The device is applied to a host, the host is connected to an external computing unit, and the device includes: An acquisition module is used to obtain the identifier of the target operator; In the identifier of the target operator, an index of the target operator in the business logic of n levels is obtained, wherein the business logic of the n levels is run on the host, each business logic of the i+1th level in the n levels is subordinate to a business logic of the ith level, the business logic of the nth level is bound to an executable handle one by one, and the executable handle is used to execute the business logic of the operator running on the external computing unit, and i is greater than or equal to 1 and less than or equal to n-1; A determination module is used to search for the business logic corresponding to the target operator at the n levels based on the index, and to search for the executable handle bound to the business logic of the target operator at the nth level.

18. The device according to claim 17, characterized in that The acquisition module is used to acquire the identifier of the cached target operator.

19. The device according to claim 18, characterized in that The acquisition module is further used to determine the optimal business logic of the target operator step by step in the business logic of the n levels based on the parameter information of the target operator before acquiring the identifier of the cached target operator, wherein the parameter information includes one or more of the specific parameters, data formats or data types, and tensor shapes used by the target operator, and the identifier of the optimal business logic of the target operator constitutes the identifier of the target operator in the hierarchical order of the n levels; and cache the identifier of the target operator.

20. The device according to any one of claims 17 to 19, characterized in that The determining module is used to: Obtain n information tables corresponding to the business logics of the n levels, wherein the information table corresponding to the business logic of the i-th level includes information of each business logic of the i-th level, and for each business logic of the i-th level, the information of the business logic includes storage address information of the business logic and address information of an information subtable, and the information subtable includes information of business logic of the i+1th level subordinate to the business logic, and the information table of the n-th level includes information of each business logic of the n-th level, and for each business logic of the n-th level, the information of the business logic includes storage address information of the business logic and an executable handle bound to the business logic; Based on the n information tables and the index, the business logic corresponding to the target operator at the n levels is searched, and the executable handle bound to the business logic of the target operator at the nth level is searched.

21. A computing device, characterized in that It includes a processor and a memory, wherein the processor is used to execute instructions stored in the memory so that the computing device executes the method for loading an operator as described in any one of claims 1 to 6, or the method for searching an operator as described in any one of claims 13 to 16.

22. A computer-readable storage medium, characterized in that: The method comprises computer program instructions. When the computer program instructions are executed by a computing device, the computing device executes the method for loading an operator as claimed in any one of claims 1 to 6, or the method for searching an operator as claimed in any one of claims 13 to 16.

Images