Method and device for loading operator, method and device for searching operator, equipment and medium

Abstract

The invention provides an operator loading method and device, an operator searching method and device, equipment and a medium, and belongs to the technical field of deep learning. The method is applied to a host, the host is externally connected with a computing unit, and the method comprises the following steps: acquiring an identifier of a target operator, acquiring an index of the target operator in business logic of n levels in the identifier of the target operator, searching the business logic corresponding to the target operator in the n levels based on the index, and searching an executable handle bound with the business logic of the target operator at the nth level. According to the scheme, the operators are searched in a graded mode, the operators can be quickly searched, and the efficiency of executing the operators is improved.

Description

Technical Field

[0001] The present disclosure relates to the field of deep learning technology, and in particular to a method for loading an operator, a method for searching an operator, an apparatus, a device and a medium. Background Art

[0002] In order to improve the execution efficiency of operators, it is considered to use the host and computing unit to coordinate computing and accelerate each other. In this way, the business logic of the operator needs to run on the host and computing unit. The computing unit includes external computing units such as neural processor unit (NPU), graphics processing unit (GPU) and digital signal processor (DSP), which are plugged into the host. Summary of the invention

[0003] The present disclosure provides a method for loading an operator, a method for searching an operator, an apparatus, a device and a medium, which can enable the business logic of the operator to run on a host and a computing unit.

[0004] In a first aspect, the present disclosure provides a method for loading an operator, the method being applied to a host, the host being connected to an external computing unit, the method comprising: loading a first business logic into the memory of the host, wherein the first business logic is the business logic of the operator to be loaded of a deep learning model running on the host, the first business logic being divided into n levels of business logic according to attributes, each business logic of the i+1th level of the n levels being subordinate to a business logic of the ith level, the business logic of the nth level being bound one-to-one to an executable handle, the executable handle being used to execute the business logic of the operator to be loaded running on the external computing unit, i being greater than or equal to 1 and less than or equal to n-1, loading a second business logic into the memory of the host, and loading the second business logic from the memory of the host into the memory of the external computing unit, wherein the second business logic is the business logic of the operator to be loaded running on the external computing unit.

[0005] In the solution disclosed in the present invention, the business logic running on the host is hierarchical, and the business logic of the lowest level is bound to an executable handle, so that when the operator is subsequently executed, the business logic running on the operator on the host can be found and executed according to the level, and the executable handle is found using the business logic of the lowest level, so that the business logic running on the operator on the external computing unit can be executed, thereby realizing the execution of the operator on the host and the external computing unit.

[0006] In an optional manner, the method further includes: assigning continuous identifiers to all business logics of the first level among the n levels, and assigning continuous identifiers to business logics of the i+1th level subordinate to any business logic of the ith level. Thus, by assigning identifiers in this manner, when searching for operators, the search can be performed in sequence, thus saving the time of searching for operators.

[0007] In an optional manner, the method further includes: obtaining the identifier of the target operator in the operator to be loaded, obtaining the index of the target operator in the business logic of the n levels from the identifier of the target operator, searching for the business logic corresponding to the target operator at the n levels based on the index, and searching for the executable handle bound to the business logic of the target operator at the nth level.

[0008] In the solution disclosed in the present disclosure, the operator identifier is used to determine the index in the business logic of n levels, and based on the index and the relationship between the business logic of the adjacent levels, the business logic corresponding to the operator at n levels is found, and the executable handle bound to the business logic of the operator at the nth level is found. In this way, there is no need to perform hash calculations and resolve hash conflicts, thereby achieving fast operator search and reducing the overall time for operator access execution.

[0009] In an optional manner, obtaining the identifier of the target operator in the operators to be loaded includes: obtaining the identifier of the target operator in a cache. In this way, when the identifier of the operator is already available, there is no need to determine the identifier of the operator, which can save the time of searching for the operator.

[0010] In an optional manner, before obtaining the cached identifier of the target operator, it also includes: based on the parameter information of the target operator, determining the optimal business logic of the target operator step by step in the business logic of the n levels, wherein the parameter information includes one or more of the specific parameters, data formats or data types used by the target operator, and the tensor shape, the identifier of the optimal business logic of the target operator is composed of the identifier of the target operator in the hierarchical order of the n levels, and the identifier of the target operator is cached. In the scheme shown in the present disclosure, the parameter information of the operator is used to search and cache the identifier of the operator, so that when the operator is searched for later, there is no need to determine the identifier of the operator, which can save the time of searching the operator.

[0011] In an optional manner, based on the index, searching for the business logic corresponding to the target operator at the n levels, and searching for the executable handle bound to the business logic at the n level of the target operator, includes: obtaining n information tables corresponding to the business logic at the n levels, wherein the information table corresponding to the business logic at the i-th level includes information of each business logic at the i-th level, for each business logic at the i-th level, the information of the business logic includes storage address information of the business logic and address information of an information subtable, the information subtable includes information of the business logic at the i+1th level subordinate to the business logic, the information table at the n-th level includes information of each business logic at the n-th level, for each business logic at the n-th level, the information of the business logic includes storage address information of the business logic and the executable handle bound to the business logic, based on the n information tables and the index, searching for the business logic corresponding to the target operator at the n levels, and searching for the executable handle bound to the business logic at the n-th level of the target operator.

[0012] In the solution disclosed in the present invention, the business logic of n levels corresponds to an information table. Since the information of the business logic is recorded in the information table, the business logic corresponding to the operator at n levels can be quickly found by using the information table and the index, and the executable handle bound to the business logic of the operator at the nth level can be found, thereby saving the time of searching the operator.

[0013] In a second aspect, the present disclosure provides a method for searching an operator, which is applied to a host, and the host is connected to an external computing unit. The method includes: obtaining an identifier of a target operator; in the identifier of the target operator, obtaining an index of the target operator in a business logic of n levels, wherein the business logic of the n levels runs on the host, and each business logic of the i+1th level in the n levels is subordinate to a business logic of the i-th level, and the business logic of the n-th level is bound one by one to an executable handle, and the executable handle is used to execute the business logic of the operator running on the external computing unit, i is greater than or equal to 1 and less than or equal to n-1; based on the index, searching for the business logic corresponding to the target operator at the n levels, and searching for the executable handle bound to the business logic of the target operator at the n-th level.

[0014] In the solution disclosed in the present invention, the business logic running on the host is hierarchical, and the business logic of the lowest level is bound to an executable handle, so that when the operator is subsequently executed, the business logic running on the operator on the host can be found and executed according to the level, and the executable handle is found using the business logic of the lowest level, so that the business logic running on the operator on the external computing unit can be executed, thereby realizing the execution of the operator on the host and the external computing unit. In addition, the identifier of the operator is used to determine the index in the business logic of n levels, and based on the index and the relationship between the business logic of the adjacent levels, the business logic corresponding to the operator at n levels is found, and the executable handle bound to the business logic of the operator at the nth level is found. In this way, there is no need to perform hash calculations and resolve hash conflicts, thereby achieving fast search for operators, thereby reducing the overall time for operator access execution.

[0015] In an optional manner, obtaining the identifier of the target operator includes: obtaining the identifier of the target operator in a cache. In this way, when the identifier of the operator is already available, there is no need to determine the identifier of the operator, which can save the time of searching for the operator.

[0016] In an optional manner, before obtaining the cached identifier of the target operator, it also includes: based on the parameter information of the target operator, determining the optimal business logic of the target operator step by step in the business logic of the n levels, wherein the parameter information includes one or more of the specific parameters, data formats or data types, and tensor shapes used by the target operator, the identifier of the optimal business logic of the target operator is composed of the identifier of the target operator in the hierarchical order of the n levels, and the identifier of the target operator is cached.

[0017] In the solution shown in the present disclosure, the operator's parameter information is used to search and cache the operator's identifier, so that when the operator is searched for later, there is no need to determine the operator's identifier, which can save the time of searching for the operator.

[0018] In an optional manner, based on the index, searching for the business logic corresponding to the target operator at the n levels, and searching for the executable handle bound to the business logic at the n level of the target operator, includes: obtaining n information tables corresponding to the business logic at the n levels, wherein the information table corresponding to the business logic at the i-th level includes information of each business logic at the i-th level, and for each business logic at the i-th level, the information of the business logic includes storage address information of the business logic and address information of an information subtable, and the information subtable includes information of the business logic at the i+1th level subordinate to the business logic, and the information table at the n-th level includes information of each business logic at the n-th level, and for each business logic at the n-th level, the information of the business logic includes storage address information of the business logic and the executable handle bound to the business logic; based on the n information tables and the index, searching for the business logic corresponding to the target operator at the n levels, and searching for the executable handle bound to the business logic at the n-th level of the target operator.

[0019] In the solution disclosed in the present invention, the business logic of n levels corresponds to an information table. Since the information of the business logic is recorded in the information table, the business logic corresponding to the operator at n levels can be quickly found by using the information table and the index, and the executable handle bound to the business logic of the operator at the nth level can be found, thereby saving the time of searching the operator.

[0020] In an optional manner, the method also includes: loading the business logic of the operator to be loaded of the deep learning model running on the host into the memory of the host, wherein the business logic of the n levels is obtained by dividing the business logic of the operator to be loaded on the host according to attributes, and the target operator belongs to the operator to be loaded; loading the business logic of the operator to be loaded on the external computing unit into the memory of the host, and loading the business logic of the operator to be loaded on the external computing unit from the memory of the host to the external computing unit; assigning continuous identifiers to all business logics of the first level among the n levels; and assigning continuous identifiers to the business logic of the i+1th level that is subordinate to any business logic of the i-th level.

[0021] The solution shown in the present disclosure loads the business logic of the operators used by the deep learning model to the running location in advance, and assigns identifiers in this way. When searching for operators, they can be searched in sequence, saving the time of searching for operators.

[0022] In an optional manner, the business logic of the operators to be loaded of the deep learning model running on the host is loaded into the memory of the host, including: when configuring the deep learning model or running the deep learning model, the business logic of all operators included in the deep learning model running on the host is loaded into the memory of the host.

[0023] In a third aspect, the present disclosure provides a device for loading an operator, the device comprising at least one module, and the at least one module is used to implement the method for loading an operator provided by the first aspect or any optional manner in the first aspect.

[0024] In some embodiments, the module in the device for loading an operator is implemented by software, and the module in the device for loading an operator is a program module. In other embodiments, the module in the device for loading an operator is implemented by hardware or firmware.

[0025] In a fourth aspect, the present disclosure provides a device for searching operators, the device comprising at least one module, and the at least one module is used to implement the method for searching operators provided by the second aspect or any optional manner in the second aspect.

[0026] In some embodiments, the module in the means for searching operators is implemented by software, and the module in the means for searching operators is a program module. In other embodiments, the module in the means for searching operators is implemented by hardware or firmware.

[0027] In a fifth aspect, the present disclosure provides a computing device, comprising a memory and a processor, wherein the processor of the computing device is used to execute instructions stored in the memory of the computing device, so that the computing device executes the method for loading operators provided in the above-mentioned first aspect or any optional manner in the first aspect, or so that the computing device executes the method for searching operators provided in the above-mentioned second aspect or any optional manner in the second aspect.

[0028] In a sixth aspect, the present disclosure provides a computer-readable storage medium, which includes computer program instructions. When the computer program instructions are executed by a computing device, the computing device executes the method for loading operators provided in the first aspect or any optional manner in the first aspect, or the computing device executes the method for searching operators provided in the second aspect or any optional manner in the second aspect.

[0029] In a seventh aspect, the present disclosure provides a computer program product comprising instructions, which, when executed by a computing device, enables the computing device to execute the method for loading operators provided in the first aspect or any optional manner in the first aspect, or enables the computing device to execute the method for searching operators provided in the second aspect or any optional manner in the second aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 This is a schematic diagram of the traditional method of searching operators;

[0031] Figure 2is a schematic diagram of the structure of a computing device provided by an exemplary embodiment of the present disclosure;

[0032] Figure 3 is a schematic diagram of a pre-staged loading operator provided by an exemplary embodiment of the present disclosure;

[0033] Figure 4 is a schematic diagram of a delayed hierarchical loading operator provided by an exemplary embodiment of the present disclosure;

[0034] Figure 5 is a schematic diagram of a hierarchical loading operator provided by an exemplary embodiment of the present disclosure;

[0035] Figure 6 is a schematic diagram of an information table provided by an exemplary embodiment of the present disclosure;

[0036] Figure 7 is a schematic diagram of calling a target operator for the first time provided by an exemplary embodiment of the present disclosure;

[0037] Figure 8 is a flowchart of a search operator provided by an exemplary embodiment of the present disclosure;

[0038] Fig. 9 is a flowchart of a search operator provided by an exemplary embodiment of the present disclosure;

[0039] Fig.10 is a structural schematic diagram of a device for loading operators provided by an exemplary embodiment of the present disclosure;

[0040] Fig.11 It is a structural diagram of a device for searching operators provided by an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0041] In order to make the objectives, technical solutions and advantages of the present disclosure more clear, the embodiments of the present disclosure will be further described in detail below with reference to the accompanying drawings.

[0042] Some terminology concepts involved in the embodiments of the present disclosure are explained below.

[0043] 1. Operators are functions used to perform various mathematical operations and operations in deep learning models. When the host and computing unit collaborate and accelerate each other (that is, in the case of heterogeneous programming), they usually include the business logic on the host side and the business logic on the external device side. The business logic on the host side runs on the host, and the business logic on the device side runs on external computing units such as NPU, GPU or DSP. The external computing unit is connected to the host through a bus. The business logic on the host side is the binary code that the operator runs on the host, and the business logic on the device side is the binary code that the operator runs on the external computing unit.

[0044] 2. The executable handle of the operator is the handle used by the Device side to execute the operator kernel function. The kernel function can be found and executed through this handle.

[0045] The background of the present disclosure and conventional techniques are described below.

[0046] Operators are the bottom layer of the entire application software stack. Operators are used by various accelerator cards, inference engines, and training frameworks. Operators need to be quickly found and run, especially in some inference scenarios, which have strict requirements on operator search time. Figure 1 When searching for an operator, a hash calculation is performed using the global identifier of the operator, and the operator is found in the hash table based on the hash value after the hash calculation. Since the search needs to be performed through the hash table, there are overheads for hash calculation and hash conflict resolution. When the number of operators is large, the time-consuming overhead for resolving hash conflicts increases rapidly. Moreover, hash search is not a stable constant time, which brings about time uncertainty. Based on this, in the embodiment of the present disclosure, the identifier of the operator is used to split out the index of the operator in the business logic of n levels, and the index at each level is used to find the executable handle of the operator, thereby executing the operator. Since there is no need to perform hash calculation or resolve hash conflicts, the search time of the operator can be saved.

[0047] The following describes the execution body of the method for loading an operator and the execution body of the method for searching an operator.

[0048] The execution subject of the method for loading an operator is a device for loading an operator, which is a host mentioned later, and the host is a computing device such as a terminal or a server. Alternatively, the device is a software device, such as a set of software programs running on the host.

[0049] The execution subject of the method for searching operators is a device for searching operators, which is a host mentioned later, and the host is a computing device such as a terminal or a server. Alternatively, the device is a software device, such as a set of software programs running on the host.

[0050] like Figure 2 As shown, a computing device 200 is provided, and the computing device 200 is optionally implemented by a general bus architecture. The computing device 200 includes at least one processor 201 , a communication bus 202 , a memory 203 , and at least one network interface 204 .

[0051] The processor 201 is, for example, a general-purpose central processing unit (CPU), a network processor (NP), a graphics processing unit (GPU), a neural-network processing unit (NPU), a data processing unit (DPU), a microprocessor, or one or more integrated circuits for implementing the disclosed solution. For example, the processor 201 includes an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The PLD is, for example, a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0052] The communication bus 202 is used to transmit information between the above components. The communication bus 202 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 2 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0053] The memory 203 is, for example, a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, or a random access memory (RAM) or other types of dynamic storage devices that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 203 is, for example, independent and connected to the processor 201 via the communication bus 202. The memory 203 can also be integrated with the processor 201.

[0054] Optionally, the memory 203 is used to store the information table mentioned below. When the computing device 200 needs to use the information table, the processor 201 accesses the memory 203 to obtain the information table stored in the memory 203 .

[0055] The network interface 204 uses any transceiver-like device for communicating with other devices or communication networks. The network interface 204 includes a wired network interface and may also include a wireless network interface. The wired network interface may be, for example, an Ethernet interface. The Ethernet interface may be an optical interface, an electrical interface, or a combination thereof. The wireless network interface may be a wireless local area network (WLAN) interface, a cellular network network interface, or a combination thereof, etc.

[0056] In a specific implementation, as an example, the processor 201 may include one or more CPUs.

[0057] In a specific implementation, as an example, the computing device 200 may include multiple processors. Each of these processors may be a single-core processor (single-CPU) or a multi-core processor (multi-CPU). The processor here may refer to one or more devices, circuits, and / or processing cores for processing data (such as computer program instructions).

[0058] In a specific implementation, as an example, the computing device 200 may also include an output device and an input device. The output device communicates with the processor 201 and may display information in a variety of ways. For example, the output device may be a liquid crystal display (LCD), a light emitting diode (LED) display device, a cathode ray tube (CRT) display device, or a projector. The input device communicates with the processor 201 and receives user input in a variety of ways. For example, the input device may be a mouse, a keyboard, a touch screen device, or a sensor device.

[0059] In some embodiments, the memory 203 is used to store a program code 2031 for executing the search operator or the load operator in the present disclosure, and the processor 201 executes the program code 2031 stored in the memory 203. That is, the computing device 200 can implement the method for searching an operator or loading an operator provided in the method embodiment through the processor 201 and the program code 2031 in the memory 203.

[0060] The following describes the application scenarios of the method for loading operators and the method for searching operators.

[0061] In the disclosed embodiment, the application scenario includes scenario one and scenario two. Scenario one is a scenario of preloading operators, that is, loading operators when configuring deep learning models, and scenario two is a scenario of delayed loading, that is, first configuring deep learning models, and loading operators when executing operators. In scenarios one and two, the interactive process of the three involved in the application, deep neural network (deep neuralnetworks, DNN) acceleration library and operator library, the application, DNN acceleration library and operator library are all software programs, all deployed on the host, and the application is a program that uses deep learning models to perform calculations. The DNN acceleration library can provide highly optimized implementations of common operations such as convolution, pooling, normalization and activation, which can accelerate calculations. The host is connected to one or more external computing units, and the types of the multiple external computing units are exactly the same, for example, the multiple external computing units are all GPUs, or there are different types of external computing units in the multiple external computing units, for example, the multiple external computing units include GPUs and NPUs.

[0062] Scenario 1: Preloading operators. Figure 3 The preloaded operator and the process of executing the operator are shown.

[0063] In an offline reasoning scenario or training scenario, load a deep learning model, which is any reasoning model, such as an image classification model or a speech recognition model, and start loading the operators included in the deep learning model.

[0064] Step 301: The application loads a file of a deep learning model. The file of the deep learning model is stored in the open neural network exchange (ONNX) format.

[0065] Among them, the application is a program that uses a deep learning model to perform calculations in an offline reasoning scenario or a training scenario.

[0066] Step 302: The application calls the Setup interface to instruct the DNN acceleration library to configure the deep learning model.

[0067] Step 303, the DNN acceleration library creates a context (create Context), sets a stream (set Stream), and sets an allocator (set Allocator).

[0068] Among them, Stream is the task queue of the external computing unit. Tasks are executed in the order of Stream. Here, the task can be the kernel function execution operation of the operator. Set Allocator is used to specify the memory allocation strategy.

[0069] Step 304: The DNN acceleration library calls the loading interface to instruct the operator library to preload the operators used in the operator library.

[0070] The operator library corresponds to an operator list, which records the information of all operators included in the operator library. The operators used in the operator library include all operators used by the deep learning model, but there may be more operators than all operators used by the deep learning model.

[0071] Step 305: The operator library loads operators in a hierarchical manner.

[0072] In this embodiment, the operator library loads the full amount of business logic on the host side and the device side into the host's memory. The business logic on the host side mainly includes the logic of operator execution preparation and the logic of the search strategy of the operator's executable handle. For example, the operator execution preparation includes the derivation of the shape of the tensor and data segmentation. Then the operator library loads the business logic on the device side into the memory of each external computing unit. For the detailed loading process, see the description below. Among them, the business logic on the host side refers to the business logic of the operator running on the host, and the business logic on the device side refers to the business logic of the operator running on the external computing unit.

[0073] It should be noted that in the scenario of preloading operators, the business logic of the Device side is loaded into each external computing unit. The reason is: it is currently unknown on which external computing unit the subsequent business logic of the Device side will run, so each external computing unit has all the business logic of the Device side. In this way, when running the business logic of the Device side subsequently, there is no need to reload it.

[0074] Step 306: The operator library returns a return value indicating that loading is complete to the DNN acceleration library.

[0075] Step 307: The DNN acceleration library returns a return value to the application indicating that the configuration is complete.

[0076] In step 308, the application provides the DNN acceleration library with input / output description (input / output desc), convolution description (conv desc), and input / output memory (i / o mem) and other computation-related description information.

[0077] The input / output description includes the format of the input / output data, etc., and the input / output memory refers to the memory occupied by the input / output data.

[0078] Step 309: The DNN acceleration library calls the Get Kernel interface to obtain the kernel function of the target operator from the operator library.

[0079] The interface for obtaining the kernel function is called to obtain the kernel function of the target operator, where the target operator is any operator of the index. The above description information is carried when the interface for obtaining the kernel function is called.

[0080] Step 310: The operator library determines the optimal business logic of the target operator among the business logics at each level, determines the identifier of the target operator, and returns the identifier of the target operator to the DNN acceleration library.

[0081] In this embodiment, the operator library uses the description information to determine the optimal business logic of the target operator in each level of business logic, and the detailed process is described later. The operator library composes the identifier of the target operator according to the hierarchical order of the optimal business logic.

[0082] It should be noted that the operator library executes the business logic at each level after determining the business logic at each level running on the host. For example, the business logic on the host is divided into three levels, namely the operation level, the tactic level, and the kernel level. The operator library first determines the business logic at the operation level, and then executes the business logic at the operation level. After determining the business logic at the tactic level, it executes the business logic at the tactic level. After determining the business logic at the kernel level, it executes the business logic at the kernel level.

[0083] Step 311, the DNN acceleration library caches the identifier of the target operator.

[0084] In step 311, after the DNN acceleration library caches the identifier of the target operator to the host's memory, when the target operator is accessed again later, the operator can be indexed in constant time according to the identifier of the target operator.

[0085] Step 312, the DNN acceleration library calls the interface for obtaining the Kernel, and requests the operator library to obtain the executable handle of the Kernel. When calling the interface, the identifier of the target operator is carried.

[0086] Step 313: The operator library uses the identifier of the target operator to determine the executable handle of the Kernel and returns the executable handle to the DNN acceleration library.

[0087] Step 314: The DNN acceleration library calls an interface for obtaining the workspace size (get workspace size) to obtain the workspace size from the operator library.

[0088] Step 315 , the operator library determines the amount of data when the kernel function is started (kernel.Get Launch Data Size), and determines the amount of intermediate data when the kernel function is running (kernel.Get Scratch Size).

[0089] Step 316: The operator library returns the amount of data when the kernel function is started and the amount of intermediate data when the kernel function is running to the DNN acceleration library.

[0090] Step 317, the DNN acceleration library allocates working memory (Allocate Workspace Mem) based on the amount of data when the kernel function is started and the amount of intermediate data when the kernel function is running.

[0091] Step 318: The DNN acceleration library calls the Setup Run Info interface to configure the run information for the operator library.

[0092] Step 319, the operator library initializes the kernel function's startup data (kernel.Initial Launch Data), and returns a return value indicating that initialization is complete to the DNN acceleration library.

[0093] Step 320, the DNN acceleration library calls the interface of the launch kernel function (Launch Kernel) to instruct the operator library to launch the kernel function (Launch Kernel).

[0094] Step 321: The operator library runs the kernel function (Kernel.Run).

[0095] The operator library instructs the external computing unit to run the kernel function. For example, the operator library determines the load of each external computing unit and uses the external computing unit with the smallest load to run the kernel function.

[0096] Step 322: If an error occurs during kernel function execution, the operator library returns error (ErrorHandling) information to the DNN acceleration library. If an error occurs during kernel function execution, the operator library returns the calculation result to the DNN acceleration library.

[0097] Step 323, the DNN acceleration library returns a check result (Check Result) to the application.

[0098] Scenario 2: Delayed loading of operators. Figure 4 The lazy loading operator and the process of executing the operator are shown.

[0099] Step 401: The application loads a file of a deep learning model. The file of the deep learning model is stored in the ONNX format.

[0100] Step 402: The application calls the Setup interface to instruct the DNN acceleration library to configure the deep learning model.

[0101] Step 403, the DNN acceleration library creates Context, sets Stream, and sets Allocator.

[0102] Step 404: The DNN acceleration library returns a return value to the application indicating that the configuration is complete.

[0103] Step 405: The application provides the DNN acceleration library with descriptive information related to the calculation, such as input / output desc, conv desc, and i / o mem.

[0104] Step 406: The DNN acceleration library instructs the operator library to load the required operators based on the description information provided by the application.

[0105] Among them, the required operators are all operators included in the deep learning model.

[0106] Step 407: The operator library loads operators in a hierarchical manner.

[0107] In this embodiment, the operator library loads the business logic of all operators on the host side and the device side into the host's memory, and then loads the business logic of the device side into the external computing unit running based on the external computing unit running the business logic of the device side. For the detailed loading process, please refer to the description below.

[0108] Step 408: The operator library returns a return value indicating that loading is complete to the DNN acceleration library.

[0109] Step 409: The DNN acceleration library calls the Get Kernel interface to obtain the kernel function of the target operator from the operator library.

[0110] The interface for obtaining the kernel function is called to obtain the kernel function of the target operator, where the target operator is any operator of the index. The above description information is carried when the interface for obtaining the kernel function is called.

[0111] In step 410, the operator library determines the optimal business logic of the target operator among the business logics at each level, determines the identifier of the target operator, and returns the identifier of the target operator to the DNN acceleration library.

[0112] In this embodiment, the operator library uses the description information to determine the optimal business logic of the target operator in each level of business logic, and the detailed process is described later. The operator library composes the identifier of the target operator according to the hierarchical order of the optimal business logic.

[0113] Step 411, the DNN acceleration library caches the identifier of the target operator.

[0114] Step 412, the DNN acceleration library calls the interface for obtaining the Kernel, and requests the operator library to obtain the executable handle of the Kernel. When calling the interface, the identifier of the target operator is carried.

[0115] Step 413: The operator library uses the identifier of the target operator to determine the executable handle of the Kernel and returns the executable handle to the DNN acceleration library.

[0116] Step 414: The DNN acceleration library calls the get workspace size interface to obtain the operating space size from the operator library.

[0117] Step 415, the operator library determines kernel.Get Launch Data Size and kernel.GetScratch Size.

[0118] Step 416: The operator library returns the amount of data when the kernel function is started and the amount of intermediate data when the kernel function is running to the DNN acceleration library.

[0119] Step 417, the DNN acceleration library allocates running memory based on the amount of data when the kernel function is started and the amount of intermediate data when the kernel function is running.

[0120] Step 418: The DNN acceleration library calls the Setup Run Info interface to configure the run information for the operator library.

[0121] Step 419, the operator library initializes the startup data of the kernel function and returns a return value indicating that the initialization is complete to the DNN acceleration library.

[0122] Step 420: The DNN acceleration library calls the interface for starting the kernel function, instructing the operator library to start the kernel function.

[0123] Step 421: The operator library runs the kernel function.

[0124] Step 422: When an error occurs during the kernel function execution, the operator library returns error information to the DNN acceleration library. When an error occurs during the kernel function execution, the operator library returns the calculation result to the DNN acceleration library.

[0125] Step 423: The DNN acceleration library returns the return value of the verification result to the application.

[0126] Figure 3 and Figure 4 The differences in the processes shown are: Figure 3 is to preload all operators, Figure 4 It is to determine the external computing unit of the operator to be executed before execution, and then load the business logic of the host side and device side of the operator to be executed into the corresponding memory of the external computing unit.

[0127] In addition Figure 4 In the process, when the host is connected to multiple external computing units, it is impossible to determine which external computing unit the operator will eventually run on when configuring the deep learning model. However, during execution, the external computing unit where the operator runs has been determined, and the business logic of the operator's Device side can be directly loaded into the memory of the external computing unit. In addition, the required operators can be loaded, avoiding the long time of loading all operators for the first time.

[0128] The following describes the hierarchical loading process, the information table generation process, the first identification of the target operator and the process of calling the target operator, and the process of searching for the operator.

[0129] 1. Hierarchical loading process.

[0130] When programming operators, programmers classify the business logic of operators running on the host based on different attributes to obtain n levels of business logic, which attributes include but are not limited to the number of input and output tensors, the shape of tensors, the data type of tensors, or the data arrangement format of tensors. Among the n levels, except for the top level (first level), the business logic of the remaining levels is subordinate to a business logic of the upper level. The business logic of the lowest level is bound to the executable handle one by one, that is, a business logic of the lowest level corresponds to an executable handle of an operator on the Device side, and different business logics of the lowest level correspond to executable handles of different operators on the Device side. Operators under the same business logic of the i-th level have the same one or more attributes. For example, the number of input and output tensors of operators under the same business logic of the i-th level is the same. For another example, the data type of the input tensor of the operator under the same business logic of the i-th level is a floating point type. Operators under different business logics of the i-th level have at least one difference in the above one or more attributes. For example, the data types of the input tensors of the operators under the two business logics of the i-th level are floating-point type and integer type respectively.

[0131] The operators to be loaded are the operators used by the deep learning model in the operator library. The host loads the business logic of the operators to be loaded running on the host into the host's memory. The host assigns continuous identifiers to all business logics of the first level, records the subordinate relationship between the business logic of each level and the business logic of the upper level to which it is subordinate, and in the process of establishing the subordinate relationship, assigns continuous identifiers to each business logic that is subordinate to the same business logic of the upper level. For example, the first level includes 3 business logics, and the identifiers of the 3 business logics are 0, 1, and 2. In the second level, the business logic subordinate to the identifier 0 includes 3 business logics, and the identifiers of the 3 business logics are 0, 1, and 2. In the second level, the business logic subordinate to the identifier 1 includes 4 business logics, and the identifiers of the 4 business logics are 0, 1, 2, and 3.

[0132] In addition, the host loads the business logic of the operator to be loaded on the Device side into the host's memory, and then further loads it into the memory of the external computing unit. During the loading process, the business logic of the lowest level (i.e., the nth level) is bound to the executable handle, so that when the business logic of the lowest level on the host side is subsequently found, its bound executable handle can be accessed, so that the operator can be executed.

[0133] For an example, see Figure 5 , the business logic of the host side of the operator to be loaded is compiled and encapsulated in the host binary (Host Bin) file, and the business logic of the operator to be loaded on the Device side is compiled and encapsulated in the Device Bin file. Both the Host Bin file and the Device Bin file belong to the host's file system. The business logic of the operator to be loaded on the host side is divided into three levels, from top to bottom, namely the operation level, the Tactic level and the kernel level. An operation is shown in the operation level, and two Tactics are shown in the Tactic level as belonging to the operation. In the kernel level, two kernels are subordinate to one Tactic, and the business logic subordinate to another Tactic in the kernel level is not shown. When assigning identifiers, the identifier assigned to the one operation is 0, the identifiers assigned to the two Tactics are 0 and 1, and the identifiers assigned to the two kernels are 0 and 1.

[0134] The Device Bin file includes Tactic0.0 and Tactic1.0. Tactic0.0 is loaded into the external computing unit Device0, and Tactic1.0 is loaded into the external computing unit Device1. Tactic0.0 includes kernel0 function and kernel1 function, and Tactic1.0 includes kernel0 function and kernel1 function. The kernel0 function on the Device side in Tactic0.0 is associated with kernel0 on the host side, and the kernel1 function on the Device side in Tactic0.0 is associated with kernel1 on the host side.

[0135] Optionally, when loading the operator to be loaded, all the business logic on the host side in the Host Bin file is loaded into the host's memory at one time, and all the business logic on the Device side in the Device Bin file is loaded into the host's memory at one time, and then all the business logic on the Device side is further loaded into the memory of the external computing unit.

[0136] In addition, by adopting a hierarchical loading mechanism, when different operators in the operators to be loaded have the same business logic, the business logic is only loaded once, which can save memory storage space.

[0137] 2. The process of generating information table.

[0138] In the process of loading the operators to be loaded, information tables of each level are generated, and the number of information tables is equal to the number of levels. For example, if there are n levels, the number of information tables is also n.

[0139] First, an information table of the top-level business logic is generated, which includes information of all the top-level business logics. The index of the information of each business logic is the identifier of the business logic. After the information table of the top-level business logic is generated, the information table is stored in the form of a sequential table, that is, a storage unit with a continuous physical address is used in the host's memory to sequentially store the information of each business logic of the top level.

[0140] Secondly, an information table of the business logic of the second level is generated. The information table includes multiple information sub-tables. Each information sub-table corresponds to one or more business logics. The business logic corresponding to each information sub-table is subordinate to a business logic of the first level. The business logics corresponding to different information sub-tables are subordinate to different business logics of the first level. Each information sub-table includes information of the corresponding business logics. The index of the information of each business logic is the identifier of the business logic.

[0141] Next, an information table of the business logic of the third level is generated. The information table includes multiple information sub-tables. Each information sub-table corresponds to one or more business logics. The business logic corresponding to each information sub-table is subordinate to a business logic of the second level. The business logics corresponding to different information sub-tables are subordinate to different business logics of the second level. Each information sub-table includes information of the corresponding business logics. The index of the information of each business logic is the identifier of the business logic.

[0142] In this way, information tables of business logic at each level are generated step by step. Each information sub-table included in the information table is stored in the host's memory in the form of a sequential table.

[0143] Wherein, assuming that there are n levels, for each business logic from the first level to the n-1th level, the information of the business logic includes the name of the business logic, the storage address information in the host memory and the address information of the information subtable, the information subtable includes the information of the next layer of business logic subordinate to the business logic, and the address information includes the first address of the information subtable in the host memory. For the business logic of the nth level, the information of the business logic includes the name of the business logic, the storage address information in the host memory and the information of the kernel function. After the business logic is obtained based on the storage address information, the executable handle bound to the business logic can be obtained through the business logic, for example, the executable handle is included in the business logic.

[0144] Figure 6 Shows Figure 5The information table of the business logic of the operator to be loaded is shown. Among them, the information table of the Operation level includes the information of all business logics of the Operation level, and the information of each business logic includes Operationname, Operation pointer (handle) and address information of the Tactic level information subtable. The information subtable of the Tactic level includes the information of all business logics of the Operation with a subordinate identifier of 0, and the information of each business logic includes Tacticname, Tactic handle and address information of the Kernel level information subtable. The information subtable of the Kernel level includes the information of all business logics of the Tactic with a subordinate identifier of 0, and the information of each business logic includes Kernel name, Kernelhandle and Kernel information. Operation name, Tactic name and Kernel name respectively represent the names of the business logics of each level. Operation handle, Tactic handle and Kernel handle respectively represent the storage address information of the business logic of each level. Kernel information includes relevant information such as the space and parameters required to run the kernel function.

[0145] It should be noted that, from the above description, it can be seen that the identifier of the target operator is divided into data domains of multiple lengths according to the number of levels. The number of data domains is the same as the number of levels. The length of the data domain corresponding to each level can be set according to actual needs. It can be the same or different. The value of each data domain is the identifier of the operator's non-business logic at this level. According to this strategy, each operator has a unique identifier.

[0146] In addition, a reserved field may be set in the identifier of the target operator, and the reserved field may be set at the end of the identifier of the target operator.

[0147] 3. The process of first determining the target operator's identity and calling the target operator.

[0148] Users who build deep learning models configure the specified parameters of each operator according to the operator list of the operator library. The specified parameters include the identifier of the business logic of the operator at the first level. When calling the target operator, the specified parameters and parameter information of the target operator are obtained. The parameter information includes one or more of the specific parameters used by the target operator, the data arrangement format of the input tensor, the shape of the input tensor, or the data type of the input tensor. The specific parameters are parameters unique to the target operator.

[0149] When calling the target operator, the index of the target operator's business logic at the first level is obtained from the specified parameters of the target operator. Based on the index, the information of the target operator's business logic at the first level is determined from the information table at the first level. In the information, the address information of the target operator's business logic at the first level is obtained, and the first business logic of the target operator at the first level is read in the memory using the address information. In addition, the address information of the second-level information subtable is obtained from the information. The information subtable is an information subtable of the business logic subordinate to the first business logic, and the address information is used to determine the second-level business logic subordinate to the first business logic.

[0150] Then, the parameter information of the target operator is obtained. The host uses the parameter information to determine the optimal business logic of the target operator step by step in the business logic of each subsequent level. The identifiers of the optimal business logic of the target operator determined step by step are combined in order from the top level to the bottom level to obtain the identifier of the target operator. In other words, in the identifier of the target operator, the corresponding identifiers in the first level to the nth level are arranged from the front to the back.

[0151] For example, see Figure 7 , n levels include 3 levels, and the 3 levels include the Operation level, the Tactic level, and the Kernel level. The specified parameters of the target operator are used at the Operation level to determine the first business logic of the target operator at the Operation level. The business logic of the Tactic level subordinate to the first business logic is determined. In the business logic of the Tactic level, the second business logic that conforms to one or more of the specific parameters, data formats, or data types is determined. The second business logic is the optimal business logic of the target operator at the Tactic level. Then, the business logic of the Kernel level subordinate to the second business logic is determined. In the business logic of the Kernel level, the third business logic that conforms to the tensor shape is determined. The third business logic is the optimal business logic of the target operator at the Kernel level. The identifier of the target operator includes the identifier of the first business logic, the identifier of the second business logic, and the identifier of the third business logic. The identifier of the first business logic, the identifier of the second business logic, and the identifier of the third business logic are arranged from front to back.

[0152] In addition, after determining the optimal business logic of the target operator in the lowest level, determine the executable handle bound to the optimal business logic, and then use the executable handle to start the launch process and execute the target operator.

[0153] The above process is the process of calling the target operator for the first time. In the process of calling the target operator for the first time, after the identifier of the target operator is determined, the identifier of the target operator is cached, and the identifier of the target operator can be directly used to search for the target operator later.

[0154] 4. The process of searching operators.

[0155] After determining the target operator's ID, cache the target operator's ID. When the target operator is used again, obtain the cached target operator's ID. For the process of finding an operator, see Figure 8 Steps 801 to 803 are shown.

[0156] Step 801, obtaining the identifier of the target operator.

[0157] In this embodiment, the host obtains the identifier of the cached target operator.

[0158] Step 802, in the identifier of the target operator, obtain the index of the target operator in the business logic of n levels, wherein the business logic of the n levels runs on the host, each business logic of the i+1th level in the n levels has a subordinate relationship with a business logic of the i-th level, the business logic of the nth level is bound one by one to the executable handle, and the executable handle is used to execute the business logic of the operator running on the external computing unit, and i is greater than or equal to 1 and less than or equal to n-1.

[0159] In this embodiment, the host splits the identifier of the target operator according to the length of the data domain corresponding to the n levels to obtain n identifiers, and the n identifiers are the indexes corresponding to the target operator in the business logic of the n levels. For example, the n levels include 3 levels, and the length of the data domain of the n levels is 1 bit. 1 bit of data is continuously obtained from the starting position of the identifier of the target operator, which is the index corresponding to the target operator in the business logic of the n levels.

[0160] Step 803: Based on the index, search for the business logic corresponding to the target operator at the n levels, and search for the executable handle bound to the business logic of the target operator at the nth level.

[0161] In this embodiment, the host uses the determined index to determine the index of the target operator at n levels in the business logic of n levels step by step, and obtains the executable handle bound to the business logic of the target operator at the nth level.

[0162] In an optional way, the executable handle is determined using the information table. The process is:

[0163] Using the index of the first level, in the information table of the first level, determine the information of the first business logic of the target operator at the first level, and in this information, find the storage address information of the first business logic in the host memory and the address information of the information subtable corresponding to the business logic of the target operator at the second level. The host uses the storage address information to find the first business logic in the host memory, and uses the address information of the information subtable to find the information subtable. In the information subtable, use the index of the second level to find the information of the second business logic of the target operator at the second level, and in this information, find the storage address information of the second business logic in the host memory and the address information of the information subtable corresponding to the third business logic of the target operator at the third level. And so on, until the information of the nth business logic of the target operator at the nth level is found. From the information of the nth business logic, find the storage address information of the nth business logic in the host memory, use the storage address information to find the nth business logic in the host memory, and then obtain the executable handle bound to the nth business logic.

[0164] In this way, the target operator runs the business logic from the first level to the nth level on the host, and the external computing unit uses the executable handle to find the kernel function, execute the kernel function, and realize the execution of the target operator.

[0165] For an example, see Fig. 9 ,by Figure 6 Take the information table shown as an example for explanation, the target operator is identified as 000, and the indexes at the Operation level, Tactic level, and Kernel level are 0, 0, and 0 respectively. The information of the business logic identified as 0 is obtained at the Operation level, and the address information of the information subtable at the Tactic level recorded in the information is used to obtain the information of the business logic identified as 0 in the information subtable at the Tactic level, and the address information of the information subtable at the Kernel level recorded in the information is used to obtain the information of the business logic identified as 0 in the information subtable at the Kernel level.

[0166] By using the method for searching operators in the embodiment of the present disclosure, there is no need to perform hash calculations and resolve hash conflicts, thereby achieving fast search for operators and reducing the overall time for operator access execution. Moreover, starting from the first-level information table, the address of the next-level information table is quickly obtained according to the index, and the executable handle of the kernel function on the external computing unit side can be accessed in a constant time. In this way, when the operator is called frequently in reasoning or training scenarios, the overall computing efficiency can be improved, and the performance stability can be improved.

[0167] In addition, during the hierarchical loading process of operators, the identifiers of each level of operators are synchronously allocated, which can realize the fast loading and subsequent indexing of operators.

[0168] In addition, the present invention discloses a bottom-level operator-level software stack for a heterogeneous development platform, and the operator's calling interface can be obtained from the software development kit (Sdk) calling interface header file provided by the chip, and infringement evidence can be collected based on the operator calling method described in the SDK calling interface header file.

[0169] The device provided by the embodiment of the present disclosure is described below.

[0170] Fig.10 : is a structural diagram of a device for loading operators provided in an embodiment of the present disclosure. The device is applied to a host, and a computing unit is connected to the host. The device can be implemented as part or all of the device through software, hardware, or a combination of both. The device provided in an embodiment of the present disclosure can implement the embodiment of the present disclosure. Figure 5 The process described above, the device includes: a loading module 1010 and a search module 1020, wherein:

[0171] Loading module 1010, used for:

[0172] Loading a first business logic into the memory of the host, wherein the first business logic is the business logic of the operator to be loaded of the deep learning model running on the host, the first business logic is divided into n levels of business logic according to attributes, each business logic of the i+1th level in the n levels is subordinate to a business logic of the ith level, the business logic of the nth level is bound to the executable handle one by one, the executable handle is used to execute the business logic of the operator to be loaded running on the external computing unit, i is greater than or equal to 1 and less than or equal to n-1, and can be specifically used to implement the aforementioned process of loading the business logic running on the host;

[0173] The second business logic is loaded into the memory of the host, and the second business logic is loaded from the memory of the host into the memory of the external computing unit, wherein the second business logic is the business logic of the operator to be loaded running on the external computing unit, and can be specifically used to implement the aforementioned process of loading the business logic running on the external computing unit.

[0174] In an optional manner, the loading module 1010 is further used for:

[0175] Allocating continuous identifiers to all business logics of the first level among the n levels;

[0176] A continuous identifier is allocated to the business logic of the (i+1)th level that is subordinate to any business logic of the (i)th level.

[0177] In an optional manner, the search module 1020 is used to:

[0178] Obtaining an identifier of a target operator in the operators to be loaded;

[0179] From the identifier of the target operator, obtain the index of the target operator in the business logic of the n levels;

[0180] Based on the index, the business logic corresponding to the target operator at the n levels is searched, and the executable handle bound to the business logic of the target operator at the nth level is searched.

[0181] In an optional manner, the search module 1020 is used to obtain the identifier of the cached target operator.

[0182] In an optional manner, the search module 1020 is also used to determine the optimal business logic of the target operator step by step in the business logic of the n levels based on the parameter information of the target operator before obtaining the cached identifier of the target operator, wherein the parameter information includes one or more specific parameters, data formats or data types, and tensor shapes used by the target operator, and the identifier of the optimal business logic of the target operator is composed of the identifier of the target operator in the hierarchical order of the n levels; and the identifier of the target operator is cached.

[0183] In an optional manner, the search module 1020 is used to:

[0184] Obtain n information tables corresponding to the business logics of the n levels, wherein the information table corresponding to the business logic of the i-th level includes information of each business logic of the i-th level, and for each business logic of the i-th level, the information of the business logic includes storage address information of the business logic and address information of an information subtable, and the information subtable includes information of the business logic of the i+1th level subordinate to the business logic, and the information table of the n-th level includes information of each business logic of the n-th level, and for each business logic of the n-th level, the information of the business logic includes storage address information of the business logic and an executable handle bound to the business logic;

[0185] Based on the n information tables and the index, the business logic corresponding to the target operator at the n levels is searched, and the executable handle bound to the business logic of the target operator at the nth level is searched.

[0186] Fig.11: is a structural diagram of a device for searching operators provided in an embodiment of the present disclosure. The device is applied to a host, and a computing unit is connected to the host. The device can be implemented as part or all of the device through software, hardware, or a combination of both. The device provided in an embodiment of the present disclosure can implement the embodiment of the present disclosure. Figure 8 The process described above, the device includes: an acquisition module 1110 and a determination module 1120, wherein:

[0187] The acquisition module 1110 is used to: acquire the identifier of the target operator;

[0188] In the identifier of the target operator, an index of the target operator in the business logic of n levels is obtained, wherein the business logic of the n levels is run on the host, each business logic of the i+1th level in the n levels is subordinate to a business logic of the ith level, and the business logic of the nth level is bound to an executable handle one by one, and the executable handle is used to execute the business logic of the operator running on the external computing unit, i is greater than or equal to 1 and less than or equal to n-1, and can be specifically used to implement the acquisition function of step 801 and step 802 and execute the implicit steps included in step 801 and step 802;

[0189] The determination module 1120 is used to find the business logic corresponding to the target operator at the n levels based on the index, and to find the executable handle bound to the business logic of the target operator at the nth level. Specifically, it can be used to implement the determination function of step 803 and execute the implicit steps contained in step 803.

[0190] In an optional manner, the acquisition module 1110 is used to:

[0191] Get the identifier of the cached target operator.

[0192] In an optional manner, the acquisition module 1110 is further configured to:

[0193] Before obtaining the cached identifier of the target operator, based on the parameter information of the target operator, determine the optimal business logic of the target operator step by step in the business logic of the n levels, wherein the parameter information includes one or more of the specific parameters, data formats or data types, and tensor shapes used by the target operator, and the identifier of the optimal business logic of the target operator constitutes the identifier of the target operator in the hierarchical order of the n levels; and cache the identifier of the target operator.

[0194] In an optional manner, the determining module 1120 is used to:

[0195] Obtain n information tables corresponding to the business logics of the n levels, wherein the information table corresponding to the business logic of the i-th level includes information of each business logic of the i-th level, and for each business logic of the i-th level, the information of the business logic includes storage address information of the business logic and address information of an information subtable, and the information subtable includes information of the business logic of the i+1th level subordinate to the business logic, and the information table of the n-th level includes information of each business logic of the n-th level, and for each business logic of the n-th level, the information of the business logic includes storage address information of the business logic and an executable handle bound to the business logic;

[0196] Based on the n information tables and the index, the business logic corresponding to the target operator at the n levels is searched, and the executable handle bound to the business logic of the target operator at the nth level is searched.

[0197] In an optional manner, the device further includes:

[0198] A loading module is used to load the business logic of the operator to be loaded of the deep learning model running on the host into the memory of the host, wherein the business logic of the n levels is obtained by dividing the business logic of the operator to be loaded on the host according to the attributes, and the target operator belongs to the operator to be loaded; the business logic of the operator to be loaded on the external computing unit is loaded into the memory of the host, and the business logic of the operator to be loaded on the external computing unit is loaded from the memory of the host to the external computing unit; a continuous identifier is assigned to all business logics of the first level of the n levels; a continuous identifier is assigned to the business logic of the i+1th level that is subordinate to any business logic of the i-th level.

[0199] In an optional manner, the loading module is used to load the business logic of all operators included in the deep learning model running on the host into the memory of the host when configuring the deep learning model or running the deep learning model.

[0200] Fig.10 The detailed process of loading operators and Fig.11 For the detailed process of searching operators by the device for searching operators shown, please refer to the descriptions in the previous embodiments, which will not be repeated here. Fig.10 The device for loading operators shown and Fig.11 The device for searching the operator shown is attached Figure 2 The computing device 200 in FIG.

[0201] In some embodiments, the present disclosure provides a computer program product, which includes computer instructions stored in a computer-readable storage medium. A processor of a computing device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computing device executes Figure 3 , Figure 4 and Figure 8 The process shown.

[0202] Those of ordinary skill in the art will appreciate that the various method steps and units described in the embodiments disclosed in this disclosure can be implemented with electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the steps and components of each embodiment have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Those of ordinary skill in the art may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this disclosure.

[0203] In the several embodiments provided in the present disclosure, it should be understood that the disclosed system architecture, device and method can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the module is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or modules, or it can be an electrical, mechanical or other form of connection.

[0204] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the embodiments of the present disclosure.

[0205] In addition, each module in each embodiment of the present disclosure may be integrated into one processing module, or each module may exist physically separately, or two or more modules may be integrated into one module. The above integrated modules may be implemented in the form of hardware or software modules.

[0206] If the integrated module is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present disclosure is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method in each embodiment of the present disclosure. The aforementioned storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard drives, ROM, RAM, magnetic disks or optical disks.

[0207] The above description is only an exemplary embodiment of the present disclosure, but the protection scope of the present disclosure is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present disclosure, and these modifications or replacements should be included in the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be based on the protection scope of the claims.

Claims

1. A method for loading an operator, characterized in that: The method is applied to a host, the host is connected to an external computing unit, and the method comprises: Loading a first business logic into the memory of the host, wherein the first business logic is the business logic of the operator to be loaded of the deep learning model running on the host, the first business logic is divided into n levels of business logic according to attributes, each business logic of the i+1th level in the n levels is subordinate to a business logic of the ith level, the business logic of the nth level is bound to an executable handle one by one, and the executable handle is used to execute the business logic of the operator to be loaded running on the external computing unit, i is greater than or equal to 1 and less than or equal to n-1; The second business logic is loaded into the memory of the host, and the second business logic is loaded from the memory of the host into the memory of the external computing unit, wherein the second business logic is the business logic of the operator to be loaded running on the external computing unit.

2. The method according to claim 1, characterized in that The method further comprises: Allocating continuous identifiers to all business logics of the first level among the n levels; A continuous identifier is allocated to the business logic of the (i+1)th level that is subordinate to any business logic of the (i)th level.

3. The method according to claim 1 or 2, characterized in that: The method further comprises: Obtaining an identifier of a target operator in the operators to be loaded; From the identifier of the target operator, obtain the index of the target operator in the business logic of the n levels; Based on the index, the business logic corresponding to the target operator at the n levels is searched, and the executable handle bound to the business logic of the target operator at the nth level is searched.

4. The method according to claim 3, characterized in that The obtaining the identifier of the target operator in the operator to be loaded includes: Get the identifier of the cached target operator.

5. The method according to claim 4, characterized in that Before obtaining the identifier of the cached target operator, the method further includes: Based on the parameter information of the target operator, determine the optimal business logic of the target operator in the n-level business logic step by step, wherein the parameter information includes one or more specific parameters, data formats or data types, and tensor shapes used by the target operator, and the identifier of the optimal business logic of the target operator constitutes the identifier of the target operator according to the hierarchical order of the n levels; The identifier of the target operator is cached.

6. The method according to any one of claims 3 to 5, characterized in that: The searching, based on the index, for the business logic corresponding to the target operator at the n levels, and searching for the executable handle bound to the business logic of the target operator at the nth level, includes: Obtain n information tables corresponding to the business logics of the n levels, wherein the information table corresponding to the business logic of the i-th level includes information of each business logic of the i-th level, and for each business logic of the i-th level, the information of the business logic includes storage address information of the business logic and address information of an information subtable, and the information subtable includes information of business logic of the i+1th level subordinate to the business logic, and the information table of the n-th level includes information of each business logic of the n-th level, and for each business logic of the n-th level, the information of the business logic includes storage address information of the business logic and an executable handle bound to the business logic; Based on the n information tables and the index, the business logic corresponding to the target operator at the n levels is searched, and the executable handle bound to the business logic of the target operator at the nth level is searched.

7. A device for loading operators, characterized in that: The device is applied to a host, the host is connected to an external computing unit, and the device includes: Load modules for: Loading a first business logic into the memory of the host, wherein the first business logic is the business logic of the operator to be loaded of the deep learning model running on the host, the first business logic is divided into n levels of business logic according to attributes, each business logic of the i+1th level in the n levels is subordinate to a business logic of the ith level, the business logic of the nth level is bound to an executable handle one by one, and the executable handle is used to execute the business logic of the operator to be loaded running on the external computing unit, i is greater than or equal to 1 and less than or equal to n-1; The second business logic is loaded into the memory of the host, and the second business logic is loaded from the memory of the host into the memory of the external computing unit, wherein the second business logic is the business logic of the operator to be loaded running on the external computing unit.

8. The device according to claim 7, characterized in that The loading module is further used for: Allocating continuous identifiers to all business logics of the first level among the n levels; A continuous identifier is allocated to the business logic of the (i+1)th level that is subordinate to any business logic of the (i)th level.

9. The device according to claim 7 or 8, characterized in that The device also includes a search module, which is used to: Obtaining an identifier of a target operator in the operators to be loaded; From the identifier of the target operator, obtain the index of the target operator in the business logic of the n levels; Based on the index, the business logic corresponding to the target operator at the n levels is searched, and the executable handle bound to the business logic of the target operator at the nth level is searched.

10. The device according to claim 9, characterized in that The search module is used to obtain the identifier of the cached target operator.

11. The device according to claim 10, characterized in that The search module is further used to determine the optimal business logic of the target operator step by step in the business logic of the n levels based on the parameter information of the target operator before obtaining the identifier of the cached target operator, wherein the parameter information includes one or more of the specific parameters, data formats or data types, and tensor shapes used by the target operator, and the identifier of the optimal business logic of the target operator is composed of the identifier of the target operator in the hierarchical order of the n levels; and the identifier of the target operator is cached.

12. The device according to any one of claims 9 to 11, characterized in that The search module is used to: Obtain n information tables corresponding to the business logics of the n levels, wherein the information table corresponding to the business logic of the i-th level includes information of each business logic of the i-th level, and for each business logic of the i-th level, the information of the business logic includes storage address information of the business logic and address information of an information subtable, and the information subtable includes information of business logic of the i+1th level subordinate to the business logic, and the information table of the n-th level includes information of each business logic of the n-th level, and for each business logic of the n-th level, the information of the business logic includes storage address information of the business logic and an executable handle bound to the business logic; Based on the n information tables and the index, the business logic corresponding to the target operator at the n levels is searched, and the executable handle bound to the business logic of the target operator at the nth level is searched.

13. A method for searching an operator, characterized in that: The method is applied to a host, the host is connected to an external computing unit, and the method comprises: Get the target operator's ID; In the identifier of the target operator, an index of the target operator in the business logic of n levels is obtained, wherein the business logic of the n levels is run on the host, each business logic of the i+1th level in the n levels is subordinate to a business logic of the ith level, the business logic of the nth level is bound to an executable handle one by one, and the executable handle is used to execute the business logic of the operator running on the external computing unit, and i is greater than or equal to 1 and less than or equal to n-1; Based on the index, the business logic corresponding to the target operator at the n levels is searched, and the executable handle bound to the business logic of the target operator at the nth level is searched.

14. The method according to claim 13, characterized in that The obtaining the identifier of the target operator includes: Get the identifier of the cached target operator.

15. The method according to claim 14, characterized in that Before obtaining the identifier of the cached target operator, the method further includes: Based on the parameter information of the target operator, determine the optimal business logic of the target operator in the n-level business logic step by step, wherein the parameter information includes one or more specific parameters, data formats or data types, and tensor shapes used by the target operator, and the identifier of the optimal business logic of the target operator constitutes the identifier of the target operator according to the hierarchical order of the n levels; The identifier of the target operator is cached.

16. The method according to any one of claims 13 to 15, characterized in that The searching, based on the index, for the business logic corresponding to the target operator at the n levels, and searching for the executable handle bound to the business logic of the target operator at the nth level, includes: Obtain n information tables corresponding to the business logics of the n levels, wherein the information table corresponding to the business logic of the i-th level includes information of each business logic of the i-th level, and for each business logic of the i-th level, the information of the business logic includes storage address information of the business logic and address information of an information subtable, and the information subtable includes information of business logic of the i+1th level subordinate to the business logic, and the information table of the n-th level includes information of each business logic of the n-th level, and for each business logic of the n-th level, the information of the business logic includes storage address information of the business logic and an executable handle bound to the business logic; Based on the n information tables and the index, the business logic corresponding to the target operator at the n levels is searched, and the executable handle bound to the business logic of the target operator at the nth level is searched.

17. A device for searching an operator, characterized in that: The device is applied to a host, the host is connected to an external computing unit, and the device includes: An acquisition module is used to obtain the identifier of the target operator; In the identifier of the target operator, an index of the target operator in the business logic of n levels is obtained, wherein the business logic of the n levels is run on the host, each business logic of the i+1th level in the n levels is subordinate to a business logic of the ith level, the business logic of the nth level is bound to an executable handle one by one, and the executable handle is used to execute the business logic of the operator running on the external computing unit, and i is greater than or equal to 1 and less than or equal to n-1; A determination module is used to search for the business logic corresponding to the target operator at the n levels based on the index, and to search for the executable handle bound to the business logic of the target operator at the nth level.

18. The device according to claim 17, characterized in that The acquisition module is used to acquire the identifier of the cached target operator.

19. The device according to claim 18, characterized in that The acquisition module is further used to determine the optimal business logic of the target operator step by step in the business logic of the n levels based on the parameter information of the target operator before acquiring the identifier of the cached target operator, wherein the parameter information includes one or more of the specific parameters, data formats or data types, and tensor shapes used by the target operator, and the identifier of the optimal business logic of the target operator constitutes the identifier of the target operator in the hierarchical order of the n levels; and cache the identifier of the target operator.

20. The device according to any one of claims 17 to 19, characterized in that The determining module is used to: Obtain n information tables corresponding to the business logics of the n levels, wherein the information table corresponding to the business logic of the i-th level includes information of each business logic of the i-th level, and for each business logic of the i-th level, the information of the business logic includes storage address information of the business logic and address information of an information subtable, and the information subtable includes information of business logic of the i+1th level subordinate to the business logic, and the information table of the n-th level includes information of each business logic of the n-th level, and for each business logic of the n-th level, the information of the business logic includes storage address information of the business logic and an executable handle bound to the business logic; Based on the n information tables and the index, the business logic corresponding to the target operator at the n levels is searched, and the executable handle bound to the business logic of the target operator at the nth level is searched.

21. A computing device, characterized in that It includes a processor and a memory, wherein the processor is used to execute instructions stored in the memory so that the computing device executes the method for loading an operator as described in any one of claims 1 to 6, or the method for searching an operator as described in any one of claims 13 to 16.

22. A computer-readable storage medium, characterized in that: The method comprises computer program instructions. When the computer program instructions are executed by a computing device, the computing device executes the method for loading an operator as claimed in any one of claims 1 to 6, or the method for searching an operator as claimed in any one of claims 13 to 16.

Images