Method of loading an operator, method of finding an operator, apparatus, device and medium

Abstract

The present disclosure provides a method for loading an operator, a method for searching an operator, a device, equipment and a medium, and belongs to the technical field of deep learning. The method is applied to a host computer which is externally connected with a computing unit. The method comprises the following steps: obtaining an identifier of a target operator, obtaining an index of the target operator in the service logic of n levels in the identifier of the target operator, searching the service logic corresponding to the n levels of the target operator based on the index, and searching an executable handle bound to the service logic of the target operator at the nth level. By using the scheme of the present disclosure, the operator is searched in a hierarchical manner, the operator can be quickly searched, and the efficiency of executing the operator is improved.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of deep learning, and particularly relates to a method for loading an operator, a method for searching an operator, an apparatus, a device and a medium. BACKGROUND

[0002] In order to improve the execution efficiency of the operator, the host and the computing unit are considered to be used for cooperative calculation and mutual acceleration, so that the business logic of the operator needs to run on the host and the computing unit. The computing unit includes an external computing unit such as a neural processor unit (NPU), a graphics processing unit (GPU) and a digital signal processor (DSP), and the computing unit is plugged into the host. SUMMARY

[0003] The present disclosure provides a method for loading an operator, a method for searching an operator, an apparatus, a device and a medium, which can enable the business logic of the operator to run on the host and the computing unit.

[0004] In a first aspect, the present disclosure provides a method for loading an operator, which is applied to a host connected with an external computing unit, and the method comprises: loading first business logic into a memory of the host, wherein the first business logic is business logic of an operator to be loaded of a deep learning model running on the host, the first business logic is divided into n levels of business logic according to an attribute, each business logic of an i+1 level in the n levels of business logic has a subordinate relationship with a business logic of an i level, a business logic of an n level is bound with an executable handle one by one, the executable handle is used to execute business logic of the operator to be loaded running on the external computing unit, i is greater than or equal to 1 and less than or equal to n-1, loading second business logic into the memory of the host, and loading the second business logic from the memory of the host into a memory of the external computing unit, wherein the second business logic is business logic of the operator to be loaded running on the external computing unit.

[0005] In the scheme shown in the present disclosure, the business logic running on the host is hierarchical, the business logic of the lowest level is bound with the executable handle, so that the business logic of the operator running on the host can be searched and executed according to the level when the operator is executed subsequently, and the executable handle is searched using the business logic of the lowest level, so that the business logic of the operator running on the external computing unit can be executed, thereby realizing the execution of the operator on the host and the external computing unit.

[0006] In an optional manner, the method further comprises: assigning a continuous identifier to all service logics in a first level of the n levels, and assigning a continuous identifier to service logics in an i+1 level which belong to any service logic in an i level. In this way, the identifiers are assigned in this manner, and when the operator is searched, the searching can be performed in sequence, thereby saving the time for searching the operator.

[0007] In an optional manner, the method further comprises: obtaining an identifier of a target operator in the to-be-loaded operator, obtaining an index of the target operator in the service logics in the n levels in the identifier of the target operator, searching the service logics in the n levels corresponding to the target operator based on the index, and searching an executable handle bound by the service logic in the n level of the target operator.

[0008] In the scheme shown in the disclosure, the index in the service logics in the n levels is determined using the identifier of the operator, the service logics in the n levels corresponding to the operator are searched based on the index and the relationship of the service logics in the adjacent levels, and the executable handle bound by the service logic in the n level of the operator is searched. In this way, the hash calculation and the hash conflict resolution are not required, thereby realizing the fast searching of the operator and reducing the overall time for accessing and executing the operator.

[0009] In an optional manner, the obtaining of the identifier of the target operator in the to-be-loaded operator comprises: obtaining the identifier of the target operator cached. In this way, when the identifier of the operator is already obtained, the time for searching the operator can be saved.

[0010] In an optional manner, before the identifier of the target operator cached is obtained, the method further comprises: determining the optimal service logic of the target operator in the service logics in the n levels based on parameter information of the target operator, wherein the parameter information comprises one or more of a specific parameter, a data format or a data type used by the target operator, a tensor shape, the identifier of the optimal service logic of the target operator is composed of the identifiers of the service logics in the n levels in the hierarchical order, and the identifier of the target operator is cached. In the scheme shown in the disclosure, the identifier of the operator is searched and cached using the parameter information of the operator, and in the subsequent searching of the operator, the identifier of the operator does not need to be determined again, thereby saving the time for searching the operator.

[0011] In an alternative manner, the target operator is searched for the corresponding service logic in the n levels based on the index, and the executable handle bound to the service logic of the target operator in the nth level is searched, including: obtaining n information tables corresponding to the service logic in the n levels, wherein the information table corresponding to the service logic in the ith level includes the information of each service logic in the ith level, and for each service logic in the ith level, the information of the service logic includes the storage address information of the service logic and the address information of an information sub-table, the information sub-table including the information of the service logic in the ith+1 level subordinate to the service logic, and the information table in the nth level includes the information of each service logic in the nth level, and for each service logic in the nth level, the information of the service logic includes the storage address information of the service logic and the executable handle bound to the service logic, and the target operator is searched for the corresponding service logic in the n levels based on the n information tables and the index, and the executable handle bound to the service logic of the target operator in the nth level is searched.

[0012] In the scheme shown in the present disclosure, the service logic in the n levels corresponds to an information table, and since the information of the service logic is recorded in the information table, the service logic corresponding to the operator in the n levels can be quickly found using the information table and the index, and the executable handle bound to the service logic of the operator in the nth level is found, thereby saving the time for searching the operator.

[0013] In a second aspect, the present disclosure provides a method for searching an operator, the method being applied to a host connected with an external computing unit, and the method including: obtaining the identification of a target operator; obtaining the index of the target operator in the service logic in the n levels in the identification of the target operator, wherein the service logic in the n levels is run on the host, each service logic in the ith+1 level in the n levels has a subordinate relationship with one service logic in the ith level, the service logic in the nth level is bound to an executable handle one by one, the executable handle is used to execute the service logic run on the external computing unit by the operator, i is greater than or equal to 1 and less than or equal to n-1; and the target operator is searched for the corresponding service logic in the n levels based on the index, and the executable handle bound to the service logic of the target operator in the nth level is searched.

[0014] In the scheme shown in the present disclosure, the service logic running on the host is hierarchical, and the service logic at the lowest level binds the executable handle, so that when the operator is executed subsequently, the service logic running on the host of the operator can be found and executed according to the hierarchy, and the executable handle is found using the service logic at the lowest level, so that the service logic running on the external computing unit of the operator can be executed, thereby realizing the execution of the operator on the host and the external computing unit. In addition, using the identifier of the operator, the index in the n-level service logic is determined, based on the index and the relationship of the service logic of the adjacent level, the service logic corresponding to the operator at the n level is found, and the executable handle bound by the service logic of the operator at the n level is found. In this way, without performing hash calculation and solving hash conflict, the operator can be quickly found, thereby reducing the overall time of operator access execution.

[0015] In an optional manner, the identifier of the target operator is obtained, including: obtaining the identifier of the cached target operator. In this way, when the identifier of the operator has already been obtained, it is not necessary to determine the identifier of the operator, and the time for finding the operator can be saved.

[0016] In an optional manner, before obtaining the identifier of the cached target operator, it further includes: based on the parameter information of the target operator, determining the optimal service logic of the target operator in the n-level service logic step by step, wherein the parameter information includes one or more of the specific parameters, data format or data type used by the target operator, and the tensor shape, the identifier of the optimal service logic of the target operator is composed of the identifier of the target operator according to the hierarchical order of the n-level hierarchy, and the identifier of the target operator is cached.

[0017] In the scheme shown in the present disclosure, the parameter information of the operator is used to find and cache the identifier of the operator, so that when the operator is found subsequently, it is not necessary to determine the identifier of the operator again, and the time for finding the operator can be saved.

[0018] In an optional manner, the searching, based on the index, the service logic corresponding to the target operator in the n levels and the executable handle bound by the service logic of the target operator in the nth level comprises: acquiring n information tables corresponding to the service logic of the n levels, wherein the information table corresponding to the service logic of the ith level comprises information of each service logic of the ith level, and for each service logic of the ith level, the information of the service logic comprises storage address information of the service logic and address information of an information sub-table, the information sub-table comprises information of the service logic of the ith+1 level belonging to the service logic, and the information table of the nth level comprises information of each service logic of the nth level, and for each service logic of the nth level, the information of the service logic comprises storage address information of the service logic and an executable handle bound by the service logic; and searching, based on the n information tables and the index, the service logic corresponding to the target operator in the n levels and the executable handle bound by the service logic of the target operator in the nth level.

[0019] In the scheme shown in the present disclosure, the service logic of the n levels corresponds to an information table, and since the information of the service logic is recorded in the information table, the service logic corresponding to the operator in the n levels and the executable handle bound by the service logic of the operator in the nth level can be quickly found using the information table and the index, thereby saving the time for searching the operator.

[0020] In an optional manner, the method further comprises: loading the service logic of the to-be-loaded operator of the deep learning model running on the host into the memory of the host, wherein the service logic of the n levels is obtained by dividing the service logic of the to-be-loaded operator running on the host according to attributes, and the target operator belongs to the to-be-loaded operator; loading the service logic of the to-be-loaded operator running on the external computing unit into the memory of the host, and loading the service logic of the to-be-loaded operator running on the external computing unit from the memory of the host to the external computing unit; assigning a continuous identifier to all service logics of the first level in the n levels; and assigning a continuous identifier to the service logics of the ith+1 level belonging to any service logic of the ith level.

[0021] In the scheme shown in the present disclosure, the service logic of the operator used by the deep learning model is loaded to the running position in advance, and the identifier is assigned in this manner, so that the operator can be sequentially searched when the operator is searched, thereby saving the time for searching the operator.

[0022] In an optional manner, the loading of the service logic of the to-be-loaded operator of the deep learning model running on the host into the memory of the host comprises: loading the service logic of all operators included in the deep learning model running on the host into the memory of the host when the deep learning model is configured or when the deep learning model is running.

[0023] In a third aspect, the present disclosure provides a device for loading an operator, the device comprising at least one module for implementing the method for loading an operator according to the first aspect or any of the optional manners of the first aspect.

[0024] In some embodiments, the modules in the device for loading an operator are implemented by software, and the modules in the device for loading an operator are program modules. In other embodiments, the modules in the device for loading an operator are implemented by hardware or firmware.

[0025] In a fourth aspect, the present disclosure provides a device for searching an operator, the device comprising at least one module for implementing the method for searching an operator according to the second aspect or any of the optional manners of the second aspect.

[0026] In some embodiments, the modules in the device for searching an operator are implemented by software, and the modules in the device for searching an operator are program modules. In other embodiments, the modules in the device for searching an operator are implemented by hardware or firmware.

[0027] In a fifth aspect, the present disclosure provides a computing device comprising a memory and a processor, the processor of the computing device being configured to execute instructions stored in the memory of the computing device, so that the computing device performs the method for loading an operator according to the first aspect or any of the optional manners of the first aspect, or so that the computing device performs the method for searching an operator according to the second aspect or any of the optional manners of the second aspect.

[0028] In a sixth aspect, the present disclosure provides a computer-readable storage medium comprising computer program instructions, when the computer program instructions are executed by a computing device, so that the computing device performs the method for loading an operator according to the first aspect or any of the optional manners of the first aspect, or so that the computing device performs the method for searching an operator according to the second aspect or any of the optional manners of the second aspect.

[0029] In a seventh aspect, the present disclosure provides a computer program product comprising instructions, when the instructions are run by a computing device, so that the computing device performs the method for loading an operator according to the first aspect or any of the optional manners of the first aspect, or so that the computing device performs the method for searching an operator according to the second aspect or any of the optional manners of the second aspect. BRIEF DESCRIPTION OF DRAWINGS

[0030] Figure 1 is a schematic diagram of a conventional method for searching an operator;

[0031] Figure 2is a structural schematic diagram of a computing device provided by one of the exemplary embodiments of the present disclosure;

[0032] Figure 3 is a schematic diagram of a pre-staged load operator provided by one of the exemplary embodiments of the present disclosure;

[0033] Figure 4 is a schematic diagram of a delayed staged load operator provided by one of the exemplary embodiments of the present disclosure;

[0034] Figure 5 is a schematic diagram of a staged load operator provided by one of the exemplary embodiments of the present disclosure;

[0035] Figure 6 is a schematic diagram of an information table provided by one of the exemplary embodiments of the present disclosure;

[0036] Figure 7 is a schematic diagram of a first call target operator provided by one of the exemplary embodiments of the present disclosure;

[0037] Figure 8 is a flowchart of a lookup operator provided by one of the exemplary embodiments of the present disclosure;

[0038] Figure 9 is a flowchart of a lookup operator provided by one of the exemplary embodiments of the present disclosure;

[0039] Figure 10 is a structural schematic diagram of an apparatus of a load operator provided by one of the exemplary embodiments of the present disclosure;

[0040] Figure 11 is a structural schematic diagram of an apparatus of a lookup operator provided by one of the exemplary embodiments of the present disclosure. DETAILED DESCRIPTION

[0041] For the purpose of the present disclosure, technical solutions and advantages, the following will be further described in detail in conjunction with the drawings.

[0042] Some terms and concepts related to the embodiments of the present disclosure are explained below.

[0043] 1、Operator, is a function used to perform various mathematical operations and operations in a deep learning model, in the case of collaborative computing between the host and the computing unit and acceleration of each other (that is, in the case of heterogeneous programming), usually includes the business logic of the host side and the business logic of the Device side, the business logic of the host side runs on the host, and the business logic of the Device side runs on the external computing unit such as NPU, GPU or DSP. The external computing unit is connected with the host through a bus. The business logic of the host side is the binary code of the operator running on the host, and the business logic of the Device side is the binary code of the operator running on the external computing unit.

[0044] 2、The executable handle of the operator is the handle used by the Device side to execute the kernel function of the operator, through which the kernel function can be found for execution.

[0045] The background of the present disclosure and the conventional technology are described below.

[0046] As the bottom layer of the entire application software stack, the operator is used by various acceleration cards, inference engines and training frameworks in the upper layer, and the operator needs to be quickly found and run, especially in some inference scenarios. In the conventional technology, referring to Figure 1 When finding the operator, the global identifier of the operator is used for hash calculation, and based on the hash value after the hash calculation, the operator is found in the hash table. Since the operator needs to be found through the hash table, there is an overhead of hash calculation and solving hash conflicts, and when the number of operators is large, the time-consuming overhead of solving hash conflicts increases rapidly. Moreover, the hash search is not a stable constant time, which brings uncertainty in time. Based on this, in the embodiment of the present disclosure, the identifier of the operator is used to split the index of the operator in the business logic of n levels, and the executable handle of the operator is found using the index in each level, so as to execute the operator. Since no hash calculation is needed, and no hash conflict needs to be solved, the search time of the operator can be saved.

[0047] The execution subject of the method for loading the operator and the execution subject of the method for finding the operator are described below.

[0048] The execution subject of the method for loading the operator is a device for loading the operator, which is the host mentioned later, and the host is a computing device such as a terminal or a server. Alternatively, the device is a software device, such as a set of software programs running on the host.

[0049] The execution subject of the method for finding the operator is a device for finding the operator, which is the host mentioned later, and the host is a computing device such as a terminal or a server. Alternatively, the device is a software device, such as a set of software programs running on the host.

[0050] As Figure 2 shown, a computing device 200 is provided, which is optionally implemented by a general bus architecture. The computing device 200 includes at least one processor 201, a communication bus 202, a memory 203, and at least one network interface 204.

[0051] The processor 201 is, for example, a general central processing unit (CPU), a network processer (NP), a graphics processing unit (GPU), a neural-network processing unit (NPU), a data processing unit (DPU), a microprocessor, or one or more integrated circuits for implementing the present disclosure. For example, the processor 201 includes an application-specific integrated circuit (ASIC), a programmable logic device (PLD) or a combination thereof. The PLD is, for example, a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL) or any combination thereof.

[0052] The communication bus 202 is used to transmit information between the above-mentioned components. The communication bus 202 can be divided into an address bus, a data bus, a control bus, etc. For the sake of representation, Figure 2 only one thick line is used in the middle, but it does not mean that there is only one bus or only one type of bus.

[0053] The memory 203 is, for example, a read-only memory (ROM) or other type of static storage device that can store static information and instructions that are not to be changed by the computer; alternatively, the memory 203 is, for example, a random access memory (RAM), or other type of dynamic storage device that can store information and instructions for execution by the processor 201; alternatively, the memory 203 is, for example, an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM), or other optical disk storage, including a compact disc, a laser disc, an optical disc, a digital versatile disc, a Blu-ray disc, and the like, a magnetic disk storage medium, or other magnetic storage device, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer, but is not limited to the foregoing. The memory 203 is, for example, independent and connected to the processor 201 through the communication bus 202. The memory 203 can also be integrated with the processor 201.

[0054] Optionally, the memory 203 is configured to store the information table mentioned below. When the computing device 200 needs to use the information table, the processor 201 accesses the memory 203 to obtain the information table stored in the memory 203.

[0055] The network interface 204 is configured to communicate with other devices or communication networks using any transceiver-like device. The network interface 204 includes a wired network interface and can also include a wireless network interface. The wired network interface can be, for example, an Ethernet interface. The Ethernet interface can be an optical interface, an electrical interface, or a combination thereof. The wireless network interface can be, for example, a wireless local area networks (WLAN) interface, a cellular network interface, or a combination thereof.

[0056] In a specific implementation, as an example, the processor 201 can include one or more CPUs.

[0057] In a specific implementation, as an example, the computing device 200 can include multiple processors. Each of the processors can be a single-CPU or a multi-CPU. The processor herein can refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions).

[0058] In a specific implementation, as an example, the computing device 200 can further include an output device and an input device. The output device is in communication with the processor 201 and can display information in a variety of ways. For example, the output device can be a liquid crystal display (LCD), a light emitting diode (LED) display device, a cathode ray tube (CRT) display device, or a projector, etc. The input device is in communication with the processor 201 and receives input of a user in a variety of ways. For example, the input device can be a mouse, a keyboard, a touch screen device, or a sensor device, etc.

[0059] In some embodiments, the memory 203 is configured to store program code 2031 for implementing a lookup operator or a loading operator in the present disclosure, and the processor 201 executes the program code 2031 stored in the memory 203. That is, the computing device 200 can implement the method of the lookup operator or the loading operator provided by the method embodiments through the processor 201 and the program code 2031 in the memory 203.

[0060] The application scenarios of the method of the loading operator and the method of the lookup operator are described below.

[0061] In the embodiments of the present disclosure, the application scenarios include scenario one and scenario two. Scenario one is a pre-loading operator scenario, that is, loading the operator when configuring the deep learning model. Scenario two is a delayed loading scenario, that is, first configuring the deep learning model, and then loading the operator when executing the operator. In scenario one and scenario two, the interaction process of the application program, the deep neural network (DNN) acceleration library, and the operator library is involved. The application program, the DNN acceleration library, and the operator library are all software programs and are deployed on a host. The application program is a program for performing computation using a deep learning model. The DNN acceleration library can provide highly optimized implementations of common operations such as convolution, pooling, normalization, and activation, and can accelerate computation. The host is connected to one or more external computing units. The types of the plurality of external computing units are completely the same, for example, the plurality of external computing units are all GPUs, or the plurality of external computing units include external computing units of different types, for example, the plurality of external computing units include GPUs and NPUs, etc.

[0062] Scenario one, the pre-loading operator scenario. Referring to the pre-loading operator shown in Figure 3 and the flow of executing the operator.

[0063] In an offline inference scenario or a training scenario, a deep learning model is loaded, and the deep learning model is an arbitrary inference model, such as an image classification model or a speech recognition model, etc., and the loading of the operators included in the deep learning model is started.

[0064] Step 301, the application loads the file of the deep learning model. The file of the deep learning model is stored in an open neural network exchange (ONNX) format.

[0065] Wherein, the application is a program for performing calculation using the deep learning model in an offline inference scenario or a training scenario.

[0066] Step 302, the application calls a configuration (Setup) interface to instruct the DNN acceleration library to configure the deep learning model.

[0067] Step 303, the DNN acceleration library creates a context, sets a stream, and sets an allocator.

[0068] Wherein, the Stream is a task queue of the external computing unit, and the tasks in the Stream are executed in the order in the Stream, and the tasks can be kernel function execution operations of the operators, and the set Allocator is used to specify a memory allocation strategy.

[0069] Step 304, the DNN acceleration library calls a loading interface to instruct the operator library to preload the operators used in the operator library.

[0070] Wherein, the operator library corresponds to an operator list, and the operator list records information of all operators included in the operator library. The operators used in the operator library include all operators used in the deep learning model, but can be more than all operators used in the deep learning model.

[0071] Step 305, the operator library performs hierarchical loading of the operators.

[0072] In this embodiment, the operator library loads full-amount host-side and Device-side business logic into the memory of the host. The host-side business logic mainly includes logic of operator execution preparation and logic of an executable handle lookup strategy of the operator. For example, the operator execution preparation includes derivation of a shape of a tensor and data partitioning. Then the operator library loads the Device-side business logic into the memory of each external computing unit. The detailed loading process is described below. Wherein, the host-side business logic refers to business logic of the operator running on the host, and the Device-side business logic refers to business logic of the operator running on the external computing unit.

[0073] It should be noted that in the scenario of preloading the operator, the business logic of the Device side is loaded into each external computing unit, because the subsequent business logic of the Device side is unknown on which external computing unit to run, so all the business logic of the Device side is on each external computing unit, so that the subsequent business logic of the Device side does not need to be reloaded when running.

[0074] In step 306, the operator library returns a return value indicating that the loading is completed to the DNN acceleration library.

[0075] In step 307, the DNN acceleration library returns a return value indicating that the configuration is completed to the application.

[0076] In step 308, the application provides the DNN acceleration library with description information related to calculation, such as input / output desc, conv desc, and i / o mem.

[0077] The input / output desc includes the format of the input / output data, and the i / o mem refers to the memory occupied by the input / output data.

[0078] In step 309, the DNN acceleration library calls an interface of a Get Kernel function to obtain the kernel function of the target operator from the operator library.

[0079] The interface of the Get Kernel function is used to obtain the kernel function of the target operator, and the target operator is any operator indexed. The above description information is carried when the interface of the Get Kernel function is called.

[0080] In step 310, the operator library determines the optimal business logic of the target operator in the business logic of each level, determines the identifier of the target operator, and returns the identifier of the target operator to the DNN acceleration library.

[0081] In this embodiment, the operator library uses the description information to determine the optimal business logic of the target operator in the business logic of each level, and the detailed process is described below. The operator library composes the identifier of the optimal business logic into the identifier of the target operator according to the hierarchical order.

[0082] It should be noted that the operator library executes the business logic of each level after determining the business logic of each level running on the host. For example, the business logic on the host is divided into three levels, which are the Operation level, the Tactic level and the kernel level. The operator library first determines the business logic of the Operation level, and then executes the business logic of the Operation level, determines the business logic of the Tactic level, and then executes the business logic of the Tactic level, and determines the business logic of the kernel level, and then executes the business logic of the kernel level.

[0083] Step 311, the DNN acceleration library caches the identifier of the target operator.

[0084] In step 311, the DNN acceleration library caches the identifier of the target operator to the memory of the host, and then when the target operator is accessed again, the operator can be indexed according to the identifier of the target operator.

[0085] Step 312, the DNN acceleration library calls an interface for obtaining a Kernel, and requests the operator library to obtain an executable handle of the Kernel, and carries the identifier of the target operator when calling the interface.

[0086] Step 313, the operator library uses the identifier of the target operator to determine the executable handle of the Kernel, and returns the executable handle to the DNN acceleration library.

[0087] Step 314, the DNN acceleration library calls an interface for obtaining a running space size (get workspace size), and obtains the running space size from the operator library.

[0088] Step 315, the operator library determines the data size when the kernel function is started (kernel.Get Launch Data Size), and determines the intermediate data size when the kernel function is running (kernel.Get Scratch Size).

[0089] Step 316, the operator library returns the data size when the kernel function is started and the intermediate data size when the kernel function is running to the DNN acceleration library.

[0090] Step 317, the DNN acceleration library allocates running memory (Allocate Workspace Mem) based on the data size when the kernel function is started and the intermediate data size when the kernel function is running.

[0091] Step 318, the DNN acceleration library calls an interface for configuring running information (Setup Run Info), and configures the running information to the operator library.

[0092] Step 319, the operator library initializes the kernel function launch data (kernel.Initial Launch Data), and returns a return value indicating that the initialization is completed to the DNN acceleration library.

[0093] Step 320, the DNN acceleration library calls the interface of launching the kernel function (Launch Kernel), and instructs the operator library to launch the kernel function (Launch Kernel).

[0094] Step 321, the operator library runs the kernel function (Kernel.Run).

[0095] Among them, the operator library instructs the external computing unit to run the kernel function. For example, the operator library determines the load of each external computing unit, uses the external computing unit with the smallest load, and runs the kernel function.

[0096] Step 322, in the case of kernel function running error, the operator library returns error (ErrorHandling) information to the DNN acceleration library, and in the case of correct running of the kernel function, the operator library returns the calculation result to the DNN acceleration library.

[0097] Step 323, the DNN acceleration library returns the check result (Check Result) to the application program.

[0098] Scenario two, the scenario of delayed loading of operators. Referring to the delayed loading of operators and the flow of executing the operators shown in Figure 4

[0099] Step 401, the application program loads the file of the deep learning model. The file of the deep learning model is stored in the ONNX format.

[0100] Step 402, the application program calls the Setup interface, and instructs the DNN acceleration library to configure the deep learning model.

[0101] Step 403, the DNN acceleration library creates Context, sets Stream and sets Allocator.

[0102] Step 404, the DNN acceleration library returns a return value indicating that the configuration is completed to the application program.

[0103] Step 405, the application program provides the input / output desc, the convolution desc, and the i / o mem and other calculation related description information to the DNN acceleration library.

[0104] Step 406, the DNN acceleration library instructs the operator library to load the required operators based on the description information provided by the application program.

[0105] ​The required operator is all operators included in the deep learning model.

[0106] At step 407, the operator library performs hierarchical loading of the operators.

[0107] In this embodiment, the operator library loads the service logic of all operators on the host side and the Device side into the memory of the host, and then loads the service logic of the Device side to the running external computing unit based on the external computing unit running the service logic of the Device side. The detailed loading process is described below.

[0108] At step 408, the operator library returns a return value indicating completion of loading to the DNN acceleration library.

[0109] At step 409, the DNN acceleration library calls an interface of a get kernel function to obtain the kernel function of the target operator from the operator library.

[0110] The interface of the get kernel function is used to obtain the kernel function of the target operator, and the target operator is any operator with an index. The above description information is carried when the interface of the get kernel function is called.

[0111] At step 410, the operator library determines the optimal service logic of the target operator in the service logic of each level, determines the identifier of the target operator, and returns the identifier of the target operator to the DNN acceleration library.

[0112] In this embodiment, the operator library uses the description information to determine the optimal service logic of the target operator in the service logic of each level, and the detailed process is described below. The operator library composes the identifier of the optimal service logic into the identifier of the target operator according to the hierarchical order.

[0113] At step 411, the DNN acceleration library caches the identifier of the target operator.

[0114] At step 412, the DNN acceleration library calls an interface of the get kernel to request the executable handle of the kernel from the operator library, and carries the identifier of the target operator when calling the interface.

[0115] At step 413, the operator library uses the identifier of the target operator to determine the executable handle of the kernel, and returns the executable handle to the DNN acceleration library.

[0116] At step 414, the DNN acceleration library calls an interface of the get workspace size to obtain the running space size from the operator library.

[0117] In step 415 , the operator library determines kernel.Get Launch Data Size and kernel.GetScratch Size.

[0118] In step 416, the operator library returns the amount of data when the kernel function is started and the amount of intermediate data when the kernel function is running to the DNN acceleration library.

[0119] In step 417 , the DNN acceleration library allocates running memory based on the amount of data when the kernel function is started and the amount of intermediate data when the kernel function is running.

[0120] In step 418, the DNN acceleration library calls the Setup Run Info interface to configure the run information for the operator library.

[0121] In step 419 , the operator library initializes the startup data of the kernel function and returns a return value indicating that the initialization is complete to the DNN acceleration library.

[0122] In step 420 , the DNN acceleration library calls the interface for starting the kernel function, instructing the operator library to start the kernel function.

[0123] Step 421: The operator library runs the kernel function.

[0124] Step 422: If the kernel function fails to execute correctly, the operator library returns the error information to the DNN acceleration library. If the kernel function completes the execution correctly, the operator library returns the calculation result to the DNN acceleration library.

[0125] In step 423, the DNN acceleration library returns the verification result to the application.

[0126] Figure 3 and Figure 4 The differences in the processes shown are: Figure 3 It is to preload all operators. Figure 4 It is to determine the external computing unit of the operator to be executed before execution, and then load the business logic of the host side and device side of the required execution operator into the corresponding memory of the external computing unit.

[0127] In addition Figure 4 In this process, when a host is connected to multiple external computing units, it's unclear which external computing unit an operator will ultimately run on during deep learning model configuration. However, during execution, the external computing unit where the operator will run is already determined, allowing the operator's device-side business logic to be directly loaded into the memory of that external computing unit. Furthermore, operators can be loaded only as needed, avoiding the time-consuming initial loading of all operators.

[0128] The hierarchical loading process, the information table generation process, the process of first determining the identification of the target operator and calling the target operator, and the process of searching for the operator are described below, respectively.

[0129] 1. Hierarchical loading process.

[0130] When writing the operator, the programmer classifies the business logic of the operator running on the host based on different attributes, including but not limited to one or more of the number of input and output tensors, the shape of the tensor, the data type of the tensor, or the data arrangement format of the tensor, to obtain n levels of business logic. In the n levels, except for the uppermost level (the first level), the business logic of each level has a dependent relationship with the business logic of the previous level. The business logic of the lowermost level is bound to an executable handle, that is, the business logic of the lowermost level corresponds to the executable handle of an operator on the Device side, and different business logics of the lowermost level correspond to the executable handles of different operators on the Device side. The operators under the same business logic of the i-th level have the same one or more attributes. For example, the number of input and output tensors of the operators under the same business logic of the i-th level is the same. For another example, the data type of the input tensor of the operators under the same business logic of the i-th level is a floating-point type. The operators under different business logics of the i-th level have at least one difference in the aforementioned one or more attributes. For example, the data type of the input tensor of the operators under two business logics of the i-th level is a floating-point type and an integer type, respectively.

[0131] The to-be-loaded operator is an operator used by a deep learning model in an operator library. The host loads the business logic of the to-be-loaded operator running on the host into the memory of the host, allocates continuous identifications to all the business logics of the first level, records the dependent relationship between the business logic of each level and the business logic of the previous level to which it belongs, and allocates continuous identifications to each business logic belonging to the same business logic of the previous level in the process of establishing the dependent relationship. For example, the first level includes three business logics, and the identifications of the three business logics are 0, 1, and 2. In the second level, the three business logics belonging to the business logic with the identification of 0 include three business logics, and the identifications of the three business logics are 0, 1, and 2. In the second level, the four business logics belonging to the business logic with the identification of 1 include four business logics, and the identifications of the four business logics are 0, 1, 2, and 3.

[0132] In addition, the host loads the business logic of the to-be-loaded operator on the Device side into the memory of the host, and then further loads it into the memory of the external computing unit. In the loading process, the business logic of the lowermost level (i.e., the n-th level) is bound to the executable handle, so that the business logic of the lowermost level on the host side is found subsequently, the bound executable handle is accessed, and thus the operator can be executed.

[0133] For example, referring to Figure 5 The business logic of the host side of the to-be-loaded operator is compiled and packaged in a Host Bin file, and the business logic of the Device side of the to-be-loaded operator is compiled and packaged in a Device Bin file. Both the Host Bin file and the Device Bin file belong to the file system of the host. The business logic of the host side of the to-be-loaded operator is divided into three levels, from top to bottom, which are an operation level, a tactic level and a kernel level. An operation is shown in the operation level, two tactics are shown in the tactic level and belong to the operation, and two kernels are shown in the kernel level and belong to one of the tactics. The business logic belonging to the other tactic is not shown in the kernel level. When the identifiers are assigned, the identifier assigned to the operation is 0, the identifiers assigned to the two tactics are 0 and 1, and the identifiers assigned to the two kernels are 0 and 1.

[0134] The Device Bin file includes Tactic0.0 and Tactic1.0, and Tactic0.0 is loaded onto the external computing unit Device0, and Tactic1.0 is loaded onto the external computing unit Device1. Tactic0.0 includes kernel0 and kernel1, and Tactic1.0 includes kernel0 and kernel1. The kernel0 on the Device side in Tactic0.0 is associated with the kernel0 on the host side, and the kernel1 on the Device side in Tactic0.0 is associated with the kernel1 on the host side.

[0135] Optionally, when the to-be-loaded operator is loaded, all the business logic on the host side in the Host Bin file is loaded into the memory of the host at one time, and all the business logic on the Device side in the Device Bin file is loaded into the memory of the host at one time. Then, all the business logic on the Device side is further loaded into the memory of the external computing unit.

[0136] In addition, when different operators in the to-be-loaded operator have the same business logic, the hierarchical loading mechanism is adopted, and the business logic is loaded only once, which can save the storage space of the memory.

[0137] 2. The generation process of the information table.

[0138] In the process of loading the to-be-loaded operator, the information table of each level is generated, and the number of information tables is equal to the number of levels. For example, when there are n levels, the number of information tables is also n.

[0139] First, an information table of the top-level service logic is generated, which includes information of all service logics of the top level. The index of the information of each service logic is the identification of the service logic. After the information table of the top-level service logic is generated, the information table is stored in the form of a sequential table, that is, the information of each service logic of the top level is stored in a storage unit with a physical address in the memory of the host in sequence.

[0140] Second, an information table of the second-level service logic is generated, which includes a plurality of information sub-tables, each information sub-table corresponding to one or more service logics, each information sub-table corresponding to the service logics subordinate to a service logic of the first level, different information sub-tables corresponding to the service logics subordinate to different service logics of the first level, and each information sub-table including the information of each corresponding service logic. The index of the information of each service logic is the identification of the service logic.

[0141] Third, an information table of the third-level service logic is generated, which includes a plurality of information sub-tables, each information sub-table corresponding to one or more service logics, each information sub-table corresponding to the service logics subordinate to a service logic of the second level, different information sub-tables corresponding to the service logics subordinate to different service logics of the second level, and each information sub-table including the information of each corresponding service logic. The index of the information of each service logic is the identification of the service logic.

[0142] By analogy, the information table of each level of service logic is generated in sequence. Each information sub-table included in the information table is stored in the form of a sequential table in the memory of the host.

[0143] Among them, assuming that there are n levels, for each service logic of the first level to the n-1 level, the information of the service logic includes the name of the service logic, the storage address information in the memory of the host, and the address information of the information sub-table, the information sub-table including the information of the next level service logic subordinate to the service logic, and the address information including the first address of the information sub-table in the memory of the host. For the service logic of the n level, the information of the service logic includes the name of the service logic, the storage address information in the memory of the host, and the information of the kernel function, and after obtaining the service logic based on the storage address information, the executable handle bound to the service logic can be obtained through the service logic, for example, the executable handle is included in the service logic.

[0144] Figure 6 It is shown that Figure 5The information table of the business logic of the to-be-loaded operator is shown. The information table of the Operation level includes information of all business logics of the Operation level, and each piece of information of the business logic includes Operationname, an Operation pointer (handle), and address information of a Tactic level information sub-table. The Tactic level information sub-table includes information of all business logics of the Operation with a subordinate identifier of 0, and each piece of information of the business logic includes Tacticname, a Tactic handle, and address information of a Kernel level information sub-table. The Kernel level information sub-table includes information of all business logics of the Tactic with a subordinate identifier of 0, and each piece of information of the business logic includes Kernelname, a Kernel handle, and Kernel information. The Operation name, the Tactic name, and the Kernel name respectively represent the names of the business logics of the respective levels. The Operation handle, the Tactic handle, and the Kernel handle respectively represent the storage address information of the business logics of the respective levels. The Kernel information includes relevant information such as a space and parameters required for running a kernel function.

[0145] It should be noted that, as can be known from the above description, the identifier of the target operator is divided into data fields of multiple lengths according to the number of levels, the number of data fields is the same as the number of levels, the length of the data field corresponding to each level can be set according to actual needs, and can be the same or different. The value of each data field is the identifier of the non-business logic of the operator at the level. According to this strategy, each operator has a unique identifier.

[0146] In addition, a reserved field can also be set in the identifier of the target operator, and the reserved field can be set at the end of the identifier of the target operator.

[0147] 3. First determine the identifier of the target operator and call the target operator.

[0148] The user constructing the deep learning model configures the specified parameters of each operator according to the operator list of the operator library. The specified parameters include the identifier of the business logic of the operator at the first level. When the target operator is called, the specified parameters and parameter information of the target operator are obtained, and the parameter information includes one or more of the specific parameters used by the target operator, the data arrangement format of the input tensor, the shape of the input tensor, or the data type of the input tensor. The specific parameter is a parameter unique to the target operator.

[0149] In the calling of the target operator, the index of the business logic of the target operator in the first level is obtained from the specified parameter of the target operator, based on which the information of the business logic of the target operator in the first level is determined from the information table in the first level, in which the address information of the business logic of the target operator in the first level is obtained, and the first business logic of the target operator in the first level is read in the memory using the address information. And in the information, the address information of the information sub-table in the second level is obtained, which is the information sub-table of the business logic subordinate to the first business logic, and using the address information, the business logic of the second level subordinate to the first business logic is determined.

[0150] Then the parameter information of the target operator is obtained. The host uses the parameter information to determine the optimal business logic of the target operator in each level of the subsequent business logic. The identification of the optimal business logic of the target operator determined step by step is combined in order from the top level to the bottom level to obtain the target operator identification. That is, in the identification of the target operator, the corresponding identification from the first level to the nth level is arranged from front to back.

[0151] For example, referring to Figure 7 , the n levels include 3 levels, the 3 levels include Operation level, Tactic level and Kernel level, and the first business logic of the target operator in the Operation level is determined using the specified parameter of the target operator. The business logic of the Tactic level subordinate to the first business logic is determined, and in the business logic of the Tactic level, the second business logic conforming to one or more of the specific parameters, data formats or data types is determined, which is the optimal business logic of the target operator in the Tactic level. Then the business logic of the Kernel level subordinate to the second business logic is determined, and in the business logic of the Kernel level, the third business logic conforming to the tensor shape is determined, which is the optimal business logic of the target operator in the Kernel level. The identification of the target operator includes the identification of the first business logic, the identification of the second business logic and the identification of the third business logic, which are arranged from front to back.

[0152] In addition, after determining the optimal business logic of the target operator in the bottom level, the executable handle bound to the optimal business logic is determined. Then the launch process is started using the executable handle, and the target operator is executed.

[0153] The above process is a process of calling the target operator for the first time. After the identification of the target operator is determined in the process of calling the target operator for the first time, the identification of the target operator is cached, and the identification of the target operator can be directly used to search for the target operator subsequently.

[0154] 4. The process of searching for the operator.

[0155] After the identification of the target operator is determined, the identification of the target operator is cached, and the identification of the cached target operator is obtained when the target operator is used again. The process of searching for the operator is shown in steps 801 to 803. Figure 8

[0156] Step 801: Obtain the identification of the target operator.

[0157] In this embodiment, the host obtains the identification of the cached target operator.

[0158] Step 802: Obtain the index of the target operator in the n-level service logic in the identification of the target operator, wherein the n-level service logic runs on the host, each service logic in the i+1 level of the n-level service logic has a subordinate relationship with a service logic in the i level, the service logic in the n level is bound with an executable handle, the executable handle is used to execute the service logic of the operator running on the external computing unit, i is greater than or equal to 1 and less than or equal to n-1.

[0159] In this embodiment, the host splits the identification of the target operator according to the length of the data field corresponding to the n levels to obtain n identifications, and the n identifications are the corresponding indexes of the target operator in the n-level service logic. For example, the n levels include three levels, and the length of the data field of the n levels is 1 bit. The 1-bit data is obtained continuously from the start position of the identification of the target operator, which is the corresponding index of the target operator in the n-level service logic.

[0160] Step 803: Based on the index, search for the service logic corresponding to the n levels of the target operator, and search for the executable handle bound with the service logic of the target operator in the n level.

[0161] In this embodiment, the host uses the determined index to determine the index of the target operator in the n levels in the n-level service logic step by step, and obtains the executable handle bound with the service logic of the target operator in the n level.

[0162] In an optional manner, the executable handle is determined using an information table. The processing process is as follows:

[0163] ​Using the index of the first level, the information of the target operator in the first service logic of the first level is determined in the information table of the first level, in which the storage address information of the first service logic in the host memory and the address information of the information sub-table corresponding to the service logic of the target operator in the second level are found. The host uses the storage address information to find the first service logic in the memory of the host, uses the address information of the information sub-table to find the information sub-table, in which the information of the target operator in the second service logic of the second level is found using the index of the second level, in which the storage address information of the second service logic in the host memory and the address information of the information sub-table corresponding to the third service logic of the target operator in the third level are found. In this way, the information of the target operator in the nth service logic of the nth level is found. From the information of the nth service logic, the storage address information of the nth service logic in the host memory is found, the nth service logic is found in the memory of the host using the storage address information, and then the executable handle bound by the nth service logic is obtained.

[0164] In this way, the service logics of the target operator in the first level to the nth level are run on the host, and the external computing unit finds the kernel function using the executable handle, executes the kernel function, and realizes the execution of the target operator.

[0165] For example, referring to Figure 9 , the information table shown in Figure 6 is taken as an example for description, the identification of the target operator is 000, the indexes of the Operation level, the Tactic level and the Kernel level are 0, 0 and 0 respectively. The information of the service logic with the identification of 0 is obtained in the Operation level, the address information of the information sub-table of the Tactic level recorded in the information is used to obtain the information of the service logic with the identification of 0 in the information sub-table of the Tactic level, and the address information of the information sub-table of the Kernel level recorded in the information is used to obtain the information of the service logic with the identification of 0 in the information sub-table of the Kernel level.

[0166] Using the method for finding the operator in the embodiments of the present disclosure, hash calculation and hash conflict solving are not needed, so that the operator can be quickly found, and the overall time for operator access and execution is reduced. Moreover, starting from the information table of the first level, the address of the information table of the next level is quickly obtained according to the index, and the executable handle of the kernel function on the side of the external computing unit can be accessed in constant time. In this way, in the case of high-frequency calling of the operator in the reasoning or training scene, the overall computing efficiency can be improved, and the performance stability can be improved.

[0167] In addition, in the hierarchical loading process of the operator, the identification of each level of the operator is synchronously allocated, so that the fast loading and subsequent indexing of the operator can be realized.

[0168] In addition, the disclosure is used for the underlying operator level software stack of the heterogeneous development platform, the calling interface of the operator can be obtained from the software development kit (SDK) calling interface header file provided by the chip, and the operator calling mode described in the SDK calling interface header file can be used for infringement evidence collection.

[0169] The device provided by the embodiment of the disclosure is described below.

[0170] Figure 10 The device provided by the embodiment of the disclosure is described below. Figure 5 The device provided by the embodiment of the disclosure is described below.

[0171] The loading module 1010 is configured to:

[0172] load the first service logic into the memory of the host, wherein the first service logic is a service logic of the to-be-loaded operator of the deep learning model running on the host, the first service logic is divided into n levels of service logic according to attributes, each service logic of an i+1 level in the n levels has a subordinate relationship with a service logic of an i level, a service logic of an n level is one-to-one bound with an executable handle, the executable handle is used to execute a service logic of the to-be-loaded operator running on the external computing unit, i is greater than or equal to 1 and less than or equal to n-1, and the foregoing process of loading the service logic running on the host can be specifically implemented.

[0173] load the second service logic into the memory of the host, and load the second service logic from the memory of the host into the memory of the external computing unit, wherein the second service logic is a service logic of the to-be-loaded operator running on the external computing unit, and the foregoing process of loading the service logic running on the external computing unit can be specifically implemented.

[0174] In an optional manner, the loading module 1010 is further configured to:

[0175] allocate continuous identifiers to all service logics of a first level in the n levels;

[0176] allocate continuous identifiers to service logics of an i+1 level subordinate to any service logic of the i level.

[0177] In an optional manner, the finding module 1020 is configured to:

[0178] obtain an identifier of a target operator in the to-be-loaded operators;

[0179] In the identifier of the target operator, an index of the target operator in the n-level service logic is obtained;

[0180] Based on the index, the corresponding service logic of the target operator in the n levels is found, and the executable handle bound by the service logic of the target operator in the nth level is found.

[0181] In an optional manner, the finding module 1020 is configured to obtain the cached identifier of the target operator.

[0182] In an optional manner, the finding module 1020 is further configured to, before obtaining the cached identifier of the target operator, determine the optimal service logic of the target operator in the n-level service logic based on parameter information of the target operator, wherein the parameter information includes one or more of specific parameters, data formats or data types used by the target operator, and tensor shape, the identifier of the optimal service logic of the target operator is composed of the identifiers of the target operator according to the hierarchical order of the n levels; and cache the identifier of the target operator.

[0183] In an optional manner, the finding module 1020 is configured to:

[0184] Obtain n information tables corresponding to the n-level service logic, wherein the information table corresponding to the i-level service logic includes the information of each service logic in the i-level, and for each service logic in the i-level, the information of the service logic includes the storage address information of the service logic and the address information of the information sub-table, which includes the information of the i+1-level service logic belonging to the service logic, and the information table of the nth level includes the information of each service logic in the nth level, and for each service logic in the nth level, the information of the service logic includes the storage address information of the service logic and the executable handle bound by the service logic;

[0185] Based on the n information tables and the index, the corresponding service logic of the target operator in the n levels is found, and the executable handle bound by the service logic of the target operator in the nth level is found.

[0186] Figure 11Figure 1 is a structural diagram of a device for searching an operator provided by an embodiment of the present disclosure. The device is applied to a host, the host is externally connected with a computing unit, and the device can be realized as part or all of the device by software, hardware, or a combination of both. The device provided by an embodiment of the present disclosure can implement the method provided by an embodiment of the present disclosure Figure 8 The flow, the device includes: an acquisition module 1110 and a determination module 1120, wherein:

[0187] The acquisition module 1110 is configured to: acquire an identifier of a target operator.

[0188] In the identifier of the target operator, an index of the target operator in n-level service logic is acquired, wherein the n-level service logic runs on the host, each service logic in an i+1 level in the n levels has a subordinate relationship with a service logic in an i level, a service logic in an n level is one-to-one bound with an executable handle, the executable handle is used to execute a service logic in which an operator runs on the external computing unit, i is greater than or equal to 1 and less than or equal to n-1, and the acquisition function of steps 801 and 802 and the implicit steps contained in steps 801 and 802 can be specifically used to implement;

[0189] The determination module 1120 is configured to: based on the index, search the target operator in the n-level corresponding service logic, and search an executable handle bound with the target operator in the n-level service logic, and the determination function of step 803 and the implicit steps contained in step 803 can be specifically used to implement.

[0190] In an optional manner, the acquisition module 1110 is configured to:

[0191] Acquire the identifier of the cached target operator.

[0192] In an optional manner, the acquisition module 1110 is further configured to:

[0193] Before acquiring the identifier of the cached target operator, based on parameter information of the target operator, the target operator optimal service logic is determined in the n-level service logic level by level, wherein the parameter information includes one or more of specific parameters, data formats or data types used by the target operator, and tensor shape, and the identifier of the target operator optimal service logic is composed of the identifier of the target operator according to the hierarchical order of the n levels; and the identifier of the target operator is cached.

[0194] In an optional manner, the determination module 1120 is configured to:

[0195] obtain n information tables corresponding to the n levels of business logics, wherein the information table corresponding to the i-th level of business logics comprises information of each business logic of the i-th level of business logics, and for each business logic of the i-th level of business logics, the information of the business logic comprises storage address information of the business logic and address information of an information sub-table comprising information of business logics of the i+1-th level of business logics subordinate to the business logic, and the information table of the n-th level of business logics comprises information of each business logic of the n-th level of business logics, and for each business logic of the n-th level of business logics, the information of the business logic comprises storage address information of the business logic and an executable handle to which the business logic is bound;

[0196] based on the n information tables and the index, find the target operator in the business logics corresponding to the n levels, and find the executable handle to which the target operator in the business logic of the n-th level is bound.

[0197] In an optional manner, the apparatus further comprises:

[0198] a loading module, configured to load, into the memory of the host, business logics of to-be-loaded operators of the deep learning model running on the host, wherein the n levels of business logics are obtained by dividing, according to attributes, business logics of the to-be-loaded operators running on the host, and the target operator belongs to the to-be-loaded operators; load, into the memory of the host, business logics of the to-be-loaded operators running on the external computing unit, load, from the memory of the host, the business logics of the to-be-loaded operators running on the external computing unit to the external computing unit, and assign continuous identifiers to all business logics of a first level of the n levels, and assign continuous identifiers to business logics of an i+1-th level subordinate to any business logic of an i-th level.

[0199] In an optional manner, the loading module is configured to load, into the memory of the host, business logics of all operators included in the deep learning model running on the host when the deep learning model is configured or when the deep learning model is running.

[0200] Figure 10 the loading operator device loads the detailed process of the loading operator and Figure 11 the finding operator device finds the detailed process of the finding operator, please refer to the description in the foregoing embodiments, which will not be repeated here. Figure 10 the loading operator device and Figure 11 the finding operator device is attached Figure 2 the computing device 200 in the computer system 100.

[0201] In some embodiments, the present disclosure provides a computer program product, which includes computer instructions stored in a computer-readable storage medium. A processor of a computing device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computing device performs Figure 3 、 Figure 4 and Figure 8 The process shown.

[0202] Those skilled in the art will appreciate that the various method steps and units described in conjunction with the embodiments disclosed in this disclosure can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the steps and components of each embodiment have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this disclosure.

[0203] In the several embodiments provided in the present disclosure, it should be understood that the disclosed system architecture, device and method can be implemented in other ways. For example, the device embodiment described above is only schematic. For example, the division of the module is only a logical function division. There may be other division methods in actual implementation. For example, multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or modules, or it can be an electrical, mechanical or other form of connection.

[0204] The modules described as separate components may or may not be physically separate, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the objectives of the embodiments of the present disclosure.

[0205] In addition, the modules in the various embodiments of the present disclosure may be integrated into a single processing module, or each module may exist physically separately, or two or more modules may be integrated into a single module. The integrated modules may be implemented in the form of hardware or software modules.

[0206] The integrated module, if implemented in the form of a software function module and sold or used as an independent product, can be stored in a computer readable storage medium. Based on such an understanding, the technical solutions of the present disclosure, essentially or in other words, the part that contributes to the prior art, or all or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods in the various embodiments of the present disclosure. The aforementioned storage medium includes: U disk, mobile hard disk, ROM, RAM, magnetic disk, or optical disk, and various other media that can store program codes.

[0207] The above description is only exemplary embodiments of the present disclosure, but the protection scope of the present disclosure is not limited thereto. Any person skilled in the art can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present disclosure, and these modifications or replacements should be covered within the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be subject to the protection scope of the claims.

Claims

1. A method of loading an operator, characterized by, The method is applied to a host connected with an external computing unit, and the method comprises: loading first service logic into a memory of the host, wherein the first service logic is service logic of a to-be-loaded operator of a deep learning model running on the host, the first service logic is divided into n levels of service logic according to attributes, each service logic of an i+1 level of the n levels has a subordinate relationship with a service logic of an i level, a service logic of an n level is bound with an executable handle one by one, the executable handle is used to execute service logic of the to-be-loaded operator running on the external computing unit, i is greater than or equal to 1 and less than or equal to n-1; loading second service logic into the memory of the host, and loading the second service logic from the memory of the host into a memory of the external computing unit, wherein the second service logic is service logic of the to-be-loaded operator running on the external computing unit.

2. The method of claim 1, wherein, The method further comprises: allocating continuous identifiers to all service logics of a first level of the n levels; allocating continuous identifiers to service logics of an i+1 level subordinate to any service logic of the i level.

3. The method according to claim 1 or 2, characterized in that, The method further comprises: obtaining an identifier of a target operator in the to-be-loaded operator; obtaining an index of the target operator in the n levels of service logic in the identifier of the target operator; based on the index, finding the target operator in the corresponding service logic of the n levels, and finding an executable handle bound with the target operator in the service logic of the n level.

4. The method of claim 3, wherein, The obtaining of the identifier of the target operator in the to-be-loaded operator comprises: obtaining the cached identifier of the target operator.

5. The method of claim 4, wherein, Before the obtaining of the cached identifier of the target operator, the method further comprises: based on parameter information of the target operator, determining the optimal service logic of the target operator in the n levels of service logic step by step, wherein the parameter information comprises one or more of specific parameters, data formats or data types used by the target operator, and tensor shape, and the identifier of the optimal service logic of the target operator is composed of the identifiers of the n levels according to the hierarchical order of the n levels; caching the identifier of the target operator.

6. The method according to any one of claims 3 to 5, characterized in that, The finding of the target operator in the corresponding service logic of the n levels based on the index, and the finding of the executable handle bound with the target operator in the service logic of the n level, comprise: obtain n information tables corresponding to the n levels of business logics, wherein the information table corresponding to the i-th level of business logics comprises information of each business logic of the i-th level of business logics, and for each business logic of the i-th level of business logics, the information of the business logic comprises storage address information of the business logic and address information of an information sub-table, the information sub-table comprising information of business logics of the i+1-th level of business logics subordinate to the business logic, and the information table of the n-th level of business logics comprises information of each business logic of the n-th level of business logics, and for each business logic of the n-th level of business logics, the information of the business logic comprises storage address information of the business logic and an executable handle to which the business logic is bound; based on the n information tables and the index, find the business logic of the target operator in the n levels and find the executable handle to which the business logic of the target operator in the n-th level is bound.

7. An apparatus for loading an operator, characterized by The device is applied to a host connected with an external computing unit, and the device comprises: a loading module, configured to: load a first business logic into a memory of the host, wherein the first business logic is a business logic of a to-be-loaded operator of a deep learning model running on the host, the first business logic is divided into n levels of business logics according to attributes, each business logic of an i+1-th level of the n levels has a subordinate relationship with a business logic of an i-th level, a business logic of an n-th level is bound to an executable handle one by one, the executable handle is used to execute a business logic of the to-be-loaded operator running on the external computing unit, i is greater than or equal to 1 and less than or equal to n-1; load a second business logic into the memory of the host and load the second business logic from the memory of the host into a memory of the external computing unit, wherein the second business logic is a business logic of the to-be-loaded operator running on the external computing unit.

8. The apparatus of claim 7, wherein, The loading module is further configured to: allocate continuous identifiers to all business logics of a first level of the n levels; allocate continuous identifiers to business logics of the i+1-th level subordinate to any business logic of the i-th level.

9. The apparatus of claim 7 or 8, wherein, The device further comprises a finding module, configured to: obtain an identifier of a target operator in the to-be-loaded operator; obtain an index of the target operator in the n levels of business logics in the identifier of the target operator; based on the index, find the business logic of the target operator in the n levels and find the executable handle to which the business logic of the target operator in the n-th level is bound.

10. The apparatus of claim 9, wherein, The finding module is configured to obtain the identifier of the target operator cached.

11. The apparatus of claim 10, wherein, The search module is further configured to, before obtaining the cached identifier of the target operator, determine optimal business logic of the target operator in the n-level business logic based on parameter information of the target operator, wherein the parameter information comprises one or more of specific parameters, data formats or data types used by the target operator, and tensor shape, and an identifier of the optimal business logic of the target operator is composed of identifiers of the n-level business logic in the hierarchical order of the n levels; and cache the identifier of the target operator.

12. The apparatus of any one of claims 9 to 11, wherein, The search module is configured to: obtain n information tables corresponding to the n-level business logic, wherein an information table corresponding to the i-level business logic comprises information of each business logic at the i-level, and for each business logic at the i-level, the information of the business logic comprises storage address information of the business logic and address information of an information sub-table comprising information of business logic at the i+1-level subordinate to the business logic, and an information table at the n-level comprises information of each business logic at the n-level, and for each business logic at the n-level, the information of the business logic comprises storage address information of the business logic and an executable handle bound to the business logic; based on the n information tables and the index, search for business logic corresponding to the target operator at the n levels, and search for an executable handle bound to the business logic of the target operator at the n-level.

13. A method of finding an operator, characterized by The method is applied to a host connected with an external computing unit, and the method comprises: obtaining an identifier of a target operator; in the identifier of the target operator, obtaining an index of the target operator in n-level business logic, wherein the n-level business logic runs on the host, each business logic at the i+1-level in the n levels has a subordinate relationship with one business logic at the i-level, and business logic at the n-level is bound one-to-one to an executable handle used to execute business logic of an operator running on the external computing unit, i is greater than or equal to 1 and less than or equal to n-1; based on the index, search for business logic corresponding to the target operator at the n levels, and search for an executable handle bound to the business logic of the target operator at the n-level.

14. The method of claim 13, wherein, The method comprises: obtaining an identifier of a target operator; 15. The method of claim 14, wherein, before obtaining the cached identifier of the target operator, determining optimal business logic of the target operator in the n-level business logic based on parameter information of the target operator, wherein the parameter information comprises one or more of specific parameters, data formats or data types used by the target operator, and tensor shape, and an identifier of the optimal business logic of the target operator is composed of identifiers of the n-level business logic in the hierarchical order of the n levels; and cache the identifier of the target operator. before obtaining the cached identifier of the target operator, the method further comprises: based on the parameter information of the target operator, determining optimal business logic of the target operator in the n-level business logic, wherein the parameter information comprises one or more of specific parameters, data formats or data types used by the target operator, and tensor shape, and an identifier of the optimal business logic of the target operator is composed of identifiers of the n-level business logic in the hierarchical order of the n levels; and cache the identifier of the target operator.

16. The method according to any one of claims 13 to 15, characterized in that, The method comprises the following steps: The method comprises the following steps: The method comprises the following steps:

17. An apparatus for finding an operator, the apparatus comprising: The device is applied to a host computer connected with an external computing unit, and the device comprises: An obtaining module is configured to obtain an identifier of a target operator; In the identifier of the target operator, an index of the target operator in n-level business logics is obtained, wherein the n-level business logics are run on the host computer, each business logic in an (i+1)th level of the n-level business logics has a subordinate relationship with a business logic in an ith level, a business logic in an nth level is one-to-one bound with an executable handle, the executable handle is used to execute a business logic run by an operator on the external computing unit, i is greater than or equal to 1 and less than or equal to n-1; A determining module is configured to, based on the index, find the target operator in corresponding business logics in the n levels and find an executable handle bound with the target operator in a business logic in the nth level.

18. The apparatus of claim 17, wherein, The obtaining module is configured to obtain the identifier of the target operator cached.

19. The apparatus of claim 18, wherein, Before obtaining the identifier of the target operator cached, the obtaining module is further configured to, based on parameter information of the target operator, determine an optimal business logic of the target operator in the n-level business logics step by step, wherein the parameter information comprises one or more of a specific parameter, a data format or a data type used by the target operator, a tensor shape, an identifier of the optimal business logic of the target operator is composed of the identifier of the target operator according to a hierarchical order of the n levels, and the identifier of the target operator is cached.

20. The apparatus of any one of claims 17 to 19, wherein, The determining module is configured to: obtaining n information tables corresponding to the n levels of business logics, wherein the information table corresponding to the i-th level of business logics comprises information of each business logic of the i-th level, and for each business logic of the i-th level, the information of the business logic comprises storage address information of the business logic and address information of an information sub-table, the information sub-table comprising information of business logics of the i+1-th level subordinate to the business logic, and the information table of the n-th level comprises information of each business logic of the n-th level, and for each business logic of the n-th level, the information of the business logic comprises storage address information of the business logic and an executable handle to which the business logic is bound; based on the n information tables and the index, searching for the target operator in the n levels of business logics, and searching for the executable handle to which the target operator in the n-th level of business logics is bound.

21. A computing device, comprising: A computer device comprising a processor and a memory, the processor being configured to execute instructions stored in the memory to cause the computer device to perform the method of loading an operator according to any one of claims 1 to 6, or the method of searching for an operator according to any one of claims 13 to 16.

22. A computer-readable storage medium, characterized in that, Computer program instructions, when executed by a computer device, cause the computer device to perform the method of loading an operator according to any one of claims 1 to 6, or the method of searching for an operator according to any one of claims 13 to 16.

Images