Method and apparatus for implementing operator based on deep learning framework, and related device

Abstract

The embodiment of the application provides a method, device and related equipment for implementing an operator based on a deep learning framework, which comprises the following steps: receiving a user-defined operator input by a user into the deep learning framework through an operator interface, wherein the user-defined operator comprises one or more function operations; scanning a plurality of native operators in a calculation graph to obtain a target native operator, wherein the target native operator corresponds to the user-defined operator; replacing the target native operator in the calculation graph in the deep learning framework with the user-defined operator; and loading the calculation graph comprising the user-defined operator to perform training. Through the embodiment of the application, the user-defined operator can be implemented through the operator interface, without the need of invasive modification on the native network of the deep learning framework, so that the openness of the user-defined behavior is facilitated.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computers, in particular to the field of artificial intelligence such as deep learning, and more particularly to a method and device for implementing an operator based on a deep learning framework and related equipment. BACKGROUND

[0002] As a basic framework in the field of artificial intelligence, the deep learning framework has gradually attracted more and more attention. However, the operator library in the deep learning framework generally cannot achieve completeness, and there is a situation that the current operator library cannot meet specific logical computing operations. Therefore, major deep learning frameworks provide users with a way to customize operators.

[0003] The first way to customize operators is to use a python front-end extension (such as torch.nn) to customize operators, which does not involve underlying C++ statement implementation. The second way is to use other third libraries (such as library pybind11) to build a shared library form of C++ and Compute Unified Device Architecture (CUDA) extension, which generally consists of three steps: 1. Use C++ to write forward calculation functions (forward functions) and backward calculation functions (backward functions) of the operator; 2. Bind the forward functions and backward functions of the operator to the python interface using pybind11; 3. Use a compilation tool to compile and package the C++ project into a dynamic link library. The third way is for torchscript users to customize operators, and the registration process of the operator is similar to the second way.

[0004] However, the above three ways are relatively complex for users to operate, and users need to understand the internal concepts of the deep learning framework, and even need to compile the entire deep learning framework. SUMMARY

[0005] To solve one or more technical problems as mentioned above, embodiments of the present application provide a method and device for implementing an operator based on a deep learning framework and related equipment in multiple aspects, which implement user-defined operators through an operator interface, do not need to make invasive modifications to the original network of the deep learning framework, and facilitate the openness of user-defined behaviors.

[0006] In a first aspect, embodiments of the present application provide a method for implementing an operator based on a deep learning framework, the deep learning framework comprising a computation graph, the computation graph comprising a plurality of native operators, the method comprising: receiving, by an operator interface, a user-defined operator input by a user to the deep learning framework, the user-defined operator comprising one or more function operations; scanning the plurality of native operators in the computation graph to obtain a target native operator, wherein the target native operator corresponds to the user-defined operator; replacing the target native operator in the computation graph in the deep learning framework with the user-defined operator; and loading the computation graph comprising the user-defined operator for training.

[0007] In a possible implementation, replacing the target native operator in the computation graph in the deep learning framework with the user-defined operator comprises: compiling the user-defined operator into a dynamic link library; setting an environment variable for loading the dynamic link library of the user-defined operator; and replacing the target native operator in the computation graph in the deep learning framework with the user-defined operator based on the dynamic link library.

[0008] In a possible implementation, loading the computation graph comprising the user-defined operator for training comprises: loading the dynamic link library to distribute the user-defined operator into a corresponding user-defined operator behavior; and training the computation graph comprising the user-defined operator behavior.

[0009] In a possible implementation, the method further comprises: if the environment variable of the user-defined operator is not set, training the computation graph by loading the native operator.

[0010] In a possible implementation, the method further comprises: obtaining a parameter of the user-defined operator generated in a training process of the computation graph of the user-defined operator; and identifying the parameter of the user-defined operator by a naming manner of the deep learning framework.

[0011] In a possible implementation, the method further comprises: saving the parameter of the user-defined operator in an order of calling the user-defined operator, to debug or analyze the user-defined operator; wherein the order of calling the user-defined operator is an order of calling the user-defined operator when loading the computation graph comprising the user-defined operator for training.

[0012] In a second aspect, embodiments of the present application provide a device for implementing a user-defined operator, the device comprising: a communication unit configured to receive, by an operator interface, a user-defined operator input by a user to a deep learning framework, the user-defined operator comprising one or more function operations; a processing unit configured to scan a plurality of native operators in the computation graph to obtain a target native operator, wherein the target native operator corresponds to the user-defined operator; the processing unit is further configured to replace the target native operator in the computation graph in the deep learning framework with the user-defined operator; and the processing unit is further configured to load the computation graph comprising the user-defined operator for training.

[0013] In a possible implementation, the processing unit is specifically configured to: compile the custom operator into a dynamic link library; set an environment variable, wherein the environment variable is used to load the dynamic link library of the custom operator; and replace a target native operator in a computation graph in the deep learning framework with the custom operator through the dynamic link library.

[0014] In a possible implementation, the processing unit is specifically configured to: load the dynamic link library to distribute the custom operator into a corresponding custom operator behavior; and train the computation graph containing the custom operator behavior.

[0015] In a possible implementation, the processing unit is specifically configured to: if the environment variable of the custom operator is not set, train the computation graph by loading the native operator.

[0016] In a possible implementation, the processing unit is further configured to: obtain a parameter of the custom operator generated in a training process of a computation graph containing the custom operator; and identify the parameter of the custom operator through a naming manner of the deep learning framework.

[0017] In a possible implementation, the processing unit is further configured to: save the parameter of the custom operator according to a calling sequence of the custom operator, to debug or analyze the custom operator; wherein the calling sequence of the custom operator is a sequence of calling the custom operator when training the computation graph containing the custom operator.

[0018] In a third aspect, an embodiment of the present application provides a computer readable storage medium, the computer readable storage medium storing a computer program, when the computer program runs on one or more processors, executes the method in the embodiments of the first aspect or the second aspect.

[0019] In a fourth aspect, an embodiment of the present application provides a chip system, the chip system comprising at least one processor, a memory and an interface circuit, the memory storing a computer program, when the computer program runs on one or more processors, executes the method in the embodiments of the first aspect or the second aspect.

[0020] In the above, the operator interface is an interface defined in advance for receiving a custom operator input by a user to a deep learning framework, and replacing a preset operator in a native model with a custom operator can load the native model for training without the user's awareness, avoiding intrusive modification to the native model. BRIEF DESCRIPTION OF DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the accompanying drawings needed to be used in the description of the embodiments or the prior art will be briefly introduced. Obviously, the accompanying drawings in the following description only only some embodiments of the present application, and for those skilled in the art, other drawings can be obtained without creative labor on the basis of these drawings.

[0022] Figure 1 is a system architecture schematic diagram of an operator implemented based on a deep learning framework provided by an embodiment of the present application;

[0023] Figure 2 is a method flow schematic diagram of an operator implemented based on a deep learning framework provided by an embodiment of the present application;

[0024] Figure 3 is a device structure schematic diagram of an operator implemented based on a deep learning framework provided by an embodiment of the present application;

[0025] Figure 4 is a structure schematic diagram of a computing device provided by an embodiment of the present application;

[0026] Figure 5 is a structure schematic diagram of a board card provided by an embodiment of the present application. DETAILED DESCRIPTION

[0027] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0028] In this document, the term "embodiment" means that the specific features, structures or characteristics described in connection with the embodiment can be included in at least one embodiment of the present application. The appearance of this phrase at various places in the specification does not necessarily mean that the same embodiment is referred to at each occurrence of the phrase, nor does it mean that other embodiments are mutually exclusive or alternative to the embodiment described. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0029] Some concepts that may be involved in the embodiments of the present application will be briefly introduced below.

[0030] 1.An operator is a basic computing unit in a deep learning framework (which can be referred to as a framework for short), and the input of the operator includes a tensor or a combination of tensors, and the operator completes a specific function by performing a calculation on the tensor or the combination of tensors, such as a convolution operator, a pooling operator, a quantization operator, a relu operator, and the like. In the deep learning framework, a tensor is used to represent data, and the tensor can be understood as a multi-dimensional array. The process of building a model using the deep learning framework can also be a process of combining different operators to meet a specific computing goal, and the richness of the operators is an important indicator for measuring the completeness of the functions of the deep learning framework product. Some specific fields require many customized, non-generic, and non-basic operators that can be combined. For example, models in the fields of 3D vision, optical flow, and the like, at which time the custom operator function of the framework needs to be used to extend and develop the framework, so that the framework can be used in specific fields.

[0031] 2.The deep learning framework is an interface, a library, or a function that can more easily and quickly build a deep learning model without understanding the details of the underlying algorithm. The deep learning framework defines a model using a set of pre-built and optimized components. The deep learning framework includes one or more of the following: TensorFlow, Keras, PyTorch, Caffe, Deeplearning4j, and the like.

[0032] In order to apply a custom operator to the deep learning framework, the following steps generally need to be completed:

[0033] (1) Operator computing logic implementation. In this step, the operator can include computing logic implemented based on C++.

[0034] (2) Associate the operator to the framework. This step is to register or associate the computing logic of the operator to the framework in a certain way, such as binding to the interface of the framework through pybind11, so that the operator can be called by the deep learning framework.

[0035] (3) Compile the operator and use it in the computation graph. This step needs to complete the compilation of the operator so that it can be applied to the actual scene. Users need to download the complete framework code to compile the operator independently, and the entire deep learning framework needs to be compiled, which has a significant gap in convenience.

[0036] However, the writing method of the custom operator of the deep learning framework is too complex, and too many concepts need to be understood. Therefore, it can be difficult for users who are not familiar with the deep learning framework to extend the custom operator under the deep learning framework.

[0037] Therefore, an embodiment of the present application provides a system. Please refer to Figure 1 , Figure 1is a system architecture schematic diagram of implementing an operator based on a deep learning framework provided by an embodiment of the present application. From Figure 1 It can be seen that the system can include a computing device 100. Wherein the computing device 100 can be an electronic device with data processing capability and data transceiving capability, the computing device 100 can include an operator interface 101 and a deep learning framework 102. The deep learning framework 102 includes a computation graph, and the computation graph includes a plurality of native operators. The deep learning framework 101 is configured in the computing device 100, and the deep learning framework 101 can provide the operator interface 102. Wherein the operator interface 102 is a pre-designed registration interface in the deep learning framework 101 in the computing device 100, through which information can be input. The operator interface 102 can be implemented by macros, static objects or methods, etc. The computing device 100 can receive the custom operator input by the user to the deep learning framework 102 through the operator interface 102. Wherein the operator can include the computing logic implemented based on C++ / CPP, such as one or more of convolution operators, pooling operators, quantization operators, relu operators, fully connected operators, etc. Therefore, the custom operator can include one or more function operations (such as convolution, pooling, quantization, relu, etc.). It can be understood that the "custom" mentioned in the embodiments of the present application is constructed by the user based on the function to be implemented. For example, in deep learning, quantization operation is a process of mapping a set of original value domain values to another target value domain range through a mathematical transformation. For example, in the embodiments of the present application, the parameters in the computation graph are converted from floating-point numbers to integer numbers. And the custom quantization can include the quantization process of converting the parameters in the computation graph from floating-point numbers to integer numbers constructed by the user.

[0038] In order to apply the input custom operator to the deep learning framework 102, the computing device 100 can scan the plurality of native operators in the computation graph to obtain a target native operator. Wherein the target native operator corresponds to the custom operator, that is, the target native operator and the custom operator implement the same function, only the implementation method is different. Then, the computing device 100 can replace the target native operator in the deep learning framework 102 with the above-mentioned custom operator, and then load the computation graph including the custom operator for training. It can be understood that the computation graph can include a computation graph for classification, a computation graph for detection or a computation graph for segmentation according to the actual needs of the user.

[0039] The computing device 100 can obtain the parameters of the custom operator generated by the custom operator in the training process of the computation graph containing the custom operator. For example, if the custom operator is a quantization operator, the parameters of the custom operator can include one or more of the following: position, scale, bit width, etc. The parameters of the custom operator can be generated by the neural network model containing the custom operator during operation. Then, the computing device 100 can identify the custom parameters of the custom operator by the naming method of the deep learning framework. For example, if a deep learning framework has three consecutive convolutional layers and one fully connected layer, the custom operator can be identified as: conv1 custom operator, conv2 custom operator, conv3 custom operator, and cnn custom operator. Different deep learning frameworks can have different naming methods. It can be understood that using the naming method of the deep learning framework can reduce the learning cost and enable the native visualization mechanism, such as the deep learning framework tensorboard, to clearly show the correspondence between the operator and the saved quantization parameters.

[0040] The computing device 100 can also save the parameters of the custom operator according to the calling order of the custom operator to debug or analyze the custom operator. The calling order of the custom operator is the order in which the custom operator is called when the computation graph containing the custom operator is loaded for training.

[0041] Referring to Figure 2 , Figure 2 is a method flow diagram for implementing an operator based on a deep learning framework provided by an embodiment of the present application. The method can be implemented based on the system architecture described above, and specifically implemented by the computing device 100 as shown in Figure 1 Figure 1 The method for implementing an operator based on a deep learning framework as shown in Figure 2 includes but is not limited to the following steps:

[0042] Step S201 receives the custom operator input by the user to the deep learning framework through the operator interface.

[0043] Specifically, when the user inputs the custom operator to the deep learning framework through the electronic device, the computing device can receive the custom operator input by the user to the deep learning framework through the operator interface. It can be understood that the deep learning framework is pre-configured in the computing device, and the deep learning framework can provide an operator interface implemented by macros, static objects or methods, etc. Therefore, the input of the custom operator can be realized through the operator interface.

[0044] Step S202 scans a plurality of native operators in the computation graph to obtain a target native operator. ​

[0045] Specifically, the deep learning framework includes a computation graph, and the computation graph includes a plurality of native operators. It can be understood that the native model is a common network structure, and the computation graph is an internal representation of the native model after being loaded into the deep learning framework. The native operator is an operator preset in the computation graph, and the operator can include a calculation logic implemented based on C++ / CPP, such as one or more of a convolution operator, a pooling operator, a quantization operator, a relu operator, and a full connection operator. Therefore, the native operator can include one or more function parameters preset in the computation graph. According to actual needs, the computing device scans the plurality of native operators in the computation graph based on the pytorch framework to obtain a target native operator. It can be understood that if the custom operator includes a quantization operator and a convolution operator, the target native operator obtained by the computing device also includes the quantization operator and the convolution operator. That is, the target native operator corresponds to the custom operator.

[0046] In step S203, the target native operator in the computation graph in the deep learning framework is replaced by the custom operator.

[0047] Specifically, the computing device can replace the target native operator in the computation graph in the deep learning framework by the custom operator through a custom just in time pass (JIT pass). It can be understood that the computation graph can be preconfigured in the deep learning framework. Because the JIT layer does not perceive the custom operator, the implementation of the custom operator is realized by loading the computation graph for training, avoiding intrusive modification of the computation graph. The computation graph can include a computation graph for classification, a computation graph for detection, or a computation graph for segmentation according to actual needs of a user.

[0048] In one possible implementation, the computing device can compile the custom operator into a dynamic link library, that is, compile the custom operator to form a dynamic link library of the custom operator. In the embodiment of the present application, by preprocessing, compiling, assembling, and linking the C++ file included in the custom operator, the custom operator can be compiled to form the dynamic link library of the custom operator. It should be noted that the deep learning framework generally needs to design a front-end system (Python) and a back-end system (C++). The custom operator received by the user includes back-end C++ code. In this case, in order to be called by the front-end Python, it needs to be compiled into a file in a specific format to be loaded into the memory by the Python front-end system and to call the C++ back-end system to implement the calculation logic (such as quantization, convolution, etc.) of the operator.

[0049] Further, the device can replace the target native operator in the computation graph in the deep learning framework with the custom operator by setting an environment variable. If the environment variable is set, the target native operator in the computation graph in the deep learning framework is replaced with the custom operator based on the dynamic link library. The dynamic loading manner facilitates user use, and different algorithms can be used by setting and resetting the environment variable without recompiling the deep learning framework to dynamically change the operator behavior. For example, if the device receives a command parameter for setting or resetting the environment variable, the target native operator in the computation graph can be replaced with the custom operator; if the device does not receive the command parameter for setting or resetting the environment variable, the target native operator in the computation graph will not be changed.

[0050] In step S204, the computation graph containing the custom operator is loaded for training.

[0051] Specifically, when the computation device loads the computation graph containing the custom operator for training, it checks whether the environment variable identifying the user-defined operator is set, and decides whether to load the dynamic link library of the custom operator according to this.

[0052] If the environment variable corresponding to the custom operator is set, the computation device can distribute the custom operator to the operator behavior defined in advance by the deep learning framework or input by the user by loading the dynamic link library corresponding to the custom operator, so as to train the computation graph containing the custom operator.

[0053] If the environment variable corresponding to the custom operator is not set, the computation device trains the computation graph by loading the preset native operator in the deep learning framework.

[0054] As one possible implementation, the computing device can generate parameters for custom operators after loading a computation graph containing custom operators for training, or during training. The deep learning framework within the computing device can maintain a global quantization parameter table. Therefore, the computing device can obtain the parameters of the custom operators generated during training from the computation graph containing custom operators, and identify these parameters using the naming conventions of the deep learning framework. For example, if a depth consists of three consecutive convolutional layers and one fully connected layer, the custom operators can be identified as: conv1 custom operator, conv2 custom operator, conv3 custom operator, and cnn custom operator, respectively. Different deep learning frameworks may use different naming conventions. That is, the computing device can use the operator naming conventions in the native PyTorch JIT graph to identify and record the parameters of each custom operator. Using the native PyTorch framework for custom operator commands reduces the learning curve for users and allows the use of native visualization mechanisms, such as the TensorBoard deep learning framework, to clearly see the correspondence between custom operators and saved parameters.

[0055] As one possible implementation, the deep learning framework of the computing device can provide an interface to serialize and save operators with custom parameters according to the order in which they are called, enabling debugging and analysis of custom operators. It's understood that the user can input multiple custom operators, so the order in which these operators are called corresponds to the order in which the computing device invokes them during training when loading a computation graph containing these operators. The computing device can save the parameters of the custom operators in a visual JSON format, with the saving order matching the order in which they are called within the network. This increases readability and further facilitates user operation and debugging.

[0056] Please see Figure 3 , Figure 3 This is a schematic diagram of a device 300 for implementing a user-defined operator according to an embodiment of this application. The device 300 can be the aforementioned computing device, or a component within the computing device, such as a chip or integrated circuit. Figure 3 As shown, the apparatus 300 for implementing user-defined operators may include a communication unit 301 and a processing unit 302. The descriptions of each unit are as follows:

[0057] The communication unit 301 is used to receive custom operators input by the user into the deep learning framework through the operator interface. The custom operators contain one or more function operations.

[0058] The processing unit 302 is configured to scan a plurality of native operators in a computation graph to obtain a target native operator, where the target native operator corresponds to the custom operator.

[0059] The processing unit 302 is further configured to replace the target native operator in the computation graph in the deep learning framework with the custom operator.

[0060] The processing unit 302 is further configured to load the computation graph containing the custom operator for training.

[0061] In a possible implementation, the processing unit 301 is specifically configured to: compile the custom operator into a dynamic link library; set an environment variable, where the environment variable is used to load the dynamic link library of the custom operator; and replace the target native operator in the computation graph in the deep learning framework with the custom operator based on the dynamic link library.

[0062] In a possible implementation, the processing unit 301 is specifically configured to: load the dynamic link library to distribute the custom operator to a corresponding custom operator behavior; and train the computation graph containing the custom operator behavior.

[0063] In a possible implementation, the processing unit 301 is further configured to: if the environment variable of the custom operator is not set, train the computation graph by loading the native operator.

[0064] In a possible implementation, the processing unit 301 is further configured to: obtain a parameter of the custom operator generated in a training process of the computation graph containing the custom operator; and identify the parameter of the custom operator by a naming manner of the deep learning framework.

[0065] In a possible implementation, the processing unit 301 is further configured to: save the parameter of the custom operator according to a calling sequence of the custom operator, to debug or analyze the custom operator; where the calling sequence of the custom operator is a sequence of calling the custom operator when training the computation graph containing the custom operator.

[0066] It should be noted that the implementation of each unit can also correspond to the description of the corresponding unit in the embodiments shown in Figure 2 .

[0067] For details, please refer to Figure 4 , Figure 4Fig. 1 is a structural schematic diagram of a computing device 400 provided by an embodiment of the present application. The computing device 400 can be a node or a device in the node, such as a chip or an integrated circuit, etc. The computing device 400 can include at least one memory 401 and at least one processor 402. Optionally, the computing device 400 can further include a bus 403. Further optionally, the computing device 400 can further include a communication interface 404, wherein the memory 401, the processor 402 and the communication interface 404 are connected through the bus 403.

[0068] The memory 401 is configured to provide a storage space, in which data such as an operating system and a computer program can be stored. The memory 401 can be one or a combination of a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a compact disc read-only memory (CD-ROM), etc.

[0069] The processor 402 is a module configured to perform arithmetic operations and / or logical operations, and can be one or a combination of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor unit (MPU), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a complex programmable logic device (CPLD), etc.

[0070] The communication interface 404 is configured to receive data transmitted from an external device and / or transmit data to an external device, and can be a wired link interface such as an Ethernet cable, or a wireless link (Wi-Fi, Bluetooth, universal wireless transmission, etc.) interface. Optionally, the communication interface 404 can further include a transmitter (such as a radio frequency transmitter, an antenna, etc.) coupled with the interface, or a receiver, etc.

[0071] In some possible implementations, the processor 402 in the computing device 400 is configured to read a computer program stored in the memory 401, and execute the aforementioned screen encoding method, for example Figure 2The embodiment describes an implementation of a user-defined operator. Specifically for execution:

[0072] Receiving a user-defined operator input by the user to the deep learning framework through the operator interface, the user-defined operator comprising one or more function operations;

[0073] Scanning a plurality of native operators in the computation graph to obtain a target native operator, wherein the target native operator corresponds to the user-defined operator;

[0074] Replacing the target native operator in the computation graph in the deep learning framework with the user-defined operator;

[0075] Loading the computation graph comprising the user-defined operator for training.

[0076] In a possible implementation, the processor 402 is specifically configured to: compile the user-defined operator into a dynamic link library; set an environment variable, wherein the environment variable is used to load the dynamic link library of the user-defined operator; and replace the target native operator in the computation graph in the deep learning framework with the user-defined operator based on the dynamic link library.

[0077] In a possible implementation, the processor 402 is specifically configured to: load the dynamic link library to distribute the user-defined operator into a corresponding user-defined operator behavior; and train the computation graph comprising the user-defined operator behavior.

[0078] In a possible implementation, the processor 402 is further configured to: if the environment variable of the user-defined operator is not set, train the computation graph by loading the native operator.

[0079] In a possible implementation, the processor 402 is further configured to: obtain a parameter of the user-defined operator generated in a training process of the computation graph comprising the user-defined operator; and identify the parameter of the user-defined operator through a naming manner of the deep learning framework.

[0080] In a possible implementation, the processor 402 is further configured to: save the parameter of the user-defined operator according to a calling sequence of the user-defined operator, so as to debug or analyze the user-defined operator; wherein the calling sequence of the user-defined operator is a sequence of calling the user-defined operator when training the computation graph comprising the user-defined operator.

[0081] It should be noted that the implementation of each unit can also correspond to the description of the corresponding embodiment shown in Figure 2 .

[0082] Please refer to Figure 5 , Figure 5 is a structural schematic diagram of a board card 500 provided by an embodiment of the present application. From Figure 5As can be seen, the board card 500 includes a storage device 504 for storing data, which includes one or more storage units 510. The storage device 504 can be connected and transmit data with the control device 508 and the chip 502 through, for example, a bus or the like. Further, the board card also includes an external interface device 506, which is configured for the data relay or transfer function between the chip (or the chip in the chip package structure) and the external equipment 512 (for example Figure 1 the user equipment shown, or a server or computer, etc.). For example, the custom operator can be passed to the chip by the external equipment through the external interface device. For another example, the calculation result of the chip can be transmitted back to the external equipment via the external interface device. According to different application scenarios, the external interface device can have different interface forms, for example, it can adopt a standard PCIE interface, etc.

[0083] The chip 502 can be a system on chip (SoC), and integrated with one or more devices for implementing the custom operator as shown in Figure 4 . The chip can be connected with other related components through the external interface device (such as the external interface device 506 shown in Figure 5 ). The related components can be, for example, a camera, a display, a mouse, a keyboard, a network card or a wifi interface. In some application scenarios, other processing units and / or interface modules (such as DRAM interface) can be integrated on the chip. The chip 502 can replace the preset operator in the original model with the custom operator, and then load the original model of the custom operator for training. Wherein, the preset operator can be stored in the storage device 504.

[0084] In one or more embodiments, the control device in the board card of the present disclosure can be configured to regulate the state of the chip. For this purpose, in one application scenario, the control device can include a micro controller unit (MCU) for regulating the working state of the chip.

[0085] According to the above description in combination with Figure 3 and Figure 4 , those skilled in the art can understand that the present disclosure also discloses an electronic device or apparatus, which can include one or more of the above-mentioned board cards, one or more of the above-mentioned chips and / or one or more of the above-mentioned combined processing devices.

[0086] According to different application scenarios, the electronic device or apparatus of the present disclosure can include a server, a cloud server, a server cluster, a data processing apparatus, a robot, a computer, a tablet computer, a smart terminal, a PC device, an Internet of Things terminal, a mobile terminal, a mobile phone, a wearable device, a visual terminal, an autonomous driving terminal, a vehicle, a household appliance, and / or a medical device. Further, the electronic device or apparatus of the present disclosure can also be used in cloud, edge, terminal, and other application scenarios related to artificial intelligence, big data, and / or cloud computing. In one or more embodiments, the electronic device or apparatus with high computing power according to the present disclosure can be applied to a cloud device (such as a cloud server), and the electronic device or apparatus with low power consumption can be applied to a terminal device and / or an edge device (such as a smart phone or a camera). In one or more embodiments, the hardware information of the cloud device and the hardware information of the terminal device and / or the edge device are compatible with each other, so that suitable hardware resources can be matched from the hardware resources of the cloud device according to the hardware information of the terminal device and / or the edge device to simulate the hardware resources of the terminal device and / or the edge device, so as to complete unified management, scheduling and collaborative work of end-cloud integration or cloud-edge integration.

[0087] It should be noted that, for the purpose of simplicity, the present disclosure describes some methods and embodiments thereof as a series of actions and combinations thereof, but those skilled in the art can understand that the schemes of the present disclosure are not limited by the order of the described actions. Therefore, those skilled in the art can understand that some steps can be executed in other orders or simultaneously according to the disclosure or teaching of the present disclosure. Further, those skilled in the art can understand that the described embodiments of the present disclosure can be regarded as optional embodiments, i.e., the actions or modules involved therein are not necessarily essential for the implementation of one or more schemes of the present disclosure. In addition, the description of some embodiments of the present disclosure also has different focuses according to different schemes. Therefore, those skilled in the art can understand that the parts not described in detail in one embodiment of the present disclosure can also refer to the relevant description of other embodiments.

[0088] In terms of specific implementation, based on the disclosure and teachings of the present disclosure, those skilled in the art can understand that the disclosed embodiments of the present disclosure can also be implemented in other manners not disclosed herein. For example, in terms of the units in the aforementioned electronic device or apparatus embodiments, the division of the units is merely logical function division, and there can be other division manners in actual implementation. For example, a plurality of units or components can be combined or integrated into another system, or a plurality of features or functions of some units or components can be selectively disabled. In terms of the connection relationship between the units or components, the units or components discussed in the foregoing embodiments can be directly or indirectly coupled to each other. In some scenarios, the aforementioned direct or indirect coupling can refer to communication connection between units or components through an interface, where the interface can support electrical, optical, acoustic, magnetic or other forms of signal transmission.

[0089] In the present disclosure, the units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units. The aforementioned components or units can be located in the same position or distributed on a plurality of network units. In addition, according to actual needs, part or all of the units can be selected to achieve the purpose of the present disclosure embodiment scheme. In addition, in some scenarios, multiple units in the present disclosure embodiment can be integrated into one unit or physically exist separately.

[0090] In some implementation scenarios, the integrated units described above can be implemented in the form of software program modules. If implemented in the form of software program modules and sold or used as independent products, the integrated units can be stored in a computer readable memory. Based on this, when the scheme of the present disclosure is embodied in the form of a software product (for example, a computer readable storage medium), the software product can be stored in a memory, which can include a plurality of instructions for causing a computer device (for example, a personal computer, a server or a network device, etc.) to execute part or all of the steps of the method of the present disclosure embodiment. The aforementioned memory can include, but is not limited to, a U disk, a flash disk, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk, and various media that can store program codes.

[0091] In some other implementation scenarios, the above-mentioned integrated units can also be implemented in the form of hardware, i.e., specific hardware circuits, which can include digital circuits and / or analog circuits, etc. The physical implementation of the hardware structure of the circuit can include but is not limited to physical devices, and the physical devices can include but are not limited to transistors or memristors, etc. In view of this, various apparatuses (such as computing apparatuses or other processing apparatuses) herein can be implemented by appropriate hardware processors, such as CPUs, GPUs, FPGAs, DSPs, ASICs, etc. Further, the aforementioned storage units or storage devices can be any appropriate storage medium (including magnetic storage media or magneto-optical storage media, etc.), which may, for example, be a variable resistance memory (Resistive Random Access Memory, RRAM), a dynamic random access memory (Dynamic Random Access Memory, DRAM), a static random access memory (Static Random Access Memory, SRAM), an enhanced dynamic random access memory (Enhanced Dynamic Random Access Memory, EDRAM), a high bandwidth memory (High Bandwidth Memory, HBM), a hybrid memory cube (Hybrid Memory Cube, HMC), a ROM, and a RAM, etc.

[0092] The foregoing can be better understood in light of the following clauses:

[0093] Clause A1, a method for implementing an operator based on a deep learning framework, the deep learning framework comprising a computation graph, the computation graph comprising a plurality of native operators, the method comprising:

[0094] receiving, through an operator interface, a user inputted custom operator to the deep learning framework, the custom operator comprising one or more function operations;

[0095] scanning the plurality of native operators in the computation graph to obtain a target native operator, wherein the target native operator corresponds to the custom operator;

[0096] replacing the target native operator in the computation graph in the deep learning framework with the custom operator;

[0097] loading the computation graph comprising the custom operator for training.

[0098] Clause A2, the method of clause A1, the replacing the target native operator in the computation graph in the deep learning framework with the custom operator comprising:

[0099] compiling the custom operator into a dynamic link library;

[0100] setting an environment variable, wherein the environment variable is used to load a dynamic link library of the custom operator;

[0101] replacing a target native operator in a computation graph in the deep learning framework with the custom operator based on the dynamic link library.

[0102] Clause A3, the method of clause A2, wherein the loading the computation graph containing the custom operator for training comprises:

[0103] loading the dynamic link library to distribute the custom operator into a corresponding custom operator behavior;

[0104] training the computation graph containing the custom operator behavior.

[0105] Clause A4, the method of any one of clauses A1-A3, further comprising:

[0106] training the computation graph by loading the native operator if the environment variable of the custom operator is not set.

[0107] Clause A5, the method of clause A1, further comprising:

[0108] obtaining parameters of the custom operator generated in a training process of the computation graph containing the custom operator;

[0109] identifying the parameters of the custom operator by a naming manner of the deep learning framework.

[0110] Clause A6, the method of clause A5, further comprising:

[0111] saving the parameters of the custom operator in an order of calling the custom operator to debug or analyze the custom operator;

[0112] wherein the order of calling the custom operator is an order of calling the custom operator when loading the computation graph containing the custom operator for training.

[0113] Clause A7, an apparatus for implementing an operator based on a deep learning framework, the deep learning framework comprising a computation graph, the computation graph comprising a plurality of native operators, the apparatus comprising:

[0114] a communication unit configured to receive a custom operator input by a user to the deep learning framework through an operator interface, the custom operator comprising one or more function operations;

[0115] a processing unit, configured to scan a plurality of native operators in the computation graph to obtain a target native operator, wherein the target native operator corresponds to the custom operator;

[0116] the processing unit, further configured to replace the target native operator in the computation graph in the deep learning framework with the custom operator;

[0117] the processing unit, further configured to load the computation graph containing the custom operator for training.

[0118] Clause A8. A computing device, comprising a processor and a memory;

[0119] the memory stores a computer program;

[0120] the processor executes the computer program, the computing device performs the method of any one of preceding clauses A1-A6.

[0121] Clause A9. A computer-readable storage medium storing a computer program which, when executed on one or more processors, performs the method of any one of clauses A1-A6.

[0122] Clause A10. A chip system comprising at least one processor, a memory, and an interface circuit, the memory storing a computer program which, when executed on one or more processors, performs the method of any one of clauses A1-A6.

[0123] While several embodiments of the disclosure have been shown and described herein, it is to be understood that all such embodiments are merely illustrative of the principles of the disclosure. Numerous modifications, changes and adaptations will now occur to those skilled in the art without departing from the spirit and scope of the disclosure. It should be understood that each of the examples described herein and relating to one aspect of the disclosure can be employed in any other aspect of the disclosure, in any appropriate combination. It is therefore intended that the disclosure not be limited to the illustrative examples and embodiments set forth herein. Rather, the scope of the disclosure is to be according to the language of the claims that follow, and any equivalents to the claims.

Claims

1. A method for implementing an operator based on a deep learning framework, characterized in that, The deep learning framework comprises a computation graph comprising a plurality of native operators, and the method comprises: receiving, by an operator interface, a user-inputted custom operator to the deep learning framework, the custom operator comprising one or more function operations; scanning the plurality of native operators in the computation graph to obtain a target native operator, wherein the target native operator corresponds to the custom operator; replacing the target native operator in the computation graph in the deep learning framework with the custom operator; loading the computation graph comprising the custom operator for training; the replacing the target native operator in the computation graph in the deep learning framework with the custom operator comprises: compiling the custom operator into a dynamic link library; setting an environment variable, wherein the environment variable is used to load the dynamic link library of the custom operator; based on the dynamic link library, replacing the target native operator in the computation graph in the deep learning framework with the custom operator.

2. The method of claim 1, wherein, the loading the computation graph comprising the custom operator for training comprises: loading the dynamic link library to distribute the custom operator into a corresponding custom operator behavior; training the computation graph comprising the custom operator behavior.

3. The method according to claim 1 or 2, characterized in that, The method further comprises: if the environment variable of the custom operator is not set, training the computation graph by loading the native operator.

4. The method of claim 1, wherein, The method further comprises: obtaining parameters of the custom operator generated in a training process of the computation graph comprising the custom operator; identifying the parameters of the custom operator by a naming manner of the deep learning framework.

5. The method of claim 4, wherein, The method further comprises: saving the parameters of the custom operator in an order of calling the custom operator to debug or analyze the custom operator; wherein the order of calling the custom operator is an order of calling the custom operator when loading the computation graph comprising the custom operator for training.

6. An apparatus for implementing an operator based on a deep learning framework, the apparatus comprising: The deep learning framework comprises a computation graph comprising a plurality of native operators, and the apparatus comprises: a communication unit configured to receive, by an operator interface, a user-inputted custom operator to the deep learning framework, the custom operator comprising one or more function operations; a processing unit configured to scan the plurality of native operators in the computation graph to obtain a target native operator, wherein the target native operator corresponds to the custom operator; the processing unit is further configured to replace the target native operator in the computation graph in the deep learning framework with the custom operator; the processing unit is further configured to load the computation graph comprising the custom operator for training; the processing unit is specifically configured to compile the custom operator into a dynamic link library; set an environment variable, wherein the environment variable is used to load the dynamic link library of the custom operator; based on the dynamic link library, replace the target native operator in the computation graph in the deep learning framework with the custom operator.

7. A computing device, comprising: The computing device comprises a processor and a memory; the memory stores a computer program; The computing device, when the processor executes the computer program, executes the method of any one of preceding claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer readable storage medium has stored therein a computer program which, when executed on one or more processors, performs the method of any one of claims 1 to 5.

9. A chip system, characterized by The chip system comprises at least one processor, a memory and an interface circuit, and the memory has stored therein a computer program which, when executed on one or more processors, performs the method of any one of claims 1 to 5.

Images