Compiling method of deep learning model, electronic device and storage medium

Abstract

The embodiment of the application provides a kind of compilation method of deep learning model, electronic equipment and storage medium.Method includes: the deep learning model is converted into calculation graph;According to the order of the calculation node in calculation graph, each original operator contained in calculation node is converted into target language fragment, and initial fragment set is obtained;At least for initial fragment set, multiple identical target language fragments are filtered into single target language fragment, and filtered fragment set is obtained;Export the compiled deep learning model, and the compiled deep learning model includes the running environment of deep learning model and filtered fragment set, wherein, running environment includes the original operator of deep learning model and the mapping relationship between each original operator and each target language fragment in filtered fragment set.The compiled deep learning model can run in less storage, for example, KB (kilobyte) level running environment in runtime.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of deep learning, and more particularly, to a deep learning model compiling method, an electronic device, a storage medium and a computer program product. BACKGROUND

[0002] In the prior art, popular deep learning inference frameworks are mainly written in C++, and in the process of writing, pre-written high-performance operators need to be provided. However, no matter how the existing framework is trimmed, it still needs a MB (million byte) level running environment to completely execute all models dependent on business. This results in that deep learning inference is difficult to deploy in a running environment with limited storage space or poor C++ / C basic library support. Moreover, the traditional framework also has the problem of long initialization time, and in the scene where it cannot be permanently in memory, users will feel obvious lag.

[0003] In view of the above problems, the present application is proposed. The present application provides a deep learning model compiling method, an electronic device, a storage medium and a computer program product. SUMMARY

[0004] The present application is proposed in view of the above problems. The present application provides a deep learning model compiling method, an electronic device, a storage medium and a computer program product.

[0005] According to an aspect of the present application, a deep learning model compiling method is provided, comprising: converting a deep learning model into a computation graph; converting each original operator contained in a calculation node in the computation graph into a target language fragment in the order of the calculation node, to obtain an initial fragment set; filtering at least the initial fragment set, a plurality of identical target language fragments into a single target language fragment, to obtain a filtered fragment set; and exporting a compiled deep learning model, the compiled deep learning model comprising a running environment of the deep learning model and the filtered fragment set, wherein the running environment comprises original operators of the deep learning model and mapping relationships between each original operator and each target language fragment in the filtered fragment set.

[0006] Exemplarily, converting each original operator contained in a calculation node in the computation graph into a target language fragment in the order of the calculation node comprises: splitting any original operator into one or more program fragments using a preset algorithm; converting the one or more program fragments into one or more new program fragments described in a target language one by one; and splicing the one or more new program fragments together to obtain a target language fragment corresponding to the original operator.

[0007] Exemplarily, before the initial fragment set is obtained by converting the original operators included in the computing nodes in the computing graph into target language fragments in the order of the computing nodes in the computing graph, the method further comprises: performing memory space planning for the memory space required by each computing node in the computing graph.

[0008] Exemplarily, before the compiled deep learning model is exported, the method further comprises: counting all memory spaces required for running the initial fragment set; and allocating memory spaces for the input data, output data and intermediate results of each original operator based on the counting result to obtain memory allocation information, wherein the running environment further comprises the memory allocation information.

[0009] Exemplarily, before the compiled deep learning model is exported, the method further comprises: serializing the original operators of the deep learning model and the mapping relationship in the order of the calling of the target language fragments in the filtered fragment set to obtain the running environment.

[0010] Exemplarily, after the compiled deep learning model is exported, the method further comprises: encapsulating the running environment and the filtered fragment set into a software tool package.

[0011] Exemplarily, before the filtered fragment set is obtained by filtering a plurality of same target language fragments into a single target language fragment at least for the initial fragment set, the method further comprises: converting a concatenation operator concatenating two deep learning models into a target language fragment; and filtering a plurality of same target language fragments into a single target language fragment at least for the initial fragment set to obtain the filtered fragment set comprises: filtering a plurality of same target language fragments into a single target language fragment for the target language fragment corresponding to the concatenation operator and the initial fragment set to obtain the filtered fragment set.

[0012] According to another aspect of the present application, an electronic device is provided, comprising a processor and a memory, wherein the memory stores computer program instructions, and the computer program instructions are used to execute the deep learning model compiling method described above when executed by the processor.

[0013] According to another aspect of the present application, a storage medium is provided, and program instructions are stored on the storage medium, and the program instructions are used to execute the deep learning model compiling method described above when executed.

[0014] According to another aspect of the present application, a computer program product is provided, and the computer program product comprises a computer program, and the computer program is used to execute the deep learning model compiling method described above when executed.

[0015] According to the compiling method of the deep learning model, the electronic device, the storage medium and the computer program product, the method comprises the following steps: converting each original operator included in a calculation node in a calculation graph into a target language segment, filtering the target language segment, and obtaining an operator described by the target language segment, and obtaining a compiled deep learning model. The compiled deep learning model obtained in this way contains an operator generated on demand, which can execute business without relying on a third-party library or a basic library, and requires less storage space, so that the compiled deep learning model can run in a running environment with less storage, for example, a KB (kilobyte) level. BRIEF DESCRIPTION OF DRAWINGS

[0016] The above and other objects, features and advantages of the present application will become more apparent from the following detailed description when taken in conjunction with the accompanying drawings. The accompanying drawings provide illustration of the embodiments of the present application and constitute a part of the specification. The embodiments of the present application are explained together with the present application and do not constitute a limitation on the present application. In the drawings, the same reference numerals generally represent the same components or steps.

[0017] Figure 1 A schematic block diagram of an example electronic device for implementing the compiling method and apparatus of the deep learning model according to an embodiment of the present application is shown;

[0018] Figure 2 A schematic flow chart of a compiling method of a deep learning model according to an embodiment of the present application is shown;

[0019] Figure 3 A partial schematic diagram of a calculation graph according to an embodiment of the present application is shown;

[0020] Figure 4 A schematic block diagram of an electronic device according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0021] In recent years, important progress has been made in the research of computer vision, deep learning, machine learning, image processing, image recognition and other technologies based on artificial intelligence. Artificial intelligence (AI) is a new science and technology that studies and develops theories, methods, technologies and application systems for simulating and extending human intelligence. Artificial intelligence is a comprehensive discipline involving chips, big data, cloud computing, the Internet of Things, distributed storage, deep learning, machine learning, neural networks and many other technology categories. Computer vision, as an important branch of artificial intelligence, is specifically to let machines recognize the world. Computer vision technology usually includes face recognition, live detection, fingerprint recognition and anti-forgery verification, biometric recognition, face detection, pedestrian detection, target detection, pedestrian recognition, image processing, image recognition, image semantic understanding, image retrieval, character recognition, video processing, video content recognition, three-dimensional reconstruction, virtual reality, augmented reality, simultaneous localization and mapping (SLAM), computational photography, robot navigation and positioning and other technologies. With the research and progress of artificial intelligence technology, this technology has been applied in many fields, such as security prevention and control, city management, traffic management, building management, park management, face passage, face attendance, logistics management, warehouse management, robots, intelligent marketing, computational photography, mobile phone images, cloud services, smart home, wearable devices, unmanned vehicles, autonomous driving, intelligent medical care, face payment, face unlocking, fingerprint unlocking, face and certificate verification, smart screens, smart televisions, cameras, mobile Internet, network live broadcast, beauty, makeup, medical cosmetology, intelligent temperature measurement and other fields.

[0022] In order to make the purpose, technical solutions and advantages of the present application more obvious, the example embodiments according to the present application will be described in detail below with reference to the drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all embodiments of the present application, and it should be understood that the present application is not limited to the example embodiments described herein. Based on the embodiments of the present application described in the present application, all other embodiments obtained by those skilled in the art without creative labor shall fall within the scope of protection of the present application.

[0023] To at least partially solve the above technical problems, the embodiment of the present application provides a deep learning model compiling method, an electronic device, a storage medium and a computer program product. According to the deep learning model compiling method of the embodiment of the present application, by converting each original operator included in the calculation node in the calculation graph into a target language segment and filtering, the compiled deep learning model can execute business without relying on third-party libraries and basic libraries, the required storage space is less, and then it can run in a smaller storage, such as KB (kilobyte) level running environment. The above deep learning model compiling method can be applied to any field related to the application of deep learning model, including but not limited to face recognition, target detection, identity authentication, video live broadcast and other fields based on static images or dynamic videos.

[0024] First, referring to Figure 1 The electronic device 100 for implementing the deep learning model compiling method and device according to the embodiment of the present application is described.

[0025] As shown in Figure 1 The electronic device 100 includes one or more processors 102, one or more storage devices 104. Optionally, the electronic device 100 can also include an input device 106, an output device 108, and an image capturing device 110, which are interconnected through a bus system 112 and / or other forms of connection mechanism (not shown). It should be noted that Figure 1 The components and structures of the electronic device 100 shown are only exemplary and are not limiting, and the electronic device can also have other components and structures as needed.

[0026] The processor 102 can be implemented in at least one of a digital signal processor (DSP), a field programmable gate array (FPGA), a programmable logic array (PLA), a microprocessor, and can be a combination of one or more of a central processing unit (CPU), a graphics processor (GPU), an application-specific integrated circuit (ASIC) or other forms of processing units with data processing capability and / or instruction execution capability, and can control other components in the electronic device 100 to perform desired functions.

[0027] The storage 104 can include one or more computer program products, which can include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory, for example, can include random access memory (RAM), cache memory, and / or the like. The non-volatile memory, for example, can include read-only memory (ROM), hard disk drives, solid-state drives, and / or the like. The computer-readable storage media can store one or more computer program instructions executable by the processor 102 to implement the client functions (implemented by the processor) in the embodiments described below and / or other desired functions. The computer-readable storage media can also store various application programs and various data, such as various data used and / or generated by the application programs, and the like.

[0028] The input device 106 can be a device used by a user to input instructions, and can include one or more of a keyboard, a mouse, a microphone, a touch screen, and the like.

[0029] The output device 108 can output various information (such as images and / or sounds) to the outside (such as a user), and can include one or more of a display, a speaker, and the like. Alternatively, the input device 106 and the output device 108 can be integrated together, implemented with the same interactive device (such as a touch screen).

[0030] The image capturing device 110 can capture images and store the captured images in the storage 104 for use by other components. The image capturing device 110 can be a separate camera or a camera in a mobile terminal, and the like. It should be understood that the image capturing device 110 is merely an example, and the electronic device 100 can not include the image capturing device 110. In this case, images can be captured using other devices with image capturing capabilities and the captured images can be transmitted to the electronic device 100.

[0031] Exemplarily, an example electronic device for implementing the compiling method of the deep learning model according to the embodiments of the present application can be implemented on a device such as a personal computer, a terminal device, or a remote server. Among them, the terminal device includes but is not limited to: a tablet computer, a mobile phone, a PDA (Personal Digital Assistant), a touch screen all-in-one machine, etc.

[0032] Below, the compiling method of the deep learning model according to the embodiments of the present application will be described with reference to the accompanying drawings. Figure 2 The compiling method of the deep learning model according to the embodiments of the present application will be described. Figure 2 A schematic flowchart of the compiling method 200 of the deep learning model according to an embodiment of the present application is shown. As shown in FIG. 2, the compiling method 200 of the deep learning model according to an embodiment of the present application includes the following steps. Figure 2As shown, the compiling method 200 of the deep learning model includes steps S210, S220, S230 and S240.

[0033] In step S210, the deep learning model is converted into a computation graph.

[0034] Exemplarily, a deep learning model, for example, a convolutional neural network (CNN), can include an input layer, a hidden layer and an output layer. Each network layer can be represented by one or more operators. After converting a deep learning model into a computation graph, the computation graph can include inputs, a plurality of operators, and biases and weights corresponding to each operator, labels, etc. The operators in the computation graph can be any type of operators. For example, the operators in the computation graph can include a loss function operator, an activation function operator, a matrix multiplication operator, a pooling operator, etc. Figure 3 A partial schematic diagram of a computation graph according to an embodiment of the present application is shown. As shown, Figure 3 As shown, X can represent an input, w can represent a weight, b can represent a bias, and y can represent a label. In Figure 3 In the above, the matrix multiplication operator, the activation function operator and the loss function operator are shown.

[0035] In step S220, each original operator included in each computation node in the computation graph is converted into a target language fragment according to the order of the computation nodes in the computation graph, to obtain an initial fragment set.

[0036] Exemplarily, in the computation graph obtained in step S210, one or more original operators can be included in each computation node. For example, for a certain convolution computation node, in order to implement its convolution function, three processing modules, i.e., pre-processing, core processing and post-processing, can be needed, and each module is implemented by using a computation function (i.e., an operator).

[0037] According to the order of the computation nodes in the computation graph, each original operator included in each computation node can be converted into a target language fragment. The target language fragment can be a program fragment written in a target language. The target language described herein can be any suitable lightweight language, for example, C language, assembly language, etc. The target language can be pre-set as needed. One original operator can be converted into one or more target language fragments, and each target language fragment can be a pre-written C language program fragment or assembly language program fragment, etc. Exemplarily, in the case where the target language is C language, each target language fragment can be a kernel function. Exemplarily, in the case where the target language is C language, each target language fragment can be represented as “XXX.C”. Based on the target language fragments converted from the original operators included in one or more computation nodes, an initial fragment set can be obtained.

[0038] Step S230, filtering the plurality of identical target language fragments into a single target language fragment at least for the initial fragment set, to obtain a filtered fragment set.

[0039] Exemplarily, the target language fragments corresponding to any two original operators can be completely identical, completely different, or partially identical and partially different, so that the initial fragment set can include a plurality of identical target language fragments. In one example, only the identical target language fragments in the initial fragment set can be filtered, so that only one of the identical target language fragments remains. In another example, the initial fragment set can also be put together with other target language fragments (for example, the target language fragments converted by the serial operators described below) to form a new fragment set, and the identical target language fragments in the new fragment set can be filtered, so that only one of the identical target language fragments remains. Based on the filtered target language fragments, a filtered fragment set can be obtained.

[0040] Step S240, deriving a compiled deep learning model, the compiled deep learning model including a running environment of the deep learning model and the filtered fragment set, wherein the running environment includes original operators of the deep learning model and mapping relationships between the original operators and the target language fragments in the filtered fragment set.

[0041] Exemplarily, the filtered fragment set obtained in step S230 and the running environment of the deep learning model are derived, for example, stored in a folder of a computer hard disk. The running environment can include original operators of the deep learning model, for example, matrix multiplication operators, activation function operators, and loss function operators. Exemplarily, the original operators can be described and stored in an abstract manner, for example, a convolution operator can be described by a character such as “convolution”, rather than a real executable program language. The running environment can also include mapping relationships between the original operators and the target language fragments in the filtered fragment set, for example, the matrix multiplication operator corresponds to the target language fragments “1111.C” and “2222.C”, the activation function operator corresponds to the target language fragment “3333.C”, and the loss function operator corresponds to the target language fragments “4444.C” and “5555.C”.

[0042] According to the compiling method of the deep learning model, the electronic device, the storage medium and the computer program product, the method comprises the following steps: converting each original operator included in a calculation node in a calculation graph into a target language segment and filtering, to obtain an operator described by a target language segment, and then obtaining a compiled deep learning model. The compiled deep learning model obtained in this way contains an operator generated on demand, which can execute a business without relying on a third-party library or a basic library, and requires less storage space, so that the compiled deep learning model can run in a running environment with less storage, for example, a KB (kilobyte) level. In addition, the runtime information (information contained in the running environment) bound in the model during compilation helps to reduce the calculation amount of model runtime initialization, making it possible to guarantee the minimum runtime memory occupation. Therefore, the deep learning model obtained by compiling can be deployed in a bare system, which is very suitable for deep learning inference in embedded scenarios, storage limited scenarios and scenarios that cannot be permanently in memory.

[0043] The compiling method of the deep learning model according to the embodiments of the present application can be deployed at a terminal, for example, at a personal terminal or a server end.

[0044] Alternatively, the compiling method of the deep learning model according to the embodiments of the present application can also be distributedly deployed at a server end (or a cloud end) and a client end. For example, the parameters of the deep learning model can be obtained at the client end, the client end transmits the parameters of the deep learning model to the server end (or the cloud end), and the server end (or the cloud end) compiles the deep learning model. Illustratively, the parameters of the deep learning model can include one or more of the weight of each network layer, the bias, the size information of the input image of the deep learning model, etc.

[0045] Illustratively, converting each original operator included in a calculation node in a calculation graph into a target language segment in the order of the calculation nodes can comprise: using a preset algorithm to split any original operator into one or more program segments; converting one or more program segments into one or more new program segments described by a target language one by one; and splicing one or more new program segments together to obtain a target language segment corresponding to the original operator.

[0046] In one embodiment, a preset algorithm can be used to split an original operator into one or more program segments. The preset algorithm can be any suitable algorithm. In one example, the preset algorithm can be an algorithm written in C++ language. For example, a certain original operator can be split into four for loops by using a pre-written C++ program. Then the four for loops are converted into a plurality of new program segments described in a target language, for example, each new program segment can be a string in C language. Finally, the plurality of new program segments, i.e. the plurality of strings, are spliced together, and a target language segment corresponding to the original operator can be obtained. As described above, an original operator, for example, convolution, can be described in an abstract way, for example, described by two characters “convolution”. The string converted from the original operator is a real convolution calculation function written in a target language such as C language.

[0047] By using the preset algorithm, each original operator contained in each computing node can be quickly and simply converted into a target language segment. This conversion method has relatively high efficiency and accuracy.

[0048] Exemplarily, before converting each original operator contained in each computing node into a target language segment in the order of the computing nodes in the computation graph to obtain an initial segment set, the method can further include: planning memory space required by each computing node in the computation graph.

[0049] In one embodiment, before obtaining the initial segment set, the method can further include planning memory space required by each computing node in the computation graph. Exemplarily, planning the memory space can include configuring different operators to alternately share the same memory space. For example, a certain computing node includes three operators in total. The first operator uses a memory of 5G when running. When the third operator runs, the memory space used by the first operator has been released, so the 5G memory space can be used to store data related to the third operator. The data related to any operator can include one or more of input data, output data, and intermediate results.

[0050] The memory planning is completed during the offline compilation period of the model, which helps to reduce the computation overhead and the amount of runtime code during the runtime of the model, so that the compiled model can better adapt to the environment with limited storage space.

[0051] Exemplarily, before exporting the compiled deep learning model, the method can further include: counting all memory spaces required by the initial segment set to run; allocating memory spaces for input data, output data, and intermediate results of each original operator based on the counting result to obtain memory allocation information, wherein the running environment further includes the memory allocation information.

[0052] In one embodiment, the initial fragment set as described above can include target language fragments "1111.C", "2222.C", "3333.C", "4444.C" and "5555.C". The total memory space required for running the initial fragment set is obtained by statistics as 10G, and the following memory allocation information can be generated: 5G memory space is allocated to the matrix multiplication operator, wherein the input data occupies 1G, the output data occupies 2G, and the intermediate result occupies 2G; 2G memory space is allocated to the activation function operator, wherein the input data occupies 0.5G, the output data occupies 1G, and the intermediate result occupies 0.5G; 3G memory space is allocated to the loss function operator, wherein the input data occupies 1G, the output data occupies 0.5G, and the intermediate result occupies 1.5G.

[0053] In the above technical solution, the total memory space required for the initial fragment set based on statistics is allocated to the input data, output data and intermediate result of each original operator, so that the allocation of memory space can be more reasonable, and excessive memory space consumption can be reduced. This solution helps to reduce the calculation overhead and the amount of runtime code of the model, so that the compiled model can better adapt to the environment with limited storage space.

[0054] Illustratively, the total memory space required for running the initial fragment set can include: obtaining the size information of the image processed by the deep learning model; and based on the size information and all target language fragments in the initial fragment set, the total memory space is counted.

[0055] The deep learning model can be applied to the processing of any type of input data. For example, the input data can be an image or audio, etc. In one embodiment, the image processed by the deep learning model can be a static image or any video frame in a dynamic video. The image can be an original image collected by an image acquisition device (such as an image sensor in a camera), or an image obtained after pre-processing (such as digitization, normalization, smoothing, etc.) of the original image. The size information of the image can include the width W of the image, the height H of the image, and the number of channels C of the image, and the size information of the image can be represented as [H, W, C].

[0056] The size information of the image processed by the deep learning model is related to the total memory space required for running the initial fragment set. For the same deep learning model, the memory space required for the image of the same size is also the same. In addition, any target language fragment can involve one or more of input data, output data and intermediate result, and the size of each data and the required memory space are fixed. Therefore, the total memory space required can be counted in combination with the size information of the image and the requirements of all target language fragments in the initial fragment set obtained in the above step S220.

[0057] The image processed by the deep learning model can be sent to the electronic device 100 by an external device (for example, a cloud server) for processing by the processor 102 of the electronic device 100, can be captured by an image capture device 110 (for example, a camera) included in the electronic device 100 and transmitted to the processor 102 for processing, or can be stored by a storage device 104 included in the electronic device 100 and transmitted to the processor 102 for processing.

[0058] In the above technical solution, the size of the memory space can be more accurately determined based on the size information of the image processed by the deep learning model and all target language segments in the initial segment set.

[0059] For example, before the compiled deep learning model is exported, the method can further include: serializing the original operators of the deep learning model and the mapping relationship in the calling order of each target language segment in the filtered segment set, to obtain a running environment.

[0060] In one embodiment, before the running environment of the deep learning model and the filtered segment set are exported through the compilation process of the deep learning model, a serialization tool (FlatBuffers, FBS) packaging tool can be called to serialize the original operators of the deep learning model and the mapping relationship, to obtain a running environment. The serialized information can be stored in a file in a predetermined memory in a specific format, so it can be more conveniently called. The predetermined memory can be a fixed hard disk or a mobile hard disk, etc.

[0061] For example, after the compiled deep learning model is exported, the method can further include: packaging the running environment and the filtered segment set into a software tool package.

[0062] In an embodiment, the exported filtered segment set and the running environment can be compiled together using a predetermined compilation tool to obtain a software development kit (Software Development Kit, SDK) of the deployed deep learning model. The predetermined compilation tool can be provided by a deployment platform of the SDK. The SDK can be stored in any suitable storage space, such as a fixed hard disk or a mobile hard disk of a computer, and the mobile hard disk can be, for example, a U disk. Since the operators are generated according to the user's needs and filtered, the size of the SDK is very small, which is very suitable for storage space-limited or embedded scenarios.

[0063] For example, after the deep learning model is deployed, the filtered segment set can be stored in a folder and exist as a running library. When the running environment of the deep learning model is called, each target language segment in the running library can be automatically called when needed.

[0064] Exemplarily, before the filtering of the multiple identical target language segments into the single target language segment to obtain the filtered segment set, the method can further comprise: converting the concatenation operator of the two deep learning models into a target language segment; and the filtering of the multiple identical target language segments into the single target language segment to obtain the filtered segment set can comprise: filtering the multiple identical target language segments into the single target language segment to obtain the filtered segment set for the target language segment corresponding to the concatenation operator and the initial segment set.

[0065] In one embodiment, the number of deep learning models is multiple, and the concatenation operator can be further included between any two deep learning models. For the case of multiple deep models, each deep model can be processed through the above steps S210-S240. In addition, the concatenation operator can also be added to steps S220, S230 and S240 for processing.

[0066] For example, in the face recognition scenario, face recognition is divided into two steps: 1, detecting whether there is a face; 2, identifying the face identity. The two steps can be implemented by using a face detection model and a face comparison model respectively. After the face detection model detects that there is a face, the face detection model can identify the face detection frame where the face is located. The face detection model can output the information of the face detection frame to the subsequent face comparison model. However, in order to facilitate comparison, the face detection frame usually needs to be enlarged before being input to the face comparison model. Therefore, there is also a magnification operator between the face detection model and the face comparison model. The intermediate magnification operator (resize) belongs to the concatenation operator. The concatenation operator can be converted into one or more target language segments by using the step S220. Subsequently, the target language segment converted by the concatenation operator and the initial segment set converted by the deep learning model are placed together for filtering to obtain the filtered segment set. The filtering of the multiple identical target language segments into the single target language segment to obtain the filtered segment set has been described in detail in the foregoing, and for the sake of brevity, it will not be described here. In the case where there is repetition between the target language segment corresponding to the concatenation operator and the target language segment corresponding to the deep learning model, they can also be integrated together for filtering, thereby further saving the memory space and adapting to the scene where the memory space is limited.

[0067] The specific functions of the serial operators need to be determined according to the business actually executed by the deep learning model and the application environment of the business. The same multiple deep learning models are applied to different businesses (for example, some are used for face recognition, and some are used for certificate recognition), and the required serial operators can be different. The same multiple deep learning models are applied to the same business, but the application environment is different (for example, both are used for face recognition, but some are used for banks and some are used for shopping malls), and the required serial operators can also be different. In summary, the serial operator can be an operator of any form, capable of realizing any function, used to serialize two deep learning models, and the specific form and function of the serial operator are not limited in the present application.

[0068] Exemplarily, before the original operators included in the computing nodes are converted into target language fragments in the order of the computing nodes in the computing graph to obtain an initial fragment set, the method can further include: optimizing the computing graph.

[0069] In an embodiment, the foregoing step S210 of converting the deep learning model into a computing graph can include multiple computing nodes and each original operator included in each computing node in the computing graph. The optimization of the computing graph can be implemented in any existing or future possible optimization manner. For example, the computing graph can be optimized by merging the same original operators in each original operator into a single operator. For example, the matrix multiplication operator and the matrix addition operator can be merged into a matrix multiplication and addition operator. Optimizing the computing graph is beneficial to program acceleration and can effectively improve the computing performance of the deep learning model.

[0070] Exemplarily, the above deep learning model can be compiled offline using a multi-level intermediate representation (MLIR) tool. The MLIR tool has a wide range of applications and helps to expand the portability of the deep learning model.

[0071] Those skilled in the art can realize that the units and algorithm steps of the examples described in combination with the embodiments disclosed herein can be realized in electronic hardware or a combination of computer software and electronic hardware. Whether the functions are realized in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0072] Figure 4 A schematic block diagram of an electronic device 400 according to an embodiment of the present application is shown. The electronic device 400 includes a memory 410 and a processor 420.

[0073] The memory 410 stores computer program instructions for implementing respective steps in the method for compiling a deep learning model according to embodiments of the present application.

[0074] The processor 420 is configured to execute the computer program instructions stored in the memory 410 to perform respective steps in the method for compiling a deep learning model according to embodiments of the present application.

[0075] In one embodiment, the computer program instructions, when executed by the processor 420, cause the processor 420 to perform the following steps: converting the deep learning model into a computation graph; converting each original operator included in a computation node in the computation graph into a target language snippet in the order of the computation nodes, to obtain an initial snippet set; filtering a plurality of identical target language snippets into a single target language snippet at least for the initial snippet set, to obtain a filtered snippet set; and exporting a compiled deep learning model, the compiled deep learning model comprising a running environment of the deep learning model and the filtered snippet set, wherein the running environment comprises the original operators of the deep learning model and mapping relationships between the original operators and the target language snippets in the filtered snippet set.

[0076] By way of example, the electronic device 400 can further include an image acquisition device 430. The image acquisition device 430 is configured to acquire an image processed by the deep learning model. The image acquisition device 430 is optional, and the electronic device 400 can also not include the image acquisition device 430. In this case, the processor 420 can acquire the image processed by the deep learning model in other manners, for example, from an external device or from the memory 410.

[0077] In addition, according to embodiments of the present application, a storage medium is also provided, on which program instructions are stored, and the program instructions, when executed by a computer or a processor, cause the computer or the processor to perform respective steps in the method for compiling a deep learning model according to embodiments of the present application. The storage medium can include, for example, a memory card of a smart phone, a memory component of a tablet computer, a hard disk of a personal computer, a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a portable compact disc read-only memory (CD-ROM), a USB memory, or any combination of the above storage media.

[0078] In one embodiment, the program instructions, when executed by the computer or the processor, can cause the computer or the processor to perform the method for compiling a deep learning model according to embodiments of the present application.

[0079] In one embodiment, the program instructions are used for performing the following steps when running: converting the deep learning model into a computation graph; converting each original operator included in a computation node in the computation graph into a target language fragment in the order of the computation nodes, to obtain an initial fragment set; filtering a plurality of same target language fragments into a single target language fragment at least for the initial fragment set, to obtain a filtered fragment set; and exporting a compiled deep learning model, the compiled deep learning model comprising a running environment of the deep learning model and the filtered fragment set, wherein the running environment comprises the original operators of the deep learning model and a mapping relationship between each original operator and each target language fragment in the filtered fragment set.

[0080] In addition, according to an embodiment of the present application, a computer program product is also provided, the computer program product comprising a computer program, the computer program being used for performing the above-mentioned deep learning model compiling method 200 when running.

[0081] The modules in the electronic device according to the embodiments of the present application can be realized by a processor of the electronic device according to the embodiments of the present application running computer program instructions stored in the memory, or can be realized when computer instructions stored in the computer readable storage medium of the computer program product according to the embodiments of the present application are run by a computer.

[0082] Although the example embodiments have been described herein with reference to the accompanying drawings, it is to be understood that the above-described example embodiments are merely illustrative and not intended to limit the scope of the present application. Various changes and modifications can be made thereto by those of ordinary skill in the art without departing from the scope and spirit of the present application. All such changes and modifications are intended to be included within the scope of the present application as defined by the appended claims.

[0083] Those of ordinary skill in the art can realize that the units and algorithm steps of each example described in connection with the embodiments disclosed herein can be realized in electronic hardware, or a combination of computer software and electronic hardware. Whether the functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Those of ordinary skill in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0084] In several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative, for example, the division of units is only a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another device, or some features can be ignored or not executed.

[0085] In the description provided herein, numerous specific details are set forth. However, it is understood that embodiments of the application can be practiced without these specific details. In some instances, well-known methods, structures and techniques have not been shown in detail in order not to obscure an understanding of this description.

[0086] Similarly, it is to be understood that the various features of the application can be used alone or in any combination depending on the embodiment of the application being used. It is therefore contemplated to this effect that the application can be practiced with the individual features of the application in any combination, including combinations that do not specifically appear in the examples described herein. Similarly, it should be understood that, in the description of the example embodiments of the application, various features of the application are sometimes grouped together in a single embodiment, figure, or description of related features. This should not be interpreted as reflecting an intention that the application requires more features than are explicitly recited in each claim. Rather, inventive aspects can lie in less than all features of a single disclosed embodiment. The following claims are hereby expressly incorporated into this detailed description, with each claim acting as a separate embodiment of the application.

[0087] Those skilled in the art will appreciate that all features described herein (including all accompanying claims, abstract and drawings), and steps or elements of any method or device so described, can be combined in any combination, except where features are mutually exclusive (i.e. where a particular feature is stated to be exclusive of another). Each and every combination and permutation of features is specifically contemplated and described herein.

[0088] Further, those skilled in the art will appreciate that the features of the different embodiments can be combined in any combination, as the combinations of features are within the scope of the application and form different embodiments of the application. For example, in the claims, any of the claimed embodiments can be used in any combination.

[0089] The various component embodiments of the present application can be implemented in hardware, or as software modules running in one or more processors, or in combinations thereof. As will be appreciated by one skilled in the art, a microprocessor or digital signal processor (DSP) can be used in practice to implement some or all of the functionality of some of the modules in the compiling apparatus of the deep learning model according to the embodiments of the present application. The present application can also be implemented as a program for executing part or all of the methods described herein on a computer (for example, a computer program and a computer program product). The program implementing the present application can be stored on a computer readable medium or can have the form of one or more signals. Such signals can be downloaded from an Internet website, or provided on a carrier signal, or in any other form.

[0090] It should be noted that the above-mentioned embodiments illustrate rather than limit the application, and that one skilled in the art will be able to design many alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses shall not be construed as limiting the claim. The word 'comprising' does not exclude the presence of elements or steps other than those listed in a claim. The word 'a' or 'an' preceding an element does not exclude the presence of a plurality of such elements. The application can be implemented by means of both hardware and software, and any combination thereof. In a unit claim, several devices can be listed with a conjunction like 'or', but it is to be understood that each of these devices can be implemented by its own hardware item. Use of the word 'about' provides an allowance of a certain amount of variation, consistent with experimental error. The word 'comprising' does not exclude the presence of other elements or steps than those listed in a claim. The word 'a' or 'an' preceding an element does not exclude the presence of a plurality of such elements. The application can be implemented by means of both hardware and software, and any combination thereof. In a unit claim, several devices can be listed with a conjunction like 'or', but it is to be understood that each of these devices can be implemented by its own hardware item. Use of the word 'first','second', and 'third', etc. does not indicate any order. These words can be interpreted as names.

[0091] The above merely illustrates the specific embodiments of the present application or the description of the specific embodiments, and the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of changes or replacements within the technical range disclosed by the present application, which shall be covered within the protection scope of the present application. The protection scope of the present application shall be subject to the protection scope of the claims.

Claims

1. A method for compiling a deep learning model, comprising: Convert deep learning models into computational graphs; The original operators contained in the computation nodes are converted into target language fragments according to the order of the computation nodes in the computation graph to obtain an initial fragment set; At least for the initial set of fragments, multiple identical target language fragments are filtered into a single target language fragment to obtain a filtered set of fragments; Export the compiled deep learning model, which includes the running environment of the deep learning model and the filtered fragment set. The running environment includes the primitive operators of the deep learning model and the mapping relationship between each primitive operator and each target language fragment in the filtered fragment set. The method further includes the following steps before exporting the compiled deep learning model: Calculate the total memory space required to run the initial set of fragments; Based on statistical results, memory space is allocated for the input data, output data, and intermediate results of each original operator to obtain memory allocation information. The runtime environment also includes the memory allocation information. And / or, The method further includes, prior to filtering multiple identical target language segments into a single target language segment for at least the initial segment set to obtain a filtered segment set: Convert the concatenation operator of two deep learning models into target language segments; The step of filtering multiple identical target language segments into a single target language segment, at least for the initial segment set, to obtain a filtered segment set includes: For the target language segment corresponding to the concatenation operator and the initial segment set, multiple identical target language segments are filtered into a single target language segment to obtain the filtered segment set.

2. The method as described in claim 1, wherein, The step of converting the primitive operators contained in the computation nodes into target language fragments according to the order of the computation nodes in the computation graph includes: A preset algorithm is used to split any original operator into one or more program segments; The one or more program fragments are converted into one or more new program fragments described in the target language in a one-to-one correspondence. By concatenating the one or more new program fragments together, a target language fragment corresponding to the original operator is obtained.

3. The method as described in claim 1, wherein, Before converting the primitive operators contained in the computation nodes into target language fragments according to the order of the computation nodes in the computation graph to obtain an initial fragment set, the method further includes: Memory space planning is performed for the memory space required by each computing node in the computation graph.

4. The method according to any one of claims 1-3, wherein, Before exporting the compiled deep learning model, the method further includes: The original operators of the deep learning model and the mapping relationship are serialized according to the calling order of each target language segment in the filtered segment set to obtain the running environment.

5. The method according to any one of claims 1-3, wherein, After exporting the compiled deep learning model, the method further includes: The runtime environment and the filtered collection of fragments are encapsulated into a software toolkit.

6. An electronic device comprising a processor and a memory, characterized in that, The memory stores computer program instructions, which, when executed by the processor, are used to perform the compilation method of the deep learning model as described in any one of claims 1 to 5.

7. A storage medium on which program instructions are stored, characterized in that, The program instructions are used at runtime to execute the compilation method of the deep learning model as described in any one of claims 1 to 5.

8. A computer program product, the computer program product comprising a computer program, characterized in that, The computer program, when running, is used to execute the compilation method of the deep learning model as described in any one of claims 1 to 5.

Images