Neural network model deployment method and device, electronic device and storage medium

Abstract

The present disclosure relates to a method and apparatus for deploying a neural network model, an electronic device, and a storage medium. The method comprises: compiling a neural network model to be deployed to obtain an offline model comprising multiple offline sub-models, each of which can be deployed to a corresponding hardware backend; and then distributing the multiple offline sub-models to corresponding hardware devices, each of which can correspond to at least one hardware backend. Embodiments of the present disclosure can fully leverage the advantages of various hardware backends for hardware devices, improving the efficiency and flexibility of neural network model deployment.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a neural network model deployment method and device, an electronic device, and a storage medium. Background Art

[0002] With the continuous development of artificial intelligence (AI) technology, the application of neural networks, such as deep learning, is becoming increasingly widespread. Deploying neural network models is crucial for enabling learning algorithms to be effective in production. The actual deployment plan for the model affects how the model is used by programs and plays a crucial role in the overall neural network learning application scenario. However, with the rapid development of edge inference hardware in recent years, the basic operator sets supported by each hardware vary, posing various challenges to the deployment of neural network models. Summary of the Invention

[0003] This disclosure proposes a technical solution for deploying a neural network model.

[0004] According to one aspect of the present disclosure, a neural network model deployment method is provided, which is applied to an electronic device, and the method includes: obtaining a neural network model to be deployed; compiling the neural network model to obtain a compiled offline model, wherein the offline model includes multiple offline sub-models, and each of the offline sub-models is deployed to a corresponding hardware backend, and various hardware backends respectively correspond to different tool chains for deploying the neural network model to hardware devices, and each hardware device corresponds to at least one hardware backend; and sending the multiple offline sub-models to the corresponding hardware devices.

[0005] In one possible implementation, compiling the neural network model to obtain a compiled offline model includes: performing structural conversion on the neural network model to obtain an internal model structure adapted to model transformation; splitting the internal model structure according to each of the hardware backends to be deployed to obtain multiple sub-models and a series relationship between the multiple sub-models, wherein each sub-model corresponds to a target hardware backend; for any sub-model, performing a model transformation operation related to the target hardware backend on the sub-model to obtain an offline sub-model deployed to the target hardware backend; and determining the offline model based on the multiple offline sub-models and the series relationship.

[0006] In a possible implementation, the target hardware backend is a hardware backend with the highest preset priority among the hardware backends deployable by the sub-model.

[0007] In a possible implementation, before splitting the internal model structure according to the hardware backends to be deployed, the method further includes: performing a model transformation operation and a model optimization operation related to the hardware backend on the internal model structure.

[0008] In a possible implementation, performing a model transformation operation related to a hardware backend on the internal model structure includes: performing a model transformation operation related to a hardware backend with a highest preset priority on the internal model structure.

[0009] In a possible implementation, before performing a model transformation operation and a model optimization operation related to a hardware backend on the internal model structure, the method further includes performing a model optimization operation on the internal model structure that is unrelated to a hardware backend.

[0010] In one possible implementation, performing a model transformation operation on the sub-model related to the target hardware backend to obtain an offline sub-model deployed to the target hardware backend includes: performing a model transformation operation on the sub-model related to the target hardware backend to obtain a sub-model in a first state; performing format conversion on the sub-model in the first state to obtain a sub-model in a second state, wherein the sub-model in the second state is adapted to the input format of the target hardware backend; and deploying the sub-model in the second state to the target hardware backend to obtain the offline sub-model.

[0011] In one possible implementation, sending the multiple offline sub-models to corresponding hardware devices includes: reading the multiple offline sub-models of the offline model and the series relationship between the multiple offline sub-models through a model interpreter; and sending each of the offline sub-models to the corresponding hardware device, wherein the model interpreter connects the multiple offline sub-models in series when the hardware device is running according to the series relationship between the multiple offline sub-models.

[0012] In one possible implementation, the hardware backend includes: a hardware backend using a hardware vendor's reasoning library, a hardware backend using a hardware vendor's operator library, or a hardware backend using an operator not provided by the hardware vendor.

[0013] According to one aspect of the present disclosure, a neural network model deployment device is provided, which is applied to an electronic device, including: an acquisition module, used to acquire a neural network model to be deployed; a compilation module, used to compile the neural network model to obtain a compiled offline model, wherein the offline model includes multiple offline sub-models, each of the offline sub-models is deployed to a corresponding hardware backend, and various hardware backends respectively correspond to different tool chains for deploying the neural network model to hardware devices, and each hardware device corresponds to at least one hardware backend; and an operation module, used to send the multiple offline sub-models to the corresponding hardware devices.

[0014] In one possible implementation, the compilation module includes: a structure conversion module, which is used to perform structural conversion on the neural network model to obtain an internal model structure that is suitable for model transformation; a splitting module, which is used to split the internal model structure according to each of the hardware backends to be deployed, to obtain multiple sub-models and a series relationship between the multiple sub-models, wherein each sub-model corresponds to a target hardware backend; an offline sub-model acquisition module, which is used to perform a model transformation operation related to the target hardware backend on any sub-model to obtain an offline sub-model deployed to the target hardware backend; and an offline model determination module, which is used to determine the offline model based on the multiple offline sub-models and the series relationship.

[0015] In a possible implementation, the target hardware backend is a hardware backend with the highest preset priority among the hardware backends deployable by the sub-model.

[0016] In a possible implementation, the compilation module also includes a first module, which is used to: perform model transformation operations and model optimization operations related to the hardware backend on the internal model structure before splitting the internal model structure according to the various hardware backends to be deployed.

[0017] In a possible implementation, performing a model transformation operation related to a hardware backend on the internal model structure includes: performing a model transformation operation related to a hardware backend with a highest preset priority on the internal model structure.

[0018] In a possible implementation, the compilation module further includes a second module configured to perform a model optimization operation unrelated to the hardware backend on the internal model structure before performing a model transformation operation and a model optimization operation related to the hardware backend on the internal model structure.

[0019] In one possible implementation, the offline sub-model acquisition model is used to: perform a model transformation operation related to the target hardware backend on the sub-model to obtain a sub-model in a first state; perform format conversion on the sub-model in the first state to obtain a sub-model in a second state, wherein the sub-model in the second state is adapted to the input format of the target hardware backend; and deploy the sub-model in the second state to the target hardware backend to obtain the offline sub-model.

[0020] In one possible implementation, the running module is used to: read the multiple offline sub-models of the offline model and the series relationship between the multiple offline sub-models through a model interpreter; and send each of the offline sub-models to the corresponding hardware device, wherein the model interpreter connects the multiple offline sub-models in series according to the series relationship between the multiple offline sub-models when the hardware device is running.

[0021] In one possible implementation, the hardware backend includes: a hardware backend using a hardware vendor's reasoning library, a hardware backend using a hardware vendor's operator library, or a hardware backend using an operator not provided by the hardware vendor.

[0022] According to one aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to call the instructions stored in the memory to execute the above method.

[0023] According to one aspect of the present disclosure, a computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the above method is implemented.

[0024] In the disclosed embodiments, a neural network model to be deployed can be compiled to generate an offline model comprising multiple offline sub-models, each of which can be deployed to a corresponding hardware backend. The multiple offline sub-models can then be distributed to corresponding hardware devices, each of which can correspond to at least one hardware backend. This can fully leverage the advantages of various hardware backends for specific hardware devices, improving the efficiency and flexibility of neural network model deployment.

[0025] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, rather than limiting the present disclosure. Other features and aspects of the present disclosure will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The accompanying drawings herein are incorporated into and constitute a part of the specification. These drawings illustrate embodiments consistent with the present disclosure and, together with the specification, are used to explain the technical solutions of the present disclosure.

[0027] Figure 1 A flowchart of a neural network model deployment method according to an embodiment of the present disclosure is shown.

[0028] Figure 2 A schematic diagram showing a neural network model deployment method according to an embodiment of the present disclosure.

[0029] Figure 3A flowchart of the compilation phase in the neural network model deployment method according to an embodiment of the present disclosure is shown.

[0030] Figure 4 A block diagram of a neural network model deployment apparatus according to an embodiment of the present disclosure is shown.

[0031] Figure 5 A block diagram of an electronic device according to an embodiment of the present disclosure is shown.

[0032] Figure 6 A block diagram of another electronic device according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0033] Various exemplary embodiments, features, and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise indicated.

[0034] The word “exemplary” is used exclusively herein to mean “serving as an example, example, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.

[0035] The term "and / or" herein simply describes an association relationship between associated objects, indicating that three relationships can exist. For example, "A and / or B" can represent the existence of three situations: A alone, A and B simultaneously, and B alone. Furthermore, the term "at least one" herein refers to any combination of at least two of any one or more of a plurality of items. For example, "at least one of A, B, and C" can represent any one or more elements selected from the set consisting of A, B, and C.

[0036] In addition, numerous specific details are provided in the following detailed description to better illustrate the present disclosure. Those skilled in the art will appreciate that the present disclosure can be practiced without certain specific details. In some instances, methods, means, components, and circuits well known to those skilled in the art are not described in detail in order to highlight the main points of the present disclosure.

[0037] In related technologies, in order to support the application of neural networks, various hardware manufacturers have produced various edge inference hardware that can run neural network models, such as central processing units (CPUs), graphics processing units (GPUs), tensor processing units (TPUs), machine learning processors (MLUs) or ARM processors (Advanced RISC Machines, ARM) and other related inference hardware.

[0038] However, different vendors' inference hardware specifications vary (for example, each inference hardware supports a different set of basic operators), which poses various challenges to the deployment of neural network models:

[0039] 1. During the development of upper-layer services, due to the significant differences between various inference hardware and the different interface designs provided by various hardware manufacturers, it is impossible to efficiently and uniformly connect multiple inference hardware within a single framework, which brings inconvenience to the development of upper-layer services.

[0040] 2. Because different inference hardware toolchains accept different definitions of neural network models, manual model modification may be required according to the hardware manufacturer's specifications (such as the inference library). Without manual modification, a trained neural network model cannot be deployed to multiple inference hardware, nor can a single training run be deployed on multiple hardware.

[0041] 3. The deployment software provided by inference hardware vendors may not support the deployment of independently designed neural network models. For example, a neural network model may contain fragments that are difficult to deploy on specific inference hardware (for example, some layers or operators included in a deep neural network model). In this case, the neural network model may fail to deploy, or during the deployment process, heterogeneous computing between the host and inference device may be required to complete the deployment of the entire model, which is a cumbersome process.

[0042] In light of this, this disclosure proposes a neural network model deployment method that compiles the neural network model to be deployed to generate an offline model consisting of multiple offline sub-models, each of which can be deployed to a corresponding hardware backend. The multiple offline sub-models are then distributed to corresponding hardware devices, each of which corresponds to at least one hardware backend. This method thus leverages the advantages of various hardware backends for specific hardware devices, improving the efficiency and flexibility of neural network model deployment.

[0043] Figure 1A flowchart of a neural network model deployment method according to an embodiment of the present disclosure is shown. Figure 1 As shown, the method is applied to an electronic device, comprising:

[0044] In step S1, a neural network model to be deployed is obtained;

[0045] In step S2, the neural network model is compiled to obtain a compiled offline model, wherein the offline model includes multiple offline sub-models, each of which is deployed to a corresponding hardware backend, each of which corresponds to a different tool chain for deploying the neural network model to a hardware device, and each hardware device corresponds to at least one hardware backend;

[0046] In step S3, the multiple offline sub-models are sent to corresponding hardware devices.

[0047] In one possible implementation, the neural network model deployment method provided in the embodiments of the present disclosure can be executed by an electronic device such as a terminal device or a server. The terminal device can be a user equipment (UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, etc. The method can be implemented by a processor calling computer-readable instructions stored in a memory. Alternatively, the method can be executed by a server.

[0048] In step S1, the neural network model to be deployed is a trained neural network model, that is, a neural network model that meets the target performance requirements is obtained by training the neural network model in the initial state through preset training samples and loss functions.

[0049] In a possible implementation, the neural network model includes at least one of a feature extraction neural network model, a classification neural network model, and a target detection neural network model.

[0050] The neural network model may be a neural network model that implements any function, such as a feature extraction neural network model that extracts input data feature information, a target detection neural network model that detects objects, a neural network model that segments objects, a neural network model that classifies objects, or a neural network model for natural language processing. The above description is merely illustrative and is not intended to be limiting in this disclosure.

[0051] After obtaining the neural network model to be deployed, the neural network model can be deployed to the corresponding hardware device. In actual applications, the neural network model can be deployed to one or more hardware devices based on the performance of the hardware devices. This disclosure does not place any specific restrictions on the number of hardware devices.

[0052] If a neural network model needs to be deployed on a hardware device, the model can be compiled to generate an offline model, which can then be run on the hardware device to deploy the neural network model on the hardware device. The hardware device can be any inference hardware manufactured by any hardware manufacturer, and this disclosure does not limit the type of hardware device.

[0053] For example, assuming that the neural network model is a neural network model A for object recognition, it may include two convolutional layers A1 and A2, a pooling layer A3, and a fully connected layer A4. According to steps S2-S3, the neural network model A can be deployed to a hardware device H (e.g., a GPU).

[0054] In step S2, the neural network model A can be input into the compiler, the neural network model A can be compiled and processed, and the input neural network model A can be deployed to multiple hardware backends to obtain offline sub-models corresponding to the multiple hardware backends.

[0055] The multiple hardware backends may correspond to multiple tool chains for deploying the neural network model A to the hardware device H. Assuming that the hardware device H is a GPU that supports the Compute Unified Device Architecture (CUDA), the tool chain for this hardware device may include the TensorRT tool chain, the CUDNN (CUDA Deep Neural Network Library, CUDNN) tool chain, etc. This disclosure does not limit the type of tool chain.

[0056] For example, the convolution layers A1 and A2 included in the neural network model A can be deployed to the hardware backend 1 to obtain the offline sub-model C1; the pooling layer A3 included in the neural network model A can be deployed to the hardware backend 2 to obtain the offline sub-model C2; the fully connected layer A4 included in the neural network model A can be deployed to the hardware backend 3 to obtain the offline sub-model C3.

[0057] Then, according to the data input and output dependency (i.e., the series relationship) between the convolutional layers A1 and A2, the pooling layer A3, and the fully connected layer A4, the offline sub-models C1 to C3 deployed to the hardware backends 1 to 3 are processed in series to obtain the compiled offline model.

[0058] In step S3, the offline sub-models C1 to C3 can be sent to the hardware device H, and the offline model composed of the offline sub-models C1 to C3 can be run in the hardware device H, that is, the compiled offline model can be run, and the offline model including multiple offline sub-models C1 to C3 can be deployed to the hardware device H.

[0059] Through this method, when deploying a neural network model to a hardware device, the neural network model is actually deployed to all backends for the hardware, which can give full play to the advantages of various hardware backends for the hardware device and take advantage of the advantages of each hardware backend.

[0060] If a neural network model needs to be deployed on multiple hardware devices, the neural network model can be compiled to generate an offline model. This offline model can then be run on multiple hardware devices to deploy the neural network model on multiple hardware devices. The hardware devices can be inference hardware produced by any hardware manufacturer, and this disclosure does not limit the type of hardware devices.

[0061] For example, assuming the neural network model is a neural network model A for object recognition, it may include two convolutional layers A1 and A2, a pooling layer A3, and a fully connected layer A4. According to steps S2 and S3, the neural network model A can be deployed to hardware devices H1 (e.g., a GPU) and H2 (e.g., a CPU).

[0062] In step S2, the neural network model A can be input into a compiler, compiled, and deployed to multiple hardware backends to obtain offline sub-models corresponding to the multiple hardware backends. The multiple hardware backends can correspond to multiple tool chains for deploying the neural network model A to hardware devices A and B, respectively.

[0063] For example, the convolution layers A1 and A2 included in the neural network model A can be deployed to the hardware backend B1 of the corresponding hardware device H1 to obtain the offline sub-model C1; the pooling layer A3 included in the neural network model A can be deployed to the hardware backend B2 of the corresponding hardware device H1 to obtain the offline sub-model C2; the fully connected layer A4 included in the neural network model A can be deployed to the hardware backend B3 of the corresponding hardware device H2 to obtain the offline sub-model C3.

[0064] Then, according to the data input and output dependency (i.e., the series relationship) between the convolutional layers A1 and A2, the pooling layer A3, and the fully connected layer A4, the offline sub-models C1 to C3 deployed to the hardware backends B1 to B3 are processed in series to obtain the compiled offline model.

[0065] In step S3, the offline sub-models C1 to C2 can be sent to the corresponding hardware device H1, and the offline sub-model C3 can be sent to the corresponding hardware device H2. The compiled offline model is run in the corresponding hardware devices H1 and H2, and the offline model including multiple offline sub-models C1 to C3 is deployed to the corresponding hardware devices H1 and H2. For example, the offline sub-models C1 and C2 included in the offline model can be deployed to the hardware device H1, and the offline model C3 can be deployed to the hardware device H2.

[0066] Through this method, when the neural network model is relatively large and the performance of an existing hardware device does not meet the deployment conditions of the neural network model, the neural network model can be deployed to multiple hardware devices to fully utilize the advantages of each hardware device.

[0067] In one possible implementation, during the deployment of the above-mentioned neural network model, each hardware device may correspond to at least one hardware backend, including: a hardware backend using the hardware manufacturer's inference library, a hardware backend using the hardware manufacturer's operator library, or a hardware backend using operators not provided by the hardware manufacturer.

[0068] Among them, the hardware backend of the hardware manufacturer's inference library is used, and the neural network model (or sub-model) to be deployed is input into the backend for deployment processing. After deployment, an offline model (or offline sub-model) that can be run directly on the hardware is obtained. It can be seen that the advantage of using the hardware backend of the hardware manufacturer's inference library is fast docking speed, and the performance and accuracy of the model after deployment can be guaranteed by the hardware manufacturer; however, the disadvantage is poor flexibility and low autonomous controllability. The inference library may not support its own models (independently developed models) well.

[0069] Using a hardware backend with a hardware vendor's operator library allows operators to combine and perform the entire offline model calculation. This approach is more flexible than using a hardware backend with an inference library, provides sufficient performance guarantees for individual operators, and requires a moderate amount of development effort.

[0070] Using hardware backends for operators not provided by hardware vendors, such as self-developed operators, offers a high degree of controllability and is more targeted to proprietary models, but requires a greater investment of time and manpower. Self-developed operators can be expressed in a pre-defined programming language.

[0071] It can be seen that different hardware backends have different advantages. The hardware devices to be deployed can simultaneously utilize the advantages of each hardware backend. In the process of neural network model deployment, the performance of each hardware backend can be fully utilized to improve the efficiency and flexibility of model deployment.

[0072] Therefore, according to the embodiments of the present disclosure, a neural network model to be deployed can be compiled to obtain an offline model comprising multiple offline sub-models, each of which can be deployed to a corresponding hardware backend. The multiple offline sub-models can then be distributed to corresponding hardware devices, each of which can correspond to at least one hardware backend. This can fully leverage the advantages of various hardware backends for specific hardware devices, improving the efficiency and flexibility of neural network model deployment.

[0073] Through the above deployment method, a neural network model can be deployed to one or more hardware devices. After the neural network model is deployed in the hardware device, input data can be obtained. The neural network model deployed to the hardware device then processes the input data to obtain a prediction result. The input data can be determined based on the function of the neural network. For example, the input data can include at least one of voice, text, images, and video.

[0074] For example, the neural network model in the embodiment of the present disclosure can be a face recognition neural network model. Through steps S1 to S3, the face recognition neural network model is deployed on the hardware device. During the application process, input data in the form of pictures can be transmitted to the hardware device. After reasoning is performed, the prediction result will be returned, that is, the part of the input image data that includes the target face.

[0075] In this way, the deployed neural network model can be run to obtain the inference results of the neural network model efficiently and quickly.

[0076] The following describes the neural network model deployment method according to the embodiment of the present disclosure.

[0077] Figure 2 A schematic diagram showing a method for deploying a neural network model according to an embodiment of the present disclosure is shown in FIG. Figure 2 As shown, the disclosed method can implement the deployment of the neural network model through two stages: compilation and operation.

[0078] like Figure 2 As shown, in the compilation stage, the neural network model to be deployed (original model) obtained in step S1 can be input into the compiler, and in step S2, the obtained neural network model is compiled by the compiler to obtain a compiled offline model.

[0079] like Figure 2 As shown, the compiler includes modules such as model representation structure, hardware-independent model transformation, model splitting, and model format conversion. The details are as follows:

[0080] The model representation structure module is used to model the model structure of the neural network model input to the compiler, as well as a set of operator definitions for obtaining the internal model structure adapted to the model transformation. Among them, the neural network model can be composed of individual computing units, and these computing units can be defined as operators. In the neural network model, the operator can correspond to the calculation logic in each layer. For example, the convolution layer is an operator, and the weight summation process in the fully connected layer can also be an operator. The present disclosure does not limit the specific form of the operator.

[0081] The hardware-independent model transformation module includes model optimization operations that are independent of the hardware backend and a set of model transformation operations that are independent of the hardware backend, which can optimize the neural network model.

[0082] The model splitting module is used to split the neural network model into multiple sub-models, each of which can be deployed to a hardware backend.

[0083] The model format conversion module converts the model representation structure into the input format of the hardware backend to facilitate the deployment of the split sub-models.

[0084] like Figure 2 As shown, each hardware backend can be connected to the compiler via a software interface. For example, Hardware A Backend 0 and Hardware A Backend 1 can be connected to the compiler via their respective software interfaces. Hardware A Backend 0 and Hardware A Backend 1 are different hardware backends corresponding to the same hardware device A. This disclosure does not limit the number and type of connected hardware backends.

[0085] Each hardware backend can include backend-related model transformations and backend model deployment modules. For example, hardware A backend 0 includes backend 0-related model transformations and backend 0 model deployment modules; hardware A backend 1 includes backend 1-related model transformations and backend 1 model deployment modules.

[0086] The backend-related model transformation module is used to perform model transformation operations and model optimization operations related to the hardware backend on the sub-model. For example, the backend 0-related model transformation module can be used to perform model transformation operations and model optimization operations related to hardware A backend 0 on the sub-model matched by the hardware backend; the backend 1-related model transformation module can be used to perform model transformation operations and model optimization operations related to hardware A backend 1 on the sub-model matched by the hardware backend.

[0087] The backend model deployment module is used to accept the transformed sub-model and convert it into the corresponding offline sub-model. For example, the backend 0 model deployment module can accept the transformed sub-model related to hardware A backend 0 and convert it into the corresponding offline sub-model; the backend 1 model deployment module can accept the transformed sub-model related to hardware A backend 1 and convert it into the corresponding offline sub-model.

[0088] like Figure 2 As shown, in the running stage, in step S3, the multiple offline sub-models obtained in step S2 can be sent to the corresponding hardware devices, so that the multiple offline sub-models are deployed to the corresponding hardware devices respectively. Figure 2 The model interpreter shown matches the multiple offline sub-models included in the offline model to the corresponding hardware devices. It then runs the received offline sub-models on each hardware device to complete the deployment of the neural network model.

[0089] For example, you can use Figure 2 The model interpreter shown matches the offline sub-models obtained during the compilation phase using hardware A backend 0 and hardware A backend 1 to hardware A, and runs these two offline sub-models on hardware A to complete model deployment on hardware A. It should be understood that for hardware B, the model interpreter can be used to match the offline sub-models obtained during the compilation phase using hardware B backend, which will not be discussed further here.

[0090] Among them, Figure 2 As shown, the hardware devices to be deployed may include multiple hardware devices, such as hardware A and hardware B. Each hardware device may include device management, memory management, etc., as well as an operator kernel function set. Each hardware device may have a built-in operating system (such as a Unix operating system, a Linux operating system, etc.) for device management and memory management, etc.; and each hardware device may also have a built-in operator kernel function set, such as a software development kit (SDK) including an operator kernel function set. The neural network model deployed on the hardware device may call the operator kernel function in the SDK. The present disclosure does not limit the number of hardware devices, the type of operating system of each hardware device, and the type of operator kernel function set.

[0091] Therefore, based on Figure 2 The neural network model deployment framework shown in the figure can compile the neural network model to be deployed (original model) through the editor during the compilation phase to obtain an offline model including multiple offline sub-models; during the running phase, the multiple offline sub-models are sent to the corresponding hardware devices, and the offline model composed of multiple offline sub-models can be run in the corresponding hardware devices to complete the deployment of the neural network model.

[0092] The following describes the process of implementing the neural network model deployment through the compilation and operation stages of the disclosed method.

[0093] Figure 3 A flowchart of the compilation phase in the neural network model deployment method according to an embodiment of the present disclosure is shown as follows: Figure 3 As shown, step S2 may include:

[0094] In step S21, the neural network model is structurally transformed to obtain an internal model structure adapted to the model transformation;

[0095] In step S22, the internal model structure is split according to each of the hardware backends to be deployed to obtain a plurality of sub-models and a series relationship between the plurality of sub-models, wherein each sub-model corresponds to a target hardware backend;

[0096] In step S23, for any sub-model, a model transformation operation related to the target hardware backend is performed on the sub-model to obtain an offline sub-model deployed to the target hardware backend;

[0097] In step S24, the offline model is determined according to the multiple offline sub-models and the series relationship.

[0098] For example, in step S21, the input neural network model (original model) can be structurally transformed according to the model representation structure module of the compiler to obtain an internal model structure suitable for the model transformation of the compiler, that is, a neural network model adapted to the model transformation state.

[0099] For example, assume that the input neural network model includes a network structure file describing the neural network and a parameter information file storing the network weights. The model representation structure module in the compiler can be used to reconstruct the input neural network model based on the input network structure file and parameter information file to obtain an internal model structure that adapts to the model transformation. During the reconstruction of the neural network model, the operators used for reconstruction are aggregated to obtain an operator definition set. This operator definition set can include the operators used to construct the internal model structure.

[0100] If the internal model structure adapted to model transformation obtained in step S21 is relatively simple, then the operators used to construct the internal model structure are also relatively simple and can be used with various hardware backends. In this case, in step S22, the internal model structure can be directly split according to the hardware backends to be deployed to obtain multiple sub-models and the series relationship between the multiple sub-models.

[0101] If the internal model structure adapted to the model transformation obtained in step S21 is relatively complex, correspondingly, among the operators used to construct the internal model structure, there are relatively complex operators, and some operators are operators that can only be supported for specific hardware backends and cannot support most hardware backends. Before step S22, the internal model structure can be subjected to model optimization operations that are unrelated to the hardware backend, and / or model transformation operations and model optimization operations related to the hardware backend, so that each operator included therein can be supported for use by various types of hardware backends. In step S22, the internal model structure after various operations is split according to each of the hardware backends to be deployed to obtain multiple sub-models and a series relationship between the multiple sub-models.

[0102] In one possible implementation, a model optimization operation that is independent of the hardware backend is performed on the internal model structure.

[0103] For example, after the internal model structure adapted to the model transformation is obtained in step S21, model optimization operations that are independent of the hardware backend can be performed on the internal model structure adapted to the model transformation, such as operator equivalent replacement, operator merging, and other model optimization operations that are independent of the hardware backend, to obtain the optimized internal model structure, that is, the neural network model that is independent of the hardware backend.

[0104] For example, suppose that the multiple operators that make up the internal model structure include the Batch Normalization (BN) operator. The Batch Normalization operator is used to standardize the output of the network layer of the neural network model to make the output of each layer more stable. In this case, the internal model structure can be transformed to a model that is independent of the hardware backend. The Batch Normalization operator can be replaced with subtraction and division operations to obtain an internal model structure that is independent of the hardware backend.

[0105] Among them, the equivalent replacement of the batch normalization operator can be expressed as: BatchNorm(x,mean,std_var)=(x-mean) / std_var, BatchNorm represents the operator used for replacement, which can output normalized batch data, x represents the data to be processed in each layer, mean represents the mean data, and std_var represents the normalization parameter.

[0106] It can be seen that by performing model optimization operations on the internal structure that are independent of the hardware backend, the operators in the internal structure that are only supported by specific hardware backends can be equivalently replaced with operators that support most hardware backends, thereby obtaining an internal model structure that is independent of the hardware backend.

[0107] Furthermore, in order to improve the processing efficiency of subsequent steps, during the model optimization operation that is independent of the hardware backend, operators can be merged on the internal model structure to obtain a simpler internal model structure that is independent of the hardware backend.

[0108] For example, assuming that the multiple operators constituting the internal model structure include a convolution operator and a multiplication operator following the convolution operator, a model transformation operation that is independent of the hardware backend can be performed on the internal model structure, and the multiplication operator can be merged into the convolution operator Conv to obtain an internal model structure that is independent of the hardware backend.

[0109] The convolution and multiplication operators can be combined as follows: Conv(x, filter) * scale = Conv(x, filter * scale), where x represents the data to be processed, filter represents the convolution kernel, and scale represents the multiplication coefficient. If both filter and scale are constants, they can be calculated at compile time.

[0110] In this way, the internal model structure obtained that is independent of the hardware backend (that is, the neural network model that is independent of the hardware backend) is a neural network model that can support most hardware backends, which is conducive to improving the deployment efficiency of the neural network model and reducing the deployment difficulty.

[0111] In a possible implementation, after a model optimization operation that is unrelated to a hardware backend is performed on the internal model structure, a model transformation operation and a model optimization operation that are related to a hardware backend are performed on the internal model structure.

[0112] For example, after obtaining the internal model structure that is independent of the hardware backend, model transformation operations and model optimization operations related to the hardware backend can be performed on the internal model structure that is independent of the hardware backend according to the hardware backend to be deployed. This may include replacing operators that are not supported by the hardware backend to be deployed with supported operators as much as possible, or performing equivalent replacement of operators based on the characteristics of the hardware backend, and other model transformation operations related to the hardware backend, to obtain the internal model structure related to the hardware backend.

[0113] For example, assuming that the hardware backend to be deployed supports network model-level operators but does not support network layer-level operators, and the internal model structure (for example, an internal model structure unrelated to the hardware backend) includes network layer-level operators, the internal model structure can be subjected to model transformation operations related to the hardware backend according to the hardware backend to be deployed, and the network layer-level operators are replaced with network model-level operators to obtain the internal model structure after the model transformation operations related to the hardware backend.

[0114] Alternatively, assuming that the internal model structure includes multiple small-scale matrix operators that can support most hardware backends, and the hardware backend to be deployed has relatively good performance and supports various large-scale or small-scale matrix operation operators, the internal model structure can be subjected to model optimization operations related to the hardware backend, and the multiple small-scale matrix operators of the internal model structure can be equivalently merged into a large-scale matrix operator that supports the hardware backend to be deployed, thereby obtaining the internal model structure after the model optimization operations related to the hardware backend.

[0115] It should be understood that the present disclosure does not limit the specific operator operation content included in the model transformation operations and model optimization operations related to the hardware backend.

[0116] In this way, we can obtain the internal model structure that supports each hardware backend, which is a neural network model that can support all hardware backends. This is conducive to making full use of the characteristics of each hardware backend, improving the deployment efficiency of the neural network model, and reducing the deployment difficulty.

[0117] In a possible implementation, performing a model transformation operation related to a hardware backend on the internal model structure includes: performing a model transformation operation related to a hardware backend with a highest preset priority on the internal model structure.

[0118] For example, the hardware backend for deploying the internal structure model may include a hardware backend that uses the hardware vendor's inference library, a hardware backend that uses the hardware vendor's operator library, or a hardware backend that uses operators not provided by the hardware vendor. Priorities for each hardware backend can be preset based on the hardware backend to be deployed. For example, a hardware backend that uses the hardware vendor's inference library can be prioritized over a hardware backend that uses the hardware vendor's operator library, and a hardware backend that uses the hardware vendor's operator library can be prioritized over a hardware backend that uses an operator library not provided by the hardware vendor.

[0119] According to the priority level preset by the hardware backend, the internal model structure can be subjected to model transformation operations related to the hardware backend with the highest preset priority. For example, when the operators included in the internal model structure can use the hardware backend of the hardware manufacturer's inference library or operator library, or the hardware backend provided by a non-hardware manufacturer, the internal model structure can be subjected to model transformation operations related to the hardware backend according to the hardware backend with the highest preset priority, that is, the hardware backend of the hardware manufacturer's inference library.

[0120] Among them, if there are still parts in the internal model structure after transformation that do not support the use of the hardware backend of the hardware manufacturer's inference library, you can continue to follow the priority order of the remaining hardware backends, that is, according to the priority order of the hardware backend using the hardware manufacturer's operator library and the hardware backend using the hardware backend provided by non-hardware manufacturers, for the part of the internal model structure that does not support the use of the hardware backend of the hardware manufacturer's inference library, perform model transformation operations related to the hardware backend using the hardware manufacturer's operator library, as well as model transformation operations related to the hardware backend using the hardware backend provided by non-hardware manufacturers.

[0121] In this way, model transformation operations related to the hardware backend can be performed on the internal model structure according to the priority of the hardware backend. Not only can the internal model structure that supports each hardware backend be obtained, but the efficiency of performing model transformation operations related to the hardware backend on the internal model structure is also improved.

[0122] After obtaining an internal model structure that supports each hardware backend, in step S22, the internal model structure can be split according to the support status of each hardware backend to be deployed, to obtain multiple sub-models and the series relationship between the multiple sub-models. Each sub-model corresponds to a target hardware backend, and each sub-model can be deployed to the target hardware backend in subsequent steps. Each target hardware backend can be connected to the compiler via a software interface.

[0123] For example, suppose the hardware backends to be deployed include TensorRT and CUDNN. Although they can match the same hardware devices (GPUs supporting the CUDA architecture), they contain different operator libraries. Using TensorRT for internal model structure deployment is more efficient, but it does not support some model fragments within the internal model structure. For these model fragments, you can use CUDNN for deployment.

[0124] Therefore, the internal model structure can be split according to the support of each hardware backend to be deployed, such as hardware backend performance, operator library type, etc., so that the multiple sub-models obtained by splitting can be more suitable for the corresponding target hardware backend.

[0125] For example, assuming the internal model structure may include two convolutional layers A1 and A2, a pooling layer A3, and a fully connected layer A4. Among the target hardware backends that the compiler can connect to via software interfaces, hardware backend 0 includes a large number of convolution operators and is more supportive of convolution operations, while hardware backend 1 and hardware backend 2 are more supportive of pooling networks and fully connected networks, respectively. In this case, the internal model structure can be split into three sub-models: sub-model 1 includes convolutional layers A1 and A2, sub-model 2 includes pooling layer A3, and sub-model 3 includes connection layer A4. Furthermore, the data output interface of sub-model 1 can be connected to the data input interface of sub-model 2, and the data output interface of sub-model 2 can be connected to the data input interface of sub-model 3. Based on the data input and output dependencies between sub-models 1 to 3, the series relationship between the sub-models can be determined. This series relationship facilitates the subsequent integration of the resulting sub-models into a neural network model with the same functionality as before the split.

[0126] Among them, sub-model 1 can correspond to hardware backend 0, sub-model 2 can correspond to hardware backend 2, and sub-model 3 can correspond to hardware backend 3.

[0127] By splitting the internal model structure, it helps to take advantage of the advantages of each hardware backend.

[0128] It should be understood that the above-mentioned splitting method according to the network layer is only for illustration. Any network layer in the neural network model can also be split into multiple sub-models. For example, the fully connected layer A4 can be split into a sub-model representing matrix multiplication and a sub-model representing matrix addition. The present disclosure does not limit the specific splitting method, which can be determined according to the support of each hardware backend to be deployed.

[0129] Among them, the hardware backend may include: a backend that uses the hardware manufacturer's inference library for model-level access, a backend that uses the hardware manufacturer's operator library for operator-level access, or a backend that uses operators provided by non-hardware manufacturers for operator-level access, such as a backend that independently sets operators, a backend that uses backup CPU operator access, etc. The present disclosure does not limit the specific type of hardware backend.

[0130] In a possible implementation, the target hardware backend is a hardware backend with the highest preset priority among the hardware backends deployable by the sub-model.

[0131] For example, the backend that uses the hardware vendor's inference library for model-level access can be preset as the highest-priority hardware backend. In this case, when splitting the internal model structure, the first step is to split it according to the hardware vendor's inference library, facilitating the separation of the parts supported by the hardware vendor's inference library from the internal model structure. For parts of the internal model structure not supported by the hardware vendor's inference library, the hardware vendor's operator library or self-configured operators can then be used for further splitting, until each sub-model split from the internal model structure corresponds to a target hardware backend.

[0132] This approach allows for splitting of internal models based on the preset priorities of the hardware backends to be deployed, improving splitting efficiency. It also facilitates deploying the resulting sub-models to the hardware backend with the highest preset priority, improving the efficiency of subsequent neural network model deployment.

[0133] As can be seen, through step S22, when deploying a model to a piece of hardware, the advantages of various access levels can be simultaneously utilized. Furthermore, since each hardware backend can be connected to the compiler via a software interface, newly added hardware devices can be quickly connected to the hardware backend corresponding to the newly added hardware device. This provides strong scalability, facilitates the rapid deployment of the neural network model to the hardware device, and achieves a better balance between access efficiency, model deployment instructions, and manpower consumption.

[0134] In step S22, a plurality of sub-models are obtained. In step S23, for any sub-model, a model transformation operation related to the target hardware backend corresponding to the sub-model is performed on the sub-model to obtain an offline sub-model deployed to the target hardware backend.

[0135] For example, from all sub-models, model transformation operations related to their corresponding target hardware backends can be performed on the sub-models one by one according to the matching target hardware backend, until all sub-models are deployed to the corresponding target hardware backend, thereby obtaining offline sub-models deployed to the target hardware backend.

[0136] Alternatively, based on the target hardware backend matched by each sub-model, model transformation operations related to the target hardware backend can be performed on each sub-model in parallel, and each sub-model can be deployed to the corresponding target hardware backend to obtain offline sub-models deployed to the target hardware backend.

[0137] It should be understood that the present disclosure may deploy sub-models on target hardware backends one by one, or may deploy sub-models on each target hardware backend in parallel. The present disclosure does not limit the specific deployment method.

[0138] In a possible implementation, step S23 may include:

[0139] In step S231, a model transformation operation related to the target hardware backend is performed on the sub-model to obtain a sub-model in a first state;

[0140] In step S232, the sub-model in the first state is format-converted to obtain a sub-model in the second state, wherein the sub-model in the second state is adapted to the input format of the target hardware backend;

[0141] In step S233, the sub-model in the second state is deployed to the target hardware backend to obtain the offline sub-model.

[0142] For example, considering that the model transformation operations related to the hardware backend applied in steps S21 to S22 do not necessarily correspond to the hardware backend where the sub-model is actually deployed, it is necessary to perform model transformation operations related to the target hardware backend on the sub-model again before actually deploying it on the target hardware backend.

[0143] In step S231, according to the back-end related model transformation module in the target hardware back-end, the sub-model can be subjected to model transformation operations related to the target hardware back-end, for example, operators not supported by the target hardware back-end can be replaced with equivalent operators supported by the target hardware back-end to obtain a sub-model in the first state.

[0144] Different target hardware backends may include different hardware backends that match the same hardware device, and the input formats of the sub-models corresponding to each target hardware backend may be different. Therefore, after obtaining the sub-model in the first state, it is necessary to convert the sub-model in the first state into the format required by the target hardware backend. In step S232, the sub-model in the first state can be format converted according to the model format conversion module of the compiler to obtain the sub-model in the second state that is adapted to the input format of the target hardware backend.

[0145] After completing steps S231 and S232, the sub-model in the second state can be deployed to the target hardware backend according to the model deployment module in the target hardware backend to obtain an offline sub-model, that is, a sub-graph composed of the operators included in the target hardware backend.

[0146] It can be seen that in the design of the hardware backend, each hardware device can correspond to one or more hardware backends, and each hardware backend can provide different levels of access ports for the compiler.

[0147] In this way, an offline sub-model deployed to each target hardware backend is obtained, which is conducive to fully utilizing the advantages of each hardware backend.

[0148] In step S23, multiple offline sub-models deployed to the corresponding target hardware backend are obtained. In step S24, the offline sub-models are connected in series to form an offline model according to the offline sub-models of the multiple sub-models and the series relationship.

[0149] Among them, since each sub-model can correspond to an offline sub-model deployed to the target hardware backend, the data input and output dependency between the sub-models is consistent with the data input and output dependency between the corresponding offline sub-models. Therefore, the series relationship between the sub-models can be determined as the series relationship between the corresponding offline sub-models.

[0150] Therefore, in the compilation process described in steps S21 to S24, multiple levels of access ports can be used simultaneously to split the neural network model into multiple offline sub-models, which are deployed to more suitable hardware backends respectively, so as to fully utilize the advantages of different hardware backends; and, by using model transformations related to the hardware backend, the neural network model to be deployed can be deployed to multiple hardware backends without manual modification, and adaptation to different hardware devices (such as inference hardware) can be achieved with one compilation, which is conducive to isolating the differences in the hardware device tool chain interfaces during runtime and providing a unified neural network model inference interface for the business layer.

[0151] After obtaining the offline model including multiple offline sub-models in step S2, in the operation stage, the multiple offline sub-models can be sent to the corresponding hardware devices in step S3 so that the hardware devices can run the offline models and realize the deployment of the offline models on the corresponding hardware devices.

[0152] In a possible implementation, step S3 may include: reading the multiple offline sub-models of the offline model and the series relationship between the multiple offline sub-models through a model interpreter; wherein, the model interpreter connects the multiple offline sub-models in series according to the series relationship between the multiple offline sub-models when the hardware device is running.

[0153] For example, assuming that multiple hardware backends in the compilation stage correspond to one hardware device H1, the model interpreter can read multiple offline sub-models of the offline model and the series relationship between the offline sub-models into the memory, that is, the dynamic random access memory (DRAM), and then send the multiple offline sub-models to the matching hardware device, that is, all the model sub-models can be sent to the hardware device H1. The model interpreter can connect the multiple offline sub-models in series when running in the hardware device H1 based on the series relationship between the multiple offline sub-models.

[0154] Alternatively, assume that the multiple hardware backends in the compilation phase correspond to multiple hardware devices, which may include hardware device H1 and hardware device H2, for example. The model interpreter can read the multiple offline sub-models of the offline model and the series relationship between the offline sub-models into the memory, and then send the multiple offline sub-models to the matching hardware devices. Each offline sub-model matching the hardware device H1 can be sent to the hardware device H1, and at the same time, each offline sub-model matching the hardware device H2 can be sent to the hardware device H2. The model interpreter can connect the multiple offline sub-models in series according to the series relationship between the multiple offline sub-models when the hardware device is running, and through the cooperation of the hardware device H1 and the hardware device H2, jointly run the offline model including the multiple offline sub-models.

[0155] In this way, the advantages of various hardware devices can be comprehensively utilized, and the differences in the hardware backend can be shielded when running offline models on the hardware devices, which is conducive to providing a more stable and unified calling interface for the business layer.

[0156] Therefore, according to the embodiments of the present disclosure, during the compilation process, multiple levels of access ports can be used simultaneously to split the neural network model into multiple offline sub-models, which are respectively deployed to more suitable hardware backends, thereby fully utilizing the advantages of different hardware backends; and, by using model transformations related to the hardware backend, the neural network model to be deployed can be deployed to multiple hardware backends without manual modification, and adaptation to different hardware devices (such as inference hardware) can be achieved with one compilation, which is conducive to isolating differences in hardware device tool chain interfaces during runtime and providing a unified neural network model inference interface for the business layer.

[0157] It is understood that the above-mentioned various method embodiments mentioned in this disclosure can be combined with each other to form combined embodiments without violating the principle logic. Due to space limitations, this disclosure will not go into details. It is understood by those skilled in the art that in the above-mentioned methods of specific implementation, the specific execution order of each step should be determined by its function and possible internal logic.

[0158] In addition, the present disclosure also provides a neural network model deployment device, an electronic device, a computer-readable storage medium, and a program, all of which can be used to implement any neural network model deployment method provided by the present disclosure. The corresponding technical solutions and descriptions can be found in the corresponding records in the method section and will not be repeated here.

[0159] Figure 4 A block diagram of a neural network model deployment device according to an embodiment of the present disclosure is shown as follows: Figure 4 As shown, the device is applied to an electronic device, including:

[0160] An acquisition module 41 is used to acquire a neural network model to be deployed;

[0161] A compilation module 42 is configured to compile the neural network model to obtain a compiled offline model, wherein the offline model includes multiple offline sub-models, each of which is deployed to a corresponding hardware backend. Each hardware backend corresponds to a different tool chain for deploying the neural network model to a hardware device, and each hardware device corresponds to at least one hardware backend.

[0162] The running module 43 is used to send the multiple offline sub-models to the corresponding hardware devices.

[0163] In one possible implementation, the compilation module includes 42: a structure conversion module, which is used to perform structure conversion on the neural network model to obtain an internal model structure that is suitable for model transformation; a splitting module, which is used to split the internal model structure according to each of the hardware backends to be deployed, to obtain multiple sub-models and a series relationship between the multiple sub-models, wherein each sub-model corresponds to a target hardware backend; an offline sub-model acquisition module, which is used to perform a model transformation operation related to the target hardware backend on any sub-model to obtain an offline sub-model deployed to the target hardware backend; and an offline model determination module, which is used to determine the offline model based on the multiple offline sub-models and the series relationship.

[0164] In a possible implementation, the target hardware backend is a hardware backend with the highest preset priority among the hardware backends deployable by the sub-model.

[0165] In a possible implementation, the compilation module 42 also includes a first module for performing model transformation operations and model optimization operations related to the hardware backend on the internal model structure before splitting the internal model structure according to the various hardware backends to be deployed.

[0166] In a possible implementation, performing a model transformation operation related to a hardware backend on the internal model structure includes: performing a model transformation operation related to a hardware backend with a highest preset priority on the internal model structure.

[0167] In a possible implementation, the compilation module 42 further includes a second module for performing a model optimization operation unrelated to the hardware backend on the internal model structure before performing a model transformation operation and a model optimization operation related to the hardware backend on the internal model structure.

[0168] In one possible implementation, the offline sub-model acquisition model is used to: perform a model transformation operation related to the target hardware backend on the sub-model to obtain a sub-model in a first state; perform format conversion on the sub-model in the first state to obtain a sub-model in a second state, wherein the sub-model in the second state is adapted to the input format of the target hardware backend; and deploy the sub-model in the second state to the target hardware backend to obtain the offline sub-model.

[0169] In one possible implementation, the running module 43 is used to: read the multiple offline sub-models of the offline model and the series relationship between the multiple offline sub-models through a model interpreter; and send each of the offline sub-models to the corresponding hardware device, wherein the model interpreter connects the multiple offline sub-models in series when the hardware device is running according to the series relationship between the multiple offline sub-models.

[0170] In one possible implementation, the hardware backend includes: a hardware backend using a hardware vendor's reasoning library, a hardware backend using a hardware vendor's operator library, or a hardware backend using an operator not provided by the hardware vendor.

[0171] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments. For the sake of brevity, it will not be repeated here.

[0172] The present disclosure also provides a computer-readable storage medium having computer program instructions stored thereon, wherein the computer program instructions implement the above method when executed by a processor. The computer-readable storage medium may be a volatile or non-volatile computer-readable storage medium.

[0173] An embodiment of the present disclosure further proposes an electronic device, comprising: a processor; and a memory for storing instructions executable by the processor; wherein the processor is configured to call the instructions stored in the memory to execute the above method.

[0174] An embodiment of the present disclosure also provides a computer program product, including computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the above method.

[0175] The electronic device may be provided as a terminal, a server, or other forms of devices.

[0176] Figure 5The block diagram of an electronic device 800 according to an embodiment of the present disclosure is shown. For example, the electronic device 800 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, or the like.

[0177] Reference Figure 5 , the electronic device 800 may include one or more of the following components: a processing component 802 , a memory 804 , a power component 806 , a multimedia component 808 , an audio component 810 , an input / output (I / O) interface 812 , a sensor component 814 , and a communication component 816 .

[0178] The processing component 802 generally controls the overall operation of the electronic device 800, such as operations associated with display, phone calls, data communications, camera operation, and recording operations. The processing component 802 may include one or more processors 820 to execute instructions to perform all or part of the steps of the above-described method. In addition, the processing component 802 may include one or more modules to facilitate interaction between the processing component 802 and other components. For example, the processing component 802 may include a multimedia module to facilitate interaction between the multimedia component 808 and the processing component 802.

[0179] The memory 804 is configured to store various types of data to support operations on the electronic device 800. Examples of such data include instructions for any application or method operating on the electronic device 800, contact data, phone book data, messages, pictures, videos, etc. The memory 804 can be implemented by any type of volatile or non-volatile storage device, or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk, or optical disk.

[0180] The power supply component 806 provides power to the various components of the electronic device 800. The power supply component 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device 800.

[0181] The multimedia component 808 includes a screen that provides an output interface between the electronic device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen can be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor can not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 808 includes a front camera and / or a rear camera. When the electronic device 800 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera can receive external multimedia data. Each front camera and rear camera can be a fixed optical lens system or have a focal length and optical zoom capability.

[0182] The audio component 810 is configured to output and / or input audio signals. For example, the audio component 810 includes a microphone (MIC), which is configured to receive external audio signals when the electronic device 800 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signal can be further stored in the memory 804 or transmitted via the communication component 816. In some embodiments, the audio component 810 also includes a speaker for outputting audio signals.

[0183] I / O interface 812 provides an interface between processing component 802 and peripheral interface modules, such as a keyboard, click wheel, buttons, etc. These buttons may include but are not limited to: a home button, volume buttons, a start button, and a lock button.

[0184] The sensor assembly 814 includes one or more sensors for providing various aspects of status assessment for the electronic device 800. For example, the sensor assembly 814 can detect the open / closed state of the electronic device 800, the relative positioning of components, such as the display and keypad of the electronic device 800. The sensor assembly 814 can also detect changes in the position of the electronic device 800 or a component of the electronic device 800, the presence or absence of user contact with the electronic device 800, the orientation or acceleration / deceleration of the electronic device 800, and the temperature change of the electronic device 800. The sensor assembly 814 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 814 may also include a light sensor, such as a complementary metal oxide semiconductor (CMOS) or charge coupled device (CCD) image sensor, for use in imaging applications. In some embodiments, the sensor assembly 814 may also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0185] The communication component 816 is configured to facilitate wired or wireless communication between the electronic device 800 and other devices. The electronic device 800 can access a wireless network based on a communication standard, such as a wireless network (WiFi), a second generation mobile communication technology (2G) or a third generation mobile communication technology (3G), or a combination thereof. In an exemplary embodiment, the communication component 816 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 816 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0186] In an exemplary embodiment, the electronic device 800 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above methods.

[0187] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 804 including computer program instructions. The computer program instructions can be executed by the processor 820 of the electronic device 800 to perform the above method.

[0188] Figure 6 FIG1 shows a block diagram of an electronic device 1900 according to an embodiment of the present disclosure. For example, the electronic device 1900 may be provided as a server. Figure 6 The electronic device 1900 includes a processing component 1922, which further includes one or more processors, and a memory resource represented by a memory 1932 for storing instructions executable by the processing component 1922, such as an application. The application stored in the memory 1932 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 1922 is configured to execute the instructions to perform the above-described method.

[0189] The electronic device 1900 may further include a power supply component 1926 configured to perform power management of the electronic device 1900, a wired or wireless network interface 1950 configured to connect the electronic device 1900 to a network, and an input / output (I / O) interface 1958. The electronic device 1900 may operate based on an operating system stored in the memory 1932, such as a Microsoft Server operating system (Windows Server 2003). TM), a graphical user interface operating system launched by Apple (Mac OSX TM ), a multi-user, multi-process computer operating system (Unix TM ), a free and open source Unix-like operating system (Linux TM ), an open-source Unix-like operating system (FreeBSD TM ) or similar.

[0190] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions that can be executed by the processing component 1922 of the electronic device 1900 to perform the above method.

[0191] The present disclosure may be a system, method and / or computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present disclosure.

[0192] Computer-readable storage media can be a tangible device that can hold and store the instructions used by the instruction execution device. Computer-readable storage media can be, for example, (but not limited to) an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (non-exhaustive list) of computer-readable storage media include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, for example, a punch card or a convex structure in a groove on which instructions are stored, and any suitable combination thereof. Computer-readable storage media used herein is not interpreted as a transient signal itself, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagated by waveguides or other transmission media (for example, light pulses by fiber optic cables), or electrical signals transmitted by wires.

[0193] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions to be stored in the computer-readable storage medium in each computing / processing device.

[0194] The computer program instructions for performing the operations of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, and conventional procedural programming languages ​​such as "C" language or similar programming languages. Computer-readable program instructions may be executed entirely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., utilizing an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be personalized by utilizing the state information of the computer-readable program instructions. The electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present disclosure.

[0195] Various aspects of the present disclosure are described herein with reference to flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present disclosure. It should be understood that each block of the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer-readable program instructions.

[0196] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device is generated that implements the functions / actions specified in one or more blocks in the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium, where these instructions cause the computer, programmable data processing device, and / or other device to operate in a specific manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing various aspects of the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0197] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0198] The flow charts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the systems, methods and computer program products according to multiple embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and the part of the module, program segment or instruction contains one or more executable instructions for realizing the prescribed logical function. In some alternative implementations, the functions marked in the box can also occur in a sequence different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented by a dedicated hardware-based system that performs the prescribed function or action, or can be implemented by a combination of dedicated hardware and computer instructions.

[0199] The computer program product may be implemented in hardware, software, or a combination thereof. In one embodiment, the computer program product is implemented as a computer storage medium. In another embodiment, the computer program product is implemented as a software product, such as a software development kit (SDK).

[0200] While various embodiments of the present disclosure have been described above, the above descriptions are illustrative, non-exhaustive, and not intended to be limiting of the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is selected to best explain the principles of the embodiments, their practical applications, or improvements to existing technologies, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A neural network model deployment method, characterized in that: Applied to electronic equipment, the method includes: Get the neural network model to be deployed; Compiling the neural network model to obtain a compiled offline model, wherein the offline model includes multiple offline sub-models, each of the offline sub-models is deployed to a corresponding hardware backend, each hardware backend corresponds to a different tool chain for deploying the neural network model to a hardware device, and each hardware device corresponds to at least one hardware backend; Sending the multiple offline sub-models to corresponding hardware devices; The step of compiling the neural network model to obtain a compiled offline model includes: Performing structural transformation on the neural network model to obtain an internal model structure adapted to the model transformation; Splitting the internal model structure according to each of the hardware backends to be deployed to obtain a plurality of sub-models and a series relationship between the plurality of sub-models, wherein each sub-model corresponds to a target hardware backend; For any sub-model, perform a model transformation operation related to the target hardware backend on the sub-model to obtain an offline sub-model deployed to the target hardware backend; The offline model is determined according to the multiple offline sub-models and the series relationship.

2. The method according to claim 1, characterized in that The target hardware backend is a hardware backend with the highest preset priority among the hardware backends deployable by the sub-model.

3. The method according to claim 2, characterized in that Before splitting the internal model structure according to each of the hardware backends to be deployed, the method further includes: The internal model structure is subjected to model transformation operations and model optimization operations related to the hardware backend.

4. The method according to claim 3, characterized in that Performing a model transformation operation related to the hardware backend on the internal model structure includes: A model transformation operation associated with a hardware backend with the highest preset priority is performed on the internal model structure.

5. The method according to claim 3, characterized in that Before performing model transformation operations and model optimization operations related to the hardware backend on the internal model structure, the method further includes: A model optimization operation is performed on the internal model structure that is independent of the hardware backend.

6. The method according to claim 1, characterized in that The performing a model transformation operation on the sub-model related to the target hardware backend to obtain an offline sub-model deployed to the target hardware backend includes: Performing a model transformation operation related to the target hardware backend on the sub-model to obtain a sub-model in a first state; Performing format conversion on the sub-model in the first state to obtain a sub-model in the second state, wherein the sub-model in the second state is adapted to an input format of the target hardware backend; The sub-model in the second state is deployed to the target hardware backend to obtain the offline sub-model.

7. The method according to any one of claims 1 to 6, characterized in that Sending the multiple offline sub-models to corresponding hardware devices includes: Reading the multiple offline sub-models of the offline model and the series relationship between the multiple offline sub-models through a model interpreter; Each of the offline sub-models is sent to a corresponding hardware device respectively, wherein the model interpreter connects the multiple offline sub-models in series when the hardware device is running according to the series relationship between the multiple offline sub-models.

8. The method according to any one of claims 1 to 6, characterized in that The hardware backend includes: a hardware backend using a hardware manufacturer's reasoning library, a hardware backend using a hardware manufacturer's operator library, or a hardware backend using an operator not provided by the hardware manufacturer.

9. A neural network model deployment device, characterized in that: Used in electronic equipment, including: The acquisition module is used to obtain the neural network model to be deployed; A compilation module, configured to compile the neural network model to obtain a compiled offline model, wherein the offline model includes multiple offline sub-models, each of which is deployed to a corresponding hardware backend, each of which corresponds to a different tool chain for deploying the neural network model to a hardware device, and each hardware device corresponds to at least one hardware backend; An operation module, configured to send the multiple offline sub-models to corresponding hardware devices; Wherein, the compilation module is used for: Performing structural transformation on the neural network model to obtain an internal model structure adapted to the model transformation; Splitting the internal model structure according to each of the hardware backends to be deployed to obtain a plurality of sub-models and a series relationship between the plurality of sub-models, wherein each sub-model corresponds to a target hardware backend; For any sub-model, perform a model transformation operation related to the target hardware backend on the sub-model to obtain an offline sub-model deployed to the target hardware backend; The offline model is determined according to the multiple offline sub-models and the series relationship.

10. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to execute the method according to any one of claims 1 to 8.

11. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images