Method and apparatus for managing a neural network model

Abstract

A method of managing deep neural network (DNN) models on a device is provided. The method includes extracting information associated with each of a plurality of DNN models, identifying common information common across the plurality of DNN models from the information, separating and storing the common information to a designated location in the device, and controlling at least one DNN model of the plurality of DNN models to access the common information.

Description

TECHNICAL FIELD

[0001] The disclosure relates to deployment or management of a neural network in a device, and more particularly, to a method and apparatus for managing a neural network model based on redundancy in a structure of the neural network model. BACKGROUND

[0002] Applications requiring / utilizing deep learning methods are common in embedded devices such as, but not limited to, smartphones, Internet of Things (IoT) devices, personal computers (PCs), tablet PCs, etc. In order to employ a deep learning method to execute an instruction of an application, a deep neural network (DNN) model needs to be deployed in an embedded device. Such deployment of a DNN model allows a user to install an application in a personal device.

[0003] Currently, a plurality of DNN models can be deployed in a device. The plurality of DNN models can be executed by different applications installed in the device. As the number of applications installed in the device increases, a larger number of DNN models needs to be deployed in the device. When an application is launched or if an instruction related to the application needs to be executed, a DNN model is loaded into a central processing unit (CPU) or other processing unit in the device, if any. When the execution is completed, the DNN model is unloaded from the CPU or from the other processing unit. If the device has an application capable of running in multiple operation modes, different DNN models are loaded / unloaded when there is a mode switch. SUMMARY

[0004] TECHNICAL PROBLEM

[0005] The process of loading and unloading a DNN model during application launch or mode switching consumes time, thereby degrading the latency performance of the device.

[0006] SOLUTION TO THE PROBLEM

[0007] To solve the above technical problem, a method of managing deep neural network (DNN) models on a device is provided. The method includes extracting information associated with each of a plurality of DNN models, identifying common information common to the plurality of DNN models from the information, separating and storing the common information to a designated location in the device, and controlling at least one DNN model of the plurality of DNN models to access the common information.

[0008] ADVANTAGEOUS EFFECTS OF THE INVENTION

[0009] The process of loading and unloading a DNN model during application launch or mode switching can be performed without consuming redundant time, and thus, the disclosure improves the performance of the device when an application is launched or switched in the device. BRIEF DESCRIPTION OF DRAWINGS

[0010] The embodiments herein are illustrated in the accompanying drawings, wherein same reference numerals in different drawings denote corresponding parts. The embodiments herein will be better understood from the following description with reference to the drawings, in which:

[0011] Figure 1 is an example scenario illustrating loading and unloading of Deep Neural Network (DNN) models during startup and mode switching of a camera application;

[0012] Figure 2 illustrates various units of a device configured to deploy DNN models in the device based on redundancy in structure of DNN models and dependencies between DNN models, according to an embodiment of the disclosure;

[0013] Figure 3A is a flow diagram illustrating a method for deploying DNN models in a device based on redundancy in structure of DNN models and dependencies between DNN models, according to an embodiment of the disclosure;

[0014] Figure 3B is a flow diagram 310 illustrating a method for managing DNN models, according to an embodiment of the disclosure;

[0015] Figure 4 is an example illustrating identification of redundancy in two DNN models, according to an embodiment of the disclosure;

[0016] Figure 5 is an example illustrating generation of optimized model data indicating redundant layers and non-redundant layers in DNN models utilized by a camera application installed in the device, according to an embodiment of the disclosure;

[0017] Figure 6 is an example illustrating loading of DNN models in different processing units of the device, according to an embodiment of the disclosure;

[0018] Figure 7 is an example illustrating generation of a model dependency graph indicating dependencies between four DNN models utilized by a camera application installed in the device, according to an embodiment of the disclosure;

[0019] Figure 8 is a use case scenario illustrating sequential execution of a detector DNN model and a classifier DNN model for detecting objects in a region of interest (ROI) of media captured by a camera application and classifying the detected objects, according to an embodiment of the disclosure;

[0020] Figure 9 is an example illustrating preloading of DNN models in different processing units of the device by a model preloader, according to an embodiment of the disclosure; and

[0021] Figure 10A and Figure 10B is a use case scenario showing that a DNN model used by a camera application is based on model dependency graph and preloading / loading / unloading of optimization model data according to an embodiment of the disclosure.

[0022] Best Mode for Carrying Out the Invention

[0023] To solve the above technical problem, a method of managing deep neural network (DNN) models on a device is provided. The method includes extracting information associated with each of a plurality of DNN models, identifying common information common to the plurality of DNN models from the information, separating and storing the common information to a designated location in the device, and controlling at least one DNN model of the plurality of DNN models to access the common information.

[0024] Accordingly, embodiments provide methods and systems for deploying deep neural network (DNN) models in a device based on redundancies in the structure of the DNN models and dependencies between the DNN models. Embodiments include identifying redundancies in the structure of the DNN models by comparing each DNN model to other DNN models. Embodiments include determining reference counts related to each layer of each DNN model. Embodiments include traversing the layers of each DNN model and initializing the reference count value for each layer during the traversal. If it is determined that a layer of a DNN model also exists in other DNN models, the reference count can be incremented. If the reference count corresponding to a layer of a DNN model is incremented, the layer of the DNN model can be identified as contributing to a redundancy in the structure of the DNN model, meaning that the layer exists in at least two DNN models. DNN model layers for which the reference count value is not incremented are considered unique. The portion of the structure of the DNN model in which the unique layer falls can be categorized as a specific region.

[0025] Embodiments include determining dependencies between DNN models, where the dependencies indicate an order of execution of the DNN models across multiple applications or within an application. The dependencies between at least two DNN models can be determined by deciding whether the at least two DNN models are executed in parallel, independently, or sequentially by at least one application. Loading and unloading of non-redundant layers of DNN models in a device can be managed based on the dependencies between DNN models across multiple applications or within an application and available memory in the device. If the DNN models are executed sequentially, and if there is redundancy in the structure of the DNN models, then the layers of a specific region of the DNN models can be loaded sequentially. Similarly, if the DNN models are executed in parallel, and if there is redundancy in the structure of the DNN models, then the layers in a specific region of the DNN models can be loaded simultaneously. If the DNN models are executed independently of each other, then there are no dependencies between the DNN models. Accordingly, the loading and unloading of the layers of the DNN models are executed independently. Embodiments herein include preloading layers of DNN models based on the identified redundancy of the DNN models and the dependencies between the DNN models across multiple applications or within an application. DETAILED DESCRIPTION

[0026] To facilitate an understanding of the principles of the present disclosure, reference is made to the embodiments illustrated in the drawings and described herein. It is explicitly contemplated that any such changes and further modifications can be made thereto without departing from the scope of the present disclosure. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs. The systems, methods, and examples provided herein are illustrative only and are not intended to be limiting.

[0027] The term "some" as used herein is defined as "none, or one, or more than one, or all." Accordingly, the terms "none," "one," "more than one," "more than one, but not all," or "all" should all fall under the definition of "some." The term "some embodiments" can refer to no embodiments or to one embodiment or to several embodiments or to all embodiments. Accordingly, the term "some embodiments" is defined as meaning "none, or one, or more than one, or all embodiments."

[0028] The terminology and structure used herein is used to describe, teach and illustrate some embodiments and particular features and elements thereof, and does not limit, restrict or narrow the scope of the claims or their equivalents.

[0029] More specifically, any terms used herein, such as but not limited to “including,” “contains,” “having,” “composes of,” and their grammatical variations, do not specify an exact limitation or constraint, and certainly do not preclude the possibility of adding one or more features or elements unless otherwise stated. Furthermore, they do not preclude the possibility of deleting one or more listed features and elements unless otherwise stated using restrictive language such as “must include” or “requires inclusion.”

[0030] Whether a feature or element is limited to being used only once, it may still be referred to as "one or more features," "one or more elements," "at least one feature," or "at least one element." Furthermore, the use of the terms "one or more" or "at least one" features or elements does not preclude the absence of that feature or element unless otherwise specified by restrictive language (e.g., "must be one or more" or "must be one or more elements").

[0031] Unless otherwise defined, all terms used herein, especially any technical and / or scientific terms, may be considered to have the same meaning as commonly understood by one of ordinary skill in the art.

[0032] This document references several “exemplary embodiments.” It should be understood that these embodiments are examples of possible implementations of any feature and / or element set forth in the appended claims. Several embodiments have been described to illustrate one or more potential ways in which specific features and / or elements of the appended claims satisfy the needs of uniqueness, utility, and non-obviousness.

[0033] The use of phrases and / or terms such as, but not limited to, “first embodiment,” “another embodiment,” “alternative embodiment,” “one embodiment,” “embodiment,” “multiple embodiments,” “some embodiments,” “other embodiments,” “additional embodiments,” “further embodiments,” “additional embodiments,” or variations thereof, does not necessarily refer to the same embodiment. Unless otherwise stated, one or more specific features and / or elements described in connection with one or more embodiments may be found in one embodiment, or may be found in more than one embodiment, or may be found in all embodiments, or may be found in no embodiment. Although one or more features and / or elements may be described herein only in the context of a single embodiment, or alternatively in the context of more than one embodiment, or further alternatively in the context of all embodiments, instead, features and / or elements may be provided individually or in any suitable combination or not at all. Conversely, any features and / or elements described in the context of separate embodiments may alternatively be implemented as being present together in the context of a single embodiment.

[0034] Any specific and all the details set forth herein are used in the context of some embodiments, and thus should not necessarily be taken as limiting factors on the scope of the appended claims. The appended claims and their legal equivalents are capable of being implemented in embodiments other than those described below as illustrative examples.

[0035] Embodiments of the present disclosure will be described in detail below with reference to the attached drawings.

[0036] The process of loading and unloading DNN models during application launch or mode switching consumes time, thereby degrading the latency performance of the device. If DNN models are preloaded in the memory of the CPU / other processing unit, the loading and unloading of DNN models can be skipped. If the applications installed in the device employ a large number of DNN models, and if the requirement of keeping DNN models preloaded in the memory of the CPU / other processing unit is high, preloading can be constrained by the device.

[0037] The process of loading and unloading multiple DNN models during application launch, application exit, and mode switching within an application degrades the performance of the device. Due to the significant memory requirement of complex DNN models, it can not be possible to preload all the required DNN models. Currently, there are methods to optimize the memory usage of each DNN model. However, when multiple DNN models are used, a specific portion of the memory needs to be reserved for storing each DNN model. This limits the number of DNN models that can be used in the device. Other processing units in the device other than the CPU, such as a graphics processing unit (GPU), a digital signal processor (DSP), a neural processing unit (NPU), and the like, can not have enough memory to preload all the DNN models used by the applications in the device. Therefore, due to memory / performance constraints, the developers / designers of the applications can deploy simpler models, which can not improve the performance of the device in terms of utilizing all the features of the applications.

[0038] Currently, transfer learning (based on machine learning) is used to develop or create new DNN models. In transfer learning, a DNN model developed for performing a first task can be reused for performing a second task. The original structure of the DNN is also changed to "create" a new DNN model. Initially (during transfer learning), a pre-trained DNN model with high accuracy, low complexity, and small size is identified. The pre-trained DNN model can be configured to perform the first task. Thereafter, the pre-trained model is trained using a new dataset, wherein there is a subtle difference between the new dataset and the data used to pre-train the DNN model. Once the training using the new dataset is complete, the DNN model can perform the second task. Due to transfer learning, the structure of the DNN model is changed. Training the pre-preserved DNN model using different datasets can result in different new DNN models. The new DNN models can have similarities in their respective structures. Therefore, if multiple DNN models are deployed in a device, and if each DNN model has structural similarities with the other DNN models, then there is unnecessary memory usage.

[0039] Figure 1 is an example scenario illustrating loading and unloading of DNN models during startup and mode switching of a camera application, according to an embodiment of the disclosure. Consider that the camera application has two modes of operation, i.e., a first mode and a second mode. As shown in Figure 1 , consider that the first mode is used when the user starts the camera application. While operating in the first mode, the camera application uses two DNN models. Consider that the DNN models are a first classifier and a first detector. The first detector can detect objects in the camera preview, and the first classifier can classify the detected objects.

[0040] While the camera application operates in the first mode, the first classifier and the first detector are loaded on one processing unit (e.g., GPU). In an example, the time taken to load the first classifier and the first detector on the GPU is approximately 2.7 seconds. When there is a mode switch, the camera application starts operating in the second mode (after switching from the first mode). After the mode switch (from the first mode to the second mode), the first classifier and the first detector are unloaded from the GPU, and a second classifier (DNN model) and a second detector (DNN model) are loaded. The time taken to load the second classifier and the second detector and to unload the first classifier and the first detector can be 2.7 seconds. When the mode is switched from the second mode to the first mode, the first classifier and the first detector are loaded after unloading the second classifier and the second detector. Therefore, the time taken for the processing of loading and unloading the classifiers and the detectors (approximately 2.7 seconds) degrades the performance of the device.

[0041] Latency performance degradation is due to memory constraints of processing units, which constrain preloading of DNN models, and also require frequent loading and unloading. In an example, models are able to be loaded at device startup. To keep each model loaded onto the processing unit, a large amount of memory is required to speed up, which is another constraint of the device. As devices advance, the number of models deployed in the device is constantly increasing. Most models have to run on faster GPUs, DSPs, and NPUs. Although the random access memory (RAM) size of the device has increased greatly, the amount of storage required to keep DNN models preloaded onto the processing unit can still be insufficient.

[0042] It can be seen that deploying DNN models directly on embedded devices presents new challenges, as DNN models utilize significant computational complexity and memory requirements.

[0043] The embodiments herein, along with the various features and advantages thereof, are explained more fully by reference to the non-limiting embodiments described in connection with the accompanying drawings. Descriptions of well-known components and processing techniques are omitted so as not to unnecessarily obscure the embodiments herein. The examples used herein are intended merely to facilitate an understanding of ways in which the embodiments herein can be practiced and to further enable those of skill in the art to practice the embodiments herein. Accordingly, the examples should not be construed as limiting the scope of the embodiments herein.

[0044] The embodiments herein disclose methods and systems that deploy deep neural network (DNN) models in a device by identifying redundancies in the structure of the DNN models in the device, and efficiently preloading / loading / unloading layers of the DNN models based on dependencies between the DNN models in the device. Embodiments include identifying redundancies in the structure of the DNN models by determining layers that are present in multiple DNN models. Embodiments include determining reference count values related to all layers in all DNN models. Embodiments include traversing each layer of each DNN model and initializing reference count values related to each layer. Embodiments include incrementing reference count values related to layers in the DNN if the layer is traversed more than once (in other DNN models). Embodiments include identifying layers as redundant contributors if the reference count values related to the layer have been incremented.

[0045] Embodiments include determining dependencies between DNN models within an application or across multiple applications. Dependencies existing between at least two DNN models can be determined by deciding whether the at least two DNN models are executed simultaneously, independently, or sequentially by at least one application. Embodiments herein include preloading different layers of DNN models based on redundancies of the DNN models and dependencies of the DNN models across multiple applications or within an application.

[0046] Reference is now made to the drawings, and more specifically to Figures 2 to 10BThe application is illustrated by and through the drawings in which like reference numerals indicate corresponding or like parts; various embodiments are shown

[0047] Figure 2 The various units of the device 200 configured to deploy the DNN models in the device 200 based on the redundancy in the structure of the DNN models and the dependencies between the DNN models, according to an embodiment of the disclosure, are shown. As shown, the device 200 includes a model redundancy analyzer 201, a model dependency analyzer 202, a model pre-loader 203, at least one processing unit 204, a memory 205, and a display 206. Examples of the device 200 include, but are not limited to, a smartphone, a tablet, a laptop, a wearable device, a personal computer (PC), an Internet of Things (IoT) device, or any other embedded device. In an embodiment, the model redundancy analyzer 201, the model dependency analyzer 202, the model pre-loader 203, and the at least one processing unit 204 can be implemented as at least one hardware processor. Figure 2

[0048] The at least one processing unit 204 includes at least one of a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), and a neural processing unit (NPU). The memory 205 can store the DNN models loaded on or unloaded from the at least one processing unit 204. The memory 205 can store information related to at least one of the details of the layers of the DNN models, the structure of the DNN models, the memory available in the at least one processing unit 204, the redundancy between the DNN models, and the dependencies between the DNN models. The information can be acquired by at least one of the model redundancy analyzer 201, the model dependency analyzer 202, and the model pre-loader 203 to manage the deployment of the DNN models in the at least one processing unit 204.

[0049] A plurality of applications can be installed in the device 300 and one or more applications can utilize the at least one DNN model to perform operations related to the applications. Each DNN model includes a plurality of layers. The deployment of the DNN models includes loading / pre-loading the DNN models in the device 200 and unloading the DNN models from the device 200. Efficient loading, unloading, and pre-loading can improve the memory usage efficiency and latency of the device 200.

[0050] ​The model redundancy analyzer 201 can identify redundancies in the structure (architecture) of the DNN models. In one embodiment, the model redundancy analyzer 201 can identify redundancies in the structure (architecture) of the DNN models by comparing each DNN model with other DNN models. For example, if the device 200 includes five DNN models, the model redundancy analyzer 201 can compare each DNN model with the other four DNN models. The comparison involves determining reference count related to each layer of each DNN model. The model redundancy analyzer 201 can iterate through each layer of each DNN model. When iterating through the layers of a DNN model for the first time, the model redundancy analyzer 201 can initialize the reference count value related to the layers of the DNN model.

[0051] While iterating through the first DNN model, if the model redundancy analyzer 201 determines that a layer in the first DNN model has been iterated through (e.g., the second DNN model), the model redundancy analyzer 201 increments the reference count value related to the layer in the first DNN model. In an embodiment, the model redundancy analyzer 201 determines whether a layer of a DNN model has been iterated through based on the details of the layer. In an example, the details include the layer type (e.g., convolution) and other parameters (e.g., kernel size, stride, padding, etc.).

[0052] If the details of the layer of the first DNN model are unique and do not match the details of any other layer in any other DNN model that has been iterated through, the layer is considered unique. On the other hand, if the details of the layer in the first DNN model match the details of a layer in the second DNN model or any other DNN model that has been iterated through, the layer in the first DNN model and the second DNN model and other DNN models are all considered to contribute to redundancies in the structure of the first DNN model, the second DNN model, and the other DNN models. The model redundancy analyzer 201 can increment the reference count value related to the layers that contribute to the redundancies.

[0053] In one embodiment, the details of a layer include parameters related to the layer, which can be the learned weight and bias values in the operation. The structure of a DNN model can represent or include a combination of operations in the network. In an example, the structure includes a 3x3 convolution block, followed by a first rectified linear unit (ReLU) operation block. The first ReLU operation block is followed by a 1x1 convolution block and a second ReLU operation block.

[0054] Once the reference count values related to all layers of all DNN models are determined, the structure of each DNN model is categorized into classes, i.e., common region and specific region. In a DNN model, a layer is considered to fall in the common region of the DNN model if it is included in the structure of at least one DNN model. Similarly, a layer falls in the specific region of a DNN model if it is a unique layer of the DNN model. The categorization operation enables to determine the contributors of redundancy in the structure of the DNN models. The model redundancy analyzer 201 can generate an optimized model data as a tree, where the root node includes the layer with the highest reference count. The layers in the subsequent levels have smaller reference count values. The leaf nodes consist of the layers with the smallest reference count values and represent the unique layers of the DNN models that fall in the specific region of the structure of the individual DNN models.

[0055] The model dependency analyzer 202 can determine the dependencies between the DNN models to be deployed in the device 200. The dependencies indicate the order in which the DNN models are executed by the applications and the order in which the different layers of the DNN models, especially the layers in the specific region of the individual DNN models, are executed loaded to or unloaded from the processing unit 204.

[0056] The dependencies between the DNN models can exist in an application, where at least two DNN models are executed sequentially or in parallel by the application. The dependencies can span one or more applications, where at least two DNN models are executed sequentially or in parallel by the individual applications. If at least two DNN models run independently, there is no dependency between the different DNN models.

[0057] The model dependency analyzer 202 generates a model dependency graph to illustrate the dependencies between the DNN models. In an embodiment, the dependencies can be determined using information provided by the applications that execute the DNN models. The information specifies whether the DNN models are sequential, parallel, or independent. The nodes of the model dependency graph represent the DNN models and the edges connecting the nodes represent the order in which the DNN models are executed by the applications. The type of the edge connecting the nodes of the model dependency graph specifies the order in which the DNN models are executed by the applications. The type of the edge specifies whether the DNN models are to be loaded simultaneously or sequentially. In an embodiment, if there is a directed edge connecting two DNN models, the DNN model represented by the source node (the source of the directed edge) is executed first and the DNN model represented by the destination node is executed second. Thus, the directed edge specifies that the DNN models are executed sequentially. In another embodiment, if there is an undirected edge between two DNN models, the DNN models connected by the undirected edge are executed in parallel. In yet another embodiment, if there is no edge between two DNN models, the DNN models are executed independently of each other. The model preloader 203 can use the model dependency graph to determine the layers of the DNN models that need to be preloaded.

[0058] The model preloader 203 preloads the layers of the DNN model based on the redundancy in the structure of the DNN model and the dependencies across multiple applications or between DNN models within an application. The model preloader 203 fetches the optimized model data, the model dependency graph, and the available memory in the at least one processing unit 204 for preloading the layers of the DNN model. Preloading reduces the latency. The model preloader 203 ensures that the layers in the common region are not preloaded multiple times.

[0059] In an embodiment, the layers of the DNN model contributing to the redundancy can be preloaded. The DNN model layers contributing to the redundancy can be assigned a priority based on the reference count value pertaining to the layer. If the reference count value pertaining to a certain layer in the DNN model is higher, the priority assigned is higher. Similarly, if the reference count value pertaining to a layer in the DNN model is lower, the priority assigned is lower. The layers of the DNN model contributing to the redundancy can be preloaded in the at least one processing unit 204 based on the priority.

[0060] Based on the model dependency graph, the optimized model data, and the available memory in the at least one processing unit 204, the model preloader 203 can determine the layers of the DNN model to be loaded or unloaded based on the available memory in the at least one processing unit 204. The model preloader 203 can load / unload the portions of the structure of the DNN model (common region and / or specific region) in the memory of the at least one processing unit 204. The loader 203 can determine the portions of the structure of the DNN model to be kept loaded / unloaded while unloading / loading the DNN model based on the memory shared by the at least one processing unit 204.

[0061] Figure 2 Exemplary units of the device 200 are shown, but it should be understood that other embodiments are not limited thereto. In other embodiments, the device 200 can include a fewer or greater number of units. Further, the labels or names of the units are for illustrative purposes only and do not limit the scope of the embodiments. One or more components can be combined together to perform the same or substantially similar functions in the device 200.

[0062] Figure 3A is a flowchart 300 illustrating a method for deploying DNN models in the device 200 based on the redundancy in the structure of the DNN models and the dependencies between the DNN models, according to an embodiment of the present disclosure. At step 301, the method includes identifying the redundancy in the structure of each DNN model based on the presence of the same layers in different DNN models in the device 200. In an embodiment, each DNN model can be compared with other DNN models by determining the reference count value pertaining to all the layers of each DNN model and comparing the reference count values.

[0063] Embodiments include traversing layers of the DNN models and initializing reference count values related to the layers of the DNN models on a first pass through the layers of the DNN models. The reference count values related to the layers of the DNN models are incremented if it is determined that the layers of the DNN models have been traversed previously when traversing different DNN models. Embodiments include determining whether a layer of a DNN model has been traversed based on details of the layer. Embodiments include identifying that a layer of a DNN model contributes to redundancy if the reference count value related to the layer is incremented.

[0064] Based on the reference count values related to all layers of all DNN models, embodiments include categorizing the structure of the DNN models into two categories, common region and specific region. Layers that fall in the common region of the DNN models contribute to redundancy. Layers that fall in the specific region of the DNN models are unique to the DNN models. Embodiments include generating optimization model data that shows the reference count values of the layers of different DNN models. In embodiments, the optimization model data is a tree, where the layers in the root node have the highest reference count. The leaf nodes of the tree consist of layers with the smallest reference count values and represent the unique layers in the individual DNN models.

[0065] At step 302, the method includes determining dependencies between the DNN models based on the execution order of the application or the plurality of applications. Embodiments include determining dependencies between the DNN models to confirm the order in which the application / plurality of applications executes the particular DNN models. The order specifies the order in which different layers of the DNN models are to be loaded for execution by the at least one processing unit 204 and the order in which the different layers of the DNN models are to be unloaded from the at least one processing unit 204 after execution is complete.

[0066] Based on the dependencies between the DNN models, embodiments include determining whether the at least two DNN models are executed by the application sequentially or in parallel. If the at least two DNN models are executed sequentially, the loading (or unloading if there is not enough memory available in the at least one processing unit 204) of the at least two DNN models follows the order of execution. If the at least two DNN models are executed in parallel, the at least two DNN models are loaded simultaneously, where multiple loading of layers in the common region of the at least two DNN layers is avoided. If the at least two DNN models run independently, there are no dependencies between the different DNN models.

[0067] Embodiments include generating a model dependency graph that shows dependencies between DNN models. Nodes of the model dependency graph represent DNN models, and edges connecting the nodes represent an order in which the DNN models are executed. The type of edge connecting the nodes of the model dependency graph specifies the order in which the DNN models are executed. If there is a directed edge connecting two DNN models, the DNN model representing the source node is executed first, and then the DNN model representing the destination node is executed. If there is an undirected edge between two DNN models, the DNN models connected by the undirected edge are executed in parallel.

[0068] At step 303, the method includes preloading, loading and unloading layers of the DNN models based on the redundancies in the structure of the DNN models and dependencies between the DNN models. Embodiments include assigning a priority to the layers of the DNN models that contribute to the redundancies based on reference count values related to the layers of the DNN models. The priority assigned to the layers of the DNN models is directly proportional to the reference count values related to the layers in the DNN models. Embodiments include preloading the layers of the DNN models in the at least one processing unit 204 based on the assigned priority, wherein the preloaded layers contribute to the redundancies of the structure of the DNN models. Embodiments assign the priority because the at least one processing unit 204 can not have sufficient memory to keep all the layers of all the DNN models preloaded at all times.

[0069] Embodiments include determining the layers of the DNN models to be loaded or unloaded based on the available memory (availability of memory capacity determined by the at least one processing unit 204). Embodiments include loading the layers in the common region and / or the specific region of the structure of the DNN models into the memory of the at least one processing unit 204. Embodiments include unloading the layers in the common region and / or the specific region of the structure of the DNN models if the memory of the at least one processing unit 204 is insufficient. The embodiments include determining to keep the parts of the structure of the DNN models loaded / unloaded while unloading / loading the DNN models based on the memory shared by the at least one processing unit 204.

[0070] In embodiments, the above described method can be performed by the processing unit 204 of the device 200.

[0071] The various actions in flowchart 300 can be performed in the order presented, in a different order or simultaneously. Further, in some embodiments, some actions listed in FIG. 3 can be omitted.

[0072] Figure 3B is a flowchart 310 illustrating a method for managing DNN models according to embodiments of the present disclosure.

[0073] Reference Figure 3BAt step 311, the processing unit 204 can extract information associated with each of the plurality of DNN models. The information associated with each of the plurality of DNN models can include parameters and structure of each of the plurality of DNN models. The parameters related to the layers can be learning weights and bias values within the operations performed within the device.

[0074] At step 313, the processing unit 204 can identify, from the information, common information common to the plurality of DNN models.

[0075] At step 315, the processing unit 204 can separate and store the common information to a designated location in the device.

[0076] At step 317, the processing unit 204 can control at least one of the plurality of DNN models to access the common information. In an embodiment, the processing unit 204 can preload a subset of the common information based on a preloadable storage capacity of the device.

[0077] In an embodiment, the processing unit 204 can determine, among the plurality of DNN models, a dependent model associated with each application installed on the device. The dependent model can include at least one of: a model that needs to run simultaneously with other models among the plurality of DNN models, and a model that has a fixed execution order related to other models among the plurality of DNN models. The model that has a fixed execution order related to other models among the plurality of DNN models can include at least one of: a model to be executed serially related to other models among the plurality of DNN models, and a model to be executed in parallel with other models among the plurality of DNN models.

[0078] Figure 4 is an example illustrating redundancy identification among two DNN models according to an embodiment of the present disclosure. As shown in Figure 4 consider two DNNs, i.e., network 1 and network 2, to be deployed on the device 200. Consider that the network 1 and the network 2 have been obtained through transfer learning, where a pre-trained DNN model is trained using different datasets to obtain the DNN models, i.e., the network 1 and the network 2. Thus, the network 1 and the network 2 can have structural similarity, which contributes to redundancy in the structure of the network 1 and the network 2. The nodes of the network 1 and the network 2 represent the layers of the DNN models.

[0079] The model redundancy analyzer 201 can traverse the layers of network 1 and network 2 to determine the layers that exist in both DNNs (i.e., network 1 and network 2). The layers that exist in both (a portion of) network 1 and network 2 are identified as contributing to redundancy. Consider that the model redundancy analyzer 201 first traverses the layers of network 1, followed by traversing the layers of network 2. When a particular layer is traversed for the first time, the model redundancy analyzer 201 initializes a reference count associated with the particular layer. Consider that the model redundancy analyzer 201 is traversing network 1 for the first time, all the layers of network 1 are initialized during the traversal.

[0080] Once the traversal of network 1 is complete, the model redundancy analyzer 201 can begin traversing the layers of network 2. The model redundancy analyzer 201 can increment the reference count value associated with those layers that exist in both network 1 and network 2. The model redundancy analyzer 201 can increment the reference count value based on the parameters associated with the layer and the weights of the layer when those layers are determined to be the same. The reference count value associated with the remaining layers of network 2 is initialized. The layers for which the associated reference count has been incremented are identified as contributing to redundancy (marked in green color).

[0081] Thereafter, the structures of network 1 and network 2 are categorized based on the reference count value to generate the optimized model data. The structures are categorized into common region (contributing to redundancy) and specific region (non-redundant). Categorizing the structures of the network (DNN model) into one of the common region and specific region enables optimal utilization of the storage of the device 200. The embodiments prevent redundant storage of data and are independent of specific chipsets or processors. The model redundancy analyzer 201 allows the deployment of the network on any chipset or processing unit.

[0082] Figure 5 is an example illustrating the generation of the optimized model data indicating the redundant and non-redundant layers in the four DNN models utilized by the camera application installed in the device 200, in accordance with an embodiment of the present disclosure. The camera application can operate in two modes (i.e., first mode and version mode). When the camera application operates in the first mode, the first classifier and the first detector are executed. When the camera application operates in the second mode, the second classifier and the second detector are executed. The user can switch the mode of operation while using the camera application, thereby executing the relevant classifier and detector.

[0083] The model redundancy analyzer 201 can traverse the layers of the first classifier, the first detector, the second classifier, and the second detector to determine the layers that exist in all four DNN models. The layers that exist in at least two DNN models can be considered to contribute to redundancy in the structures of the first classifier, the first detector, the second classifier, and the second detector. The layers that fall in the specific region of the structures of the first classifier, the first detector, the second classifier, and the second detector are unique layers.

[0084] As Figure 5 shown, consider that there are 242 layers in the first classifier, the first detector, the second classifier, and the second detector. The optimization model data is presented as a tree, with layers having a higher reference count acting as parents to layers having a lower reference count. Layers at leaf nodes are unique to the respective DNN model.

[0085] Consider that layers 0-158 exist in the first classifier, the first detector, the second classifier, and the second detector. These layers contribute to redundancy in the structure of the four DNNs. Each of the layers 0-158 has a reference count of 4, as layers 0-158 exist in the first classifier, the first detector, the second classifier, and the second detector. Layers 159-217 exist in the first classifier and the second classifier. Layers 159-217 have a reference count of 2. Layers 159-189 exist in the first detector and the second detector. Layers 159-189 have a reference count of 2. The remaining layers are non-redundant, being unique to the respective DNN model. Layers 218-219 are unique to the first classifier, layers 218-219 are unique to the second classifier, layers 190-235 are unique to the second detector, and layers 190-242 are unique to the first detector. The unique layers have a reference count of 1, being leaf nodes.

[0086] It can be noted that 159-189 exist in the first classifier, the first detector, the second classifier, and the second detector. Layers 218-219 exist in the first classifier and the second classifier. Layers 190-235 exist in the first detector and the second detector. However, the contents of these layers are different, and thus are not considered to be the same. If these layers were considered to be the same, then the reference count values pertaining to these layers should have been incremented and placed in a parent level node (relative to the current level).

[0087] The first classifier, the first detector, the second classifier, and the second detector thus share their respective structures as there are layers in the common region. The first classifier and the second classifier share 90% of the structure, i.e. 90% of the layers of the first classifier are present in the second classifier. The first detector and the second detector share 70% of the structure, i.e. 70% of the layers of the first detector are present in the second classifier. The model redundancy analyzer 201 allows retraining of the structure with a new dataset. If the layers of the first classifier have been loaded and the user performs a mode switch, which requires loading of the second classifier, the embodiment does not require loading of the entire structure of the second classifier. Instead, only the 10% of the layers unique to the second classifier need to be loaded. Thus, there is no need to load the layers in the common region (layers not present in the leaf nodes) when running the DNN model. If the previously loaded DNN model shares the structure with the currently executed DNN model, only the unique layers of the DNN model need to be loaded. Thus, the optimized model data allows visualizing the redundancy in the structure of the DNN model, which can be used for efficient preloading.

[0088] Figure 6 is an example illustrating loading of DNN models in different processing units of the device 200, according to an embodiment of the disclosure. Consider that four DNN models (i.e. model 1, model 2, model 3, and model 4) are executed by three applications (application 1, application 2, and application 3) installed in the device 200. Consider that the application 1 executes the model 1 and the model 2, the application 2 executes the model 3, and the application 3 executes the model 4. The device 200 comprises four processing units, i.e. DSP, NPU, CPU, and GPU. The loading of the DNN models on the memory of the four processing units is further based on the model dependency graph.

[0089] The model dependency graph illustrates the dependencies between the DNN models based on the type of edges connecting the nodes (DNN models) of the model dependency graph. The edges of the model dependency graph specify whether the DNN models should be loaded in parallel or sequentially. If there is a directional edge between two DNN models, the DNN models are executed sequentially. As shown in the example in Figure 6 As shown in the example in, there is a directional edge between the model 1 and the model 2, with the model 1 acting as the source node and the model 2 acting as the destination node. The execution of the model 2 is after the execution of the model 1.

[0090] The model 1 and the model 2 are independent of the model 3 and the model 4. The model 3 is independent of the model 1, the model 2, and the model 4. The model 4 is independent of the model 1, the model 2, and the model 3. Thus, there is no edge between the model 1 and the model 3, between the model 1 and the model 4, between the model 2 and the model 3, between the model 2 and the model 4, and between the model 3 and the model 4. The model dependency graph is used to manage the loading / unloading of the DNN models in the processing units.

[0091] Figure 7is an example illustrating generation of a model dependency graph indicating dependencies between four DNN models utilized by a camera application installed in the device 200, according to an embodiment of the disclosure. The camera application can operate in a first mode and a version mode. When operating in the first mode, the first classifier and the first detector are executed. When operating in the second mode, the second classifier and the second detector are executed. When there is a mode switch, the relevant classifier and detector are executed.

[0092] The model dependency analyzer 202 can determine dependencies between the first classifier, the first detector, the second classifier, and the second detector. The dependencies exist between the DNN models executed by the camera application. The first detector and the first classifier are sequentially executed. In the first mode, the first detector is executed first, followed by the first classifier. The edges of the model dependency graph specify the order in which the DNN models should be loaded. Since the first detector and the first classifier are sequentially executed, the first classifier is loaded after the first detector. Therefore, there is a directed edge between the first detector and the first classifier, with the first detector representing the source node and the first classifier representing the destination node.

[0093] When there is a mode switch from the first mode to the second mode, the second detector and the second classifier are executed. The second detector and the second classifier are sequentially executed, i.e., the second detector is executed first, followed by the second classifier. Since the second detector and the second classifier are sequentially executed, the second classifier is loaded after the second detector. Therefore, there is a directed edge between the second detector and the second classifier, with the second detector representing the source node and the second classifier representing the destination node.

[0094] The first detector and the first classifier are executed independently of the second detector and the second classifier. Therefore, there is no dependency between the first detector and any of the second detector and the second classifier. Similarly, there is no dependency between the first classifier and any of the second detector and the second classifier. Therefore, the model dependency graph consists of two model dependency subgraphs.

[0095] Figure 8is a use case scenario showing sequential execution of a detector DNN model and a classifier DNN model for detecting objects in a region of interest (ROI) of a camera application captured media and classifying the detected objects, according to an embodiment of the disclosure. Consider a frame is captured using a camera application. In both modes of the camera application, i.e., scene detector mode and Bixby second mode, the detector model (first or second) is executed on the frame to obtain at least one ROI in the frame. Consider that three ROIs have been obtained, where each ROI includes at least one detected object. After the ROIs are detected, the classifier model (first or second) is executed on each ROI to classify the objects in each of the three ROIs.

[0096] A series of DNN inferences can be obtained for the frame. A single detector inference is followed by three classifier inferences (one for each ROI). The first classifier is dependent on the first detector and needs to be loaded after the first detector model has been loaded. Similarly, the second classifier is dependent on the second detector model and needs to be loaded after the second detector model has been loaded. In addition to the above, the embodiments include collecting information about the order in which the classifiers and detectors are to be executed.

[0097] The embodiments allow reusing the input / output (IO) and internal memory previously used for executing the detector model to execute the classifier model. The reuse is supported due to the information obtained using the model dependency graph. For example, using the model dependency graph, the embodiments can determine that the classifier is executed after the detector is executed. The detector and classifier models are therefore not loaded simultaneously, and the non-redundant portion of the classifier is loaded if there is redundancy in the structure of the detector and classifier models. This enables improving the memory usage efficiency and latency of the device 200. It can be noted that the detector and classifier models can be added simultaneously, but since the detector and classifier models are sequentially executed, the memory can be reused.

[0098] Figure 9 is an example showing preloading of DNN models by the model preloader 203 in different processing units of the device 200, according to an embodiment of the disclosure. As Figure 9As shown, the model pre-loader 203 obtains the model dependency graph and the optimized model data. The model pre-loader 203 determines the available memory in each processing unit 204. Based on the model dependency graph, the optimized model data, and the available memory, the model pre-loader 203 can select the layers of the DNN model that need to be pre-loaded. The model pre-loader 203 can determine whether to load / unload portions of the structure of the DNN model in the memory of each processing unit 204, determine which portions of the structure of the DNN model to keep loaded / unloaded when unloading / loading the DNN model, and share memory between the DNN models loaded in the memory of the processing units 204.

[0099] Figure 10A and Figure 10B is a use case scenario showing the pre-loading / loading / unloading of the DNN model used by the camera application based on the model dependency graph and the optimized model data according to an embodiment of the disclosure. The operation mode of the camera application is the first mode and the second mode. As shown in Figure 10A and Figure 10B The model dependency graph and the optimized model data related to the execution of the DNN model by the camera application are used to determine the layers to load when executing the model, the layers to unload when the model execution is completed, and the layers of the model that need to be kept loaded after the model execution is completed. The gray blocks (labeled B) do not need to be unloaded if sufficient memory is available to keep them loaded. Otherwise, these blocks can be deleted or unloaded to save memory so that other required blocks can be loaded.

[0100] Consider that the NPU and the DSP share their respective internal memories. Consider that the second mode is initially used. Thus, the second detector and the second classifier can be loaded in and unloaded from the DSP and the NPU. The second detector is first loaded onto the DSP, and based on the redundancies identified using the optimized model data, after executing the second detector, a specific region of the second classifier structure (consisting of non-redundant layers) is loaded onto the NPU. The second detector and the second classifier share a common region (layers 0-158). Since the NPU and the DSP share their respective memories, there is no need to load the redundant layers again.

[0101] In another scenario, the second detector can be pre-loaded onto the DSP, and when the camera application switches to the second mode, after executing the second detector (the second detector has detected the object captured by the camera), a specific region of the second classifier can be loaded onto the NPU. If there is sufficient available space in the memory of the DSP and / or the NPU, during the loading of the specific region of the second classifier, there is no need to unload the specific region of the second detector.

[0102] In yet another scenario, the second classifier can be preloaded onto the NPU and when the camera application switches to the second mode, the specific region of the second detector can be loaded onto the DSP. If there is sufficient available space in the memory of the DSP and / or NPU, then during the loading of the specific region of the second detector, there is no need to unload the specific region of the second classifier.

[0103] When using the first mode, the first detector and the first classifier can be loaded in or unloaded from the DSP and NPU. The first detector is first loaded onto the DSP and based on the redundancies identified using the optimization model data, the specific region of the first classifier structure (consisting of non-redundant layers) is loaded onto the NPU after the first detector execution. Based on the optimization model data, the specific region of the first detector is added. This is because the second classifier and the first detector share a common region (layers 0-217).

[0104] In another scenario, the first detector can be preloaded onto the DSP and when the camera application switches to the second mode, the specific region of the first classifier is loaded onto the NPU after the first detector execution. If there is sufficient available space in the memory of the DSP and / or NPU, then during the loading of the specific region of the first classifier in the NPU, there is no need to unload the specific region of the first detector from the DSP. It can be noted that the first detector and the first classifier share a common region (layers 0-158).

[0105] In yet another scenario, the first classifier can be preloaded onto the NPU and when the camera application switches to the first mode, the specific region of the first detector can be loaded onto the DSP. If there is sufficient available space in the memory of the DSP and / or NPU, then during the loading of the specific region of the first detector in the DSP, there is no need to unload the specific region of the first classifier from the NPU.

[0106] Embodiments allow for improved memory utilization during preloading. Embodiments help in preloading multiple DNN models while using slightly higher memory required for loading a single DNN model.

[0107] Embodiments disclosed herein can be implemented by at least one software program running on at least one hardware device and performing network management functions to control network elements. Figure 2 The network elements shown in FIG. 1 include blocks that can be at least one of hardware devices or a combination of hardware devices and software modules.

[0108] The embodiments disclosed herein describe methods and systems for deploying deep neural network (DNN) models in a device based on redundant layers in different DNN models and dependencies between the DNN models. It is therefore understood that the scope of protection is extended to such a program and in addition to a computer readable means having a message therein, such computer readable storage apparatus also includes the program codes means for implementing one or more steps of the method when the program runs on a server or a mobile device or any suitable programmable device to produce the apparatus. The method is implemented in the preferred embodiments through or together with software programs written in, for example, the Very high speed integrated circuit Hardware Description Language (VHDL) and other programming languages such as assembly language, C, etc. The method is implemented by using one or more VHDL or several software modules being executed on at least one hardware device. A hardware device can be any kind of portable device that can be programmed. The device can also include, for example, the following means: a hardware means such as an application-specific integrated circuit (ASIC), or a combination of hardware and software means, such as an ASIC and a field-programmable gate array (FPGA), or at least one microprocessor and at least one memory to which at least one software module is loaded. The embodiments of the method described herein can be implemented partly in hardware and partly in software. Alternatively, the disclosure can be implemented, for example, using a plurality of central processing units (CPUs) on different hardware devices.

[0109] The drawings and the foregoing description give examples of embodiments. Those skilled in the art will appreciate that one or more of the described elements can well be combined into a single functional element. Alternatively, certain elements can be split into multiple functional elements. Elements from one embodiment can be added to another embodiment. For example, the order of processes described herein can be changed and not every process described is necessary. The scope of the

[0110] Furthermore, the acts performed as the flowsheets do not have to be performed in the order described; nor do they necessarily all have to be performed. Also, those acts that are not dependent on other acts can be performed in parallel with the other acts. The scope of the embodiments is not intended to be limited to just the specifically set out examples. Numerous alternatives (e.g., variations in sizes, dimensions, names, and numbers of sections, etc.) can be resorted to without departing from the scope of the embodiments. The scope of the embodiments is at least as broad as that of the claims.

[0111] Benefits, other advantages, and solutions to problems have been described above with regard to specific embodiments. However, the benefits, advantages, solutions to problems and any component(s) that can cause any benefit, advantage, or solution to occur or become more pronounced are not to be construed as a critical, required, or essential feature or component of any or all the claims. The claims should not be construed as including any

[0112] Although the subject matter has been described in language specific to structural features, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features described. One skilled in the art will readily recognize that the equivalent structures and functions can be substituted for the specific ones described. Apparatus and processes suitable for the practice of the present subject matter have been described above. The descriptions are intended to be illustrative, not limiting. Thus, it will be apparent to those skilled in the art that modifications can be made to the information collection and analysis system and still be within the scope of the various embodiments. Furthermore, it is contemplated that a "generic" system can be designed that can be modified for use in accordance with any one or more of the various embodiments.

Claims

1. A method of managing deep neural network (DNN) models on a device, the method comprising: extracting information associated with each of a plurality of DNN models; determining reference count values associated with layers of the plurality of DNN models; assigning a priority to each of the layers of the plurality of DNN models based on the reference count values; identifying, from the information, common information shared across the plurality of DNN models based on the reference count values; separating and storing the common information to a designated location in the device; controlling at least one of the plurality of DNN models to access the common information; and preloading a classification of layers of the plurality of DNN models into the common information based on the priority assigned to each of the layers of the plurality of DNN models. The information associated with each of the plurality of DNN models comprises parameters and structures of each of the plurality of DNN models.

2. The method of claim 1, wherein, 3. The method of claim 1, further comprising: preloading a subset of the common information based on a pre-loadable memory capacity of the device.

4. The method of claim 3, further comprising: determining, among the plurality of DNN models, dependent models associated with each application installed on the device, wherein the dependent models comprise at least one of: models that need to run simultaneously with other models of the plurality of DNN models, and models that have a fixed execution order relative to other models of the plurality of DNN models. The models that have a fixed execution order relative to other models of the plurality of DNN models comprise at least one of:

5. The method of claim 4, wherein, models to be executed serially relative to other models of the plurality of DNN models; and models to be executed in parallel relative to other models of the plurality of DNN models.

6. An apparatus for managing deep neural network (DNN) models, the apparatus comprising: a memory; and a processor configured to: determine reference count values associated with layers of at least two DNN models, assign a priority to each of the layers of the at least two DNN models based on the reference count values, identify redundancies in structures of the at least two DNN models based on a presence of at least one layer in the at least two DNN models, determine dependencies between the at least two DNN models that specify a mode of execution of the at least two DNN models, and deploy the at least two DNN models based on the redundancies and dependencies of the at least two DNN models, wherein the processor is configured to preload layers of the at least two DNN models that contribute to the redundancies based on the priority assigned to each of the layers of the at least two DNN models. The deployment of the at least two DNN models is further performed based on availability of a capacity of the memory. The presence of the at least one layer in the at least two DNN models is detected based on at least one reference count value associated with the at least one layer in the structures of the at least two DNN models, 7. The apparatus of claim 6, wherein, ​ 8. The apparatus of claim 6, wherein, ​ wherein the reference count values are initialized during an initial traversal of structures of the at least two DNN models.

9. The apparatus of claim 8, wherein, The at least one reference count value associated with the at least one layer is incremented when traversing the at least one layer in the at least two DNN models, wherein the at least one layer in the at least two DNN models contributes to redundancy in the structures of the at least two DNN models.

10. The apparatus of claim 9, wherein, The processor is configured to determine that a layer contributes to redundancy in the structures when a reference count value associated with the layer is incremented.

11. The apparatus of claim 9, wherein, The processor is configured to preload at least one layer that contributes to the redundancy prior to execution of the at least two DNN models.

12. The apparatus of claim 6, wherein, The mode of execution of the at least two DNN models comprises: sequentially executing the at least two DNN models by the application; and parallel execution of the at least two DNN models by the application.

13. The apparatus of claim 6, wherein, The processor is configured to preload layers of the at least two DNN models based on redundancy and dependencies of the at least two DNN models.

14. The apparatus of claim 6, wherein, The processor is configured to determine reference count values associated with all layers of the at least two DNN models and compare each of the at least two DNN models based on the reference count values.

Images