Electronic device for processing neural network model using multiple processors and operating method thereof

Abstract

A method of processing a neural network model by using a plurality of processors includes assigning at least one slice to each of a plurality of layers included in the neural network model, assigning each of the plurality of layers to the plurality of processors for processing each of the at least one slice based on respective processing times of the plurality of processors, and processing the neural network model by using the plurality of processors based on a result of the assigning.

Description

TECHNICAL FIELD

[0001] The disclosure relates to an electronic device for processing a neural network model using a plurality of processors and a method of operating the electronic device. BACKGROUND

[0002] An electronic device can use a deep learning technique based on a neural network model to perform face recognition, speech recognition, image processing, etc., and provide a result thereof as an output.

[0003] An electronic device can process a neural network model by using a plurality of processors (e.g., a central processing unit (CPU), a graphics processing unit (GPU), a neural processing unit (NPU), a digital signal processor (DSP), etc.). The electronic device can process a neural network model using a plurality of processors by assigning a plurality of layers constituting a neural network model to the plurality of processors.

[0004] However, processing speed and accuracy can differ depending on the performance of each processor and the characteristics of each layer. Accordingly, there is a need for a method of allocating layers of a neural network model to be processed by each processor based on the processing capability of the plurality of processors and the characteristics of the layers. SUMMARY

[0005] SOLUTION TO PROBLEM

[0006] The disclosure provides an electronic device for processing a neural network model using a plurality of processors and a method of operating the electronic device.

[0007] The disclosure also provides a computer-readable recording medium having recorded thereon a program for executing the method on a computer. The technical problem to be solved is not limited to the above-described technical problem, and other technical problems can exist. BRIEF DESCRIPTION OF DRAWINGS

[0008] The above and other aspects, features, and advantages of certain embodiments of the disclosure will be more apparent from the following description taken in conjunction with the accompanying drawings, in which:

[0009] Figure 1 is a diagram illustrating an example of processing a neural network model in an electronic device according to an embodiment;

[0010] Figure 2 is a diagram illustrating an example of allocating layers of a neural network model to at least one slice according to an embodiment;

[0011] Figure 3 is a block diagram for describing an internal configuration of an electronic device according to an embodiment;

[0012] Figure 4is a block diagram for describing an internal configuration of an electronic device according to an embodiment;

[0013] Figure 5 is a flowchart of a method of processing a neural network model by using a plurality of processors according to an embodiment;

[0014] Figure 6 is a flowchart of a method of allocating slices to a plurality of processors according to an embodiment;

[0015] Figure 7 is a diagram showing an example of allocating processors to slices according to an embodiment;

[0016] Figure 8 is a diagram showing an example of allocating memory in a layer according to an embodiment;

[0017] Figure 9 is a flowchart of a method of allocating memory for I / O data of a layer according to an embodiment;

[0018] Figure 10 is a diagram showing an example of identifying blobs within a layer according to an embodiment;

[0019] Figure 11 is a diagram showing an example of allocating memory to blobs of a neural network model including blobs within a layer according to an embodiment; and

[0020] Figure 12 is a diagram showing an example of processing a neural network model by a plurality of processors according to an embodiment. DETAILED DESCRIPTION

[0021] Additional aspects will be set forth in part in the description which follows, and in part will be apparent from the description, or can be learned by practice of the presented embodiments of the disclosure.

[0022] According to a first aspect of the disclosure, there is provided a method of processing a neural network model by using a plurality of processors, the method including: allocating at least one slice to each layer among a plurality of layers included in the neural network model; allocating each layer among the plurality of layers to a processor among the plurality of processors for processing at least one slice allocated to each layer based on a processing time of each of the plurality of processors; and processing the neural network model by using the plurality of processors based on a result of the allocation, wherein the processing time includes a switching time taken by each of the plurality of processors to receive data for processing a current slice from a previous processor among the plurality of processors which processed a previous slice.

[0023] According to a second aspect of the disclosure, there is provided an electronic device for processing a neural network model, the electronic device comprising: a memory configured to store the neural network model; and at least one processor configured to: allocate at least one slice to each layer among a plurality of layers included in the neural network model; allocate each layer among the plurality of layers to a processor among a plurality of processors for processing the at least one slice allocated to each layer based on a processing time of each of the plurality of processors; process the neural network model based on a result of the allocation; and output a result of processing the neural network model, wherein the processing time includes a switching time that each processor among the plurality of processors takes to receive data for processing a current slice from a previous processor among the plurality of processors that processed a previous slice.

[0024] According to another aspect of the disclosure, there is provided a computer-readable recording medium having recorded thereon a program for implementing the method of the first aspect.

[0025] Hereinafter, embodiments of the disclosure will be described in detail with reference to the accompanying drawings so that those skilled in the art can easily implement the disclosure. However, the disclosure can be implemented in many different forms and should not be interpreted as limited to the embodiments set forth herein. In order to clearly illustrate the disclosure, portions unrelated to the description are omitted, and throughout the specification, like parts are denoted by like reference numerals.

[0026] Throughout the disclosure, the expression "at least one of a, b, or c" indicates only a, only b, only c; both a and b; both a and c; both b and c; all of a, b, and c; or variations thereof.

[0027] Throughout the specification, when one part "connects" to another part, this includes not only "direct connection" with another element therebetween, but also "electrical connection". In addition, unless explicitly described to the contrary, the word "comprise" and variations such as "comprises" or "comprising" will be understood to imply the inclusion of stated elements but not the exclusion of any other elements.

[0028] The functions related to artificial intelligence according to the disclosure are operated by a processor and a memory. The processor can include one or more processors. In this case, the one or more processors can be a general-purpose processor such as a CPU, an AP, and a digital signal processor (DSP), a graphics-specialized processor such as a GPU and a visual processing unit (VPU), or an artificial intelligence-specialized processor such as an NPU, respectively. The one or more processors control to process input data according to a predefined operation rule or an AI model stored in the memory. Alternatively, when the one or more processors are an artificial intelligence-specialized processor, the artificial intelligence-specialized processor can be designed to have a hardware structure specialized for processing a specific AI model.

[0029] The predefined operation rule or the AI model can be constructed through learning. Here, the construction through learning means that, when a basic AI model is trained according to a learning algorithm by using a plurality of pieces of learning data, the predefined operation rule or the AI model set to perform a desired characteristic (or purpose) will be constructed. Such learning can be performed in a device that operates the AI according to the disclosure, or can be performed through a separate server and / or system. Examples of the learning algorithm include supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but are not limited to the above examples.

[0030] The AI model can include a plurality of neural network layers. Each of the plurality of neural network layers has a plurality of weight values, and performs a neural network operation through an operation between an operation result of a previous layer and the plurality of weight values. The weight values of the neural network layers can be optimized through a learning result of the AI model. For example, the plurality of weight values can be updated so as to reduce or minimize a loss value or a cost value obtained by the AI model during a learning process. The artificial neural network can include a deep neural network (DNN), and can include, for example, a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), and a deep Q-network, but is not limited to the above examples.

[0031] Hereinafter, the disclosure will be described in detail with reference to the accompanying drawings.

[0032] Figure 1 is a diagram illustrating an example of processing a neural network model 110 in an electronic device 1000 according to an embodiment.

[0033] Referring to Figure 1 , the electronic device 1000 can process the neural network model 110 by using a processor 1300 including a plurality of processors 1310, 1320, and 1330.

[0034] According to an embodiment, the electronic device 1000 can process the neural network model 110 by using a plurality of processors via a compilation operation and a task execution operation. For example, the electronic device 1000 can perform a compilation operation to allocate tasks to be executed by the plurality of processors, and perform a task execution operation to execute the tasks using the plurality of processors according to the allocation result in the compilation operation. The compilation operation according to an embodiment can be performed by at least one of the plurality of processors.

[0035] According to an embodiment, the compilation operation can include an operation for allocating a memory for storing data for executing a task, and an operation for allocating a task to be executed by the plurality of processors.

[0036] The electronic device 1000 according to an embodiment can be implemented in various forms. For example, the electronic device 1000 described herein can include a digital camera, a smartphone, a laptop computer, a tablet PC, an e-book terminal, a digital broadcasting terminal, a personal digital assistant (PDA), a portable multimedia player (PMP), a navigation device, and an MP3 player, but is not limited thereto. The electronic device 1000 described herein can be a wearable device that can be worn by a user. The wearable device can include at least one of an accessory-type device (e.g., a watch, a ring, a bracelet, an anklet, a necklace, glasses, contact lenses, etc.), a head-mounted device (HMD), a fabric or clothing-integrated device (e.g., electronic clothing), a body-attachable device (e.g., a skin pad), and a bio-implantable device (e.g., an implantable circuit), but is not limited thereto. Hereinafter, for convenience of explanation, a case in which the electronic device 1000 is a smartphone will be described as an example.

[0037] The electronic device 1000 according to an embodiment can perform various operations using the neural network model 110. For example, the neural network model 110 can include an AI model such as a deep neural network (DNN), a recurrent neural network (RNN), and a convolutional neural network (CNN). In addition to the above examples, the neural network model 110 according to an embodiment can also include various types of AI models.

[0038] According to an embodiment, the electronic device 1000 can perform various operations such as recognizing data or generating new data by using at least one neural network model 110, and provide a result thereof to a user.

[0039] The neural network model 110 according to an embodiment can include a plurality of layers. Each layer included in the neural network model 110 can include at least one function for processing data. For example, the neural network model 110 including a CNN model can include a combination of various types of layers such as a convolution layer, a max-pooling layer, and a flattening layer.

[0040] For example, in a convolution layer, an operation for extracting feature information about input data can be performed. In a max pooling layer, an operation for extracting main data from input data can be performed. In a flattening layer, an operation for converting values of input data into 1-dimensional values can be performed. In addition to the above-described examples, the neural network model 110 can include various types of layers.

[0041] The electronic device 1000 according to an embodiment can include a plurality of processors 1310, 1320, and 1330. For example, the electronic device 1000 can include various types of processors such as a CPU, a GPU, an NPU, and a DSP. The processors can have different performance and features from each other. For example, the CPU is slower than the other processors, but shows higher accuracy and superior energy efficiency.

[0042] Table 1 below shows relative characteristics of the CPU, the GPU, the NPU, and the DSP.

[0043] Table 1

[0044]

[0045] The characteristics of the processors described in Table 1 are merely examples and can vary according to various factors such as characteristics of tasks performed by the respective processors or a current state of the processors.

[0046] According to an embodiment, in a compilation operation, one processor among the processors 1310, 1320, and 1330 of the electronic device 1000 can be assigned to certain layers included in the neural network model 110. In the electronic device 1000, based on the performance of each processor 1310, 1320, and 1330 as shown in Table 1 and the characteristics of each layer included in the neural network model 110, the processor can be assigned to each layer.

[0047] According to an embodiment, the speed and accuracy of the processor for processing a layer can differ from each other according to the characteristics of the layer. The electronic device 1000 according to an embodiment can assign a processor among a plurality of processors for processing each of the respective layers by determining which processor is suitable for processing each layer of the neural network model 110.

[0048] According to an embodiment, when assigning the processor for processing a layer, the electronic device 1000 can process the neural network model 110 by using a plurality of processors according to the result of the assignment.

[0049] Figure 2 is a diagram illustrating an example of assigning layers of the neural network model 110 to at least one slice according to an embodiment.

[0050] Figure 2The nodes shown in the middle represent some of the layers that make up the neural network model 110. According to an embodiment, the neural network model 110 can be processed in the electronic device 1000 since the results of the operations performed in each layer are transmitted to the next layer in the direction indicated by the arrows.

[0051] According to an embodiment, a plurality of layers of the neural network model 110 can be allocated to at least one slice, and the layers can be allocated to the processors in units of slices. According to an embodiment, the electronic device 1000 can allocate the layers to the processors in units of a slice including at least one layer rather than in units of one layer, thereby reducing the amount of calculation generated by the allocation operation.

[0052] According to an embodiment, the electronic device 1000 can determine at least one of the plurality of layers that make up the neural network model 110 as a slice point, thereby allocating the plurality of layers to the slices. According to an embodiment, each layer can be assigned to a different slice based on the slice point.

[0053] For example, the electronic device 1000 can sequentially determine whether each of the layers "conv2d_9" to "activation_11" is determined as a slice point. In addition, it can sequentially determine whether each of the layers "conv2d_7" to "activation_8" after the layer "activation_11" is a slice point. It can also sequentially determine whether each of the layers "average_pooling2d_1" to "activation_6" is a slice point.

[0054] According to an embodiment, the electronic device 1000 can determine the slice point based on at least one of whether each layer is a branching point of the plurality of layers, whether each layer is a point at which the plurality of layers are combined, whether each layer includes a task that can be processed by the same processor, and whether each layer includes a task that requires high accuracy. According to an embodiment, as with the various criteria described above, the current layer can be determined as a slice point based on whether it is possible to switch to a processor different from the processor of the previous layer in the current layer. In addition to the above examples, the slice point can be determined according to various criteria.

[0055] For example, since "conv2d_9" is one of the plurality of layers branching from "max_pooling2d_2", it can be determined as a slice point since it corresponds to a branching point of the plurality of layers. Accordingly, "conv2d_9" can belong to a new slice 1 210.

[0056] According to the above criteria for determining a slice point, the layers "activation_9" to "actionvation_11" are not determined as a slice point, and thus the layers "activation_9" to "actionvation_11" can be assigned to slice 1 210.

[0057] Since "conv2d_7" is one of the multiple layers branching from "max_pooling2d_2" like "conv2d_9", "conv2d_7" can also be determined as a slice point and can belong to a new slice 2 220. The layers "actionvation_7" to "actionvation_8" are not determined as a slice point, and can be assigned to slice 2 220.

[0058] Since "average_pooling2d_1" is one of the multiple layers branching from "max_pooling2d_2" like "conv2d_9", "average_pooling2d_1" is determined as a slice point and can belong to a new slice 3 230. The layer "conv2d_12" is not determined as a slice point, and can be assigned to slice 3 230.

[0059] The layer "activation_12" can be determined as a slice point because it does not include a task that can be processed by the same processor as the layers of slice 3 230, or includes a task requiring high accuracy and can belong to a new slice 4 240.

[0060] For example, when none of the processors capable of processing "activation_12" is capable of processing the layers "activation_12" and "conv2d_12" (which are layers included in slice 3 230), the processors assigned to slice 3 230 cannot process "activation_12", and thus "activation_12" can not belong to slice 3 230. According to an embodiment, when a processor is assigned to a slice, the layers belonging to the same slice can be processed by the same processor.

[0061] In another example, even when "activation_12" includes a task requiring high accuracy, "activation_12" can be determined as a slice point, so that a processor suitable for "activation_12" can be assigned without being affected by other layers.

[0062] Since "conv2d_6" is one of the plurality of layers branching from "max_pooling2d_2" like "conv2d_9", "conv2d_6" can also be determined as a slicing point and can belong to a new slice 5 250. Layer "activation_6" is not determined as a slicing point and can be assigned to slice 5 250.

[0063] According to an embodiment, each slice can be allocated to one layer. According to an embodiment of the disclosure, the electronic device 1000 can determine a processor processing each slice based on a processing time of each processor for each slice.

[0064] The processing time according to an embodiment can include a time taken for a current processor to process a task of at least one layer included in a current slice, and a switching time taken for the current processor to receive data required to process the current slice from another processor. For example, the data required to process the current slice and the data received from another processor can include data input to a layer included in the current slice. In addition, another processor can be a previous processor processing a previous slice and outputting data input to the current slice.

[0065] According to an embodiment, when a previous processor for processing a previous slice is different from a current processor for processing a current slice, the switching time can indicate a time taken to deliver data regarding a result of processing the previous slice to the current processor.

[0066] For example, when the first processor 1310 is allocated to slice 3 230, the switching time of the second processor 1320 for slice 4 240 can indicate a time taken to transfer output data of "conv2d_12" from the first processor 1310 to the second processor 1320. In addition, the switching time of the third processor 1330 for slice 4 240 can indicate a time taken to transfer output data of "conv2d_12" from the first processor 1310 to the third processor 1330. The switching time according to an embodiment can also include a time taken to transfer data from the first processor 1310 to the second processor 1320, and a time taken to convert the data to a data format of the second processor 1320. In addition, the switching time according to an embodiment can increase as the size of data to be transferred increases.

[0067] In addition, when a previous processor for processing a previous slice is the same as a current processor for processing a current slice, data is not transferred between processors, and thus the switching time according to an embodiment can be determined as 0. For example, when the first processor 1310 is allocated to slice 3 230, the switching time of the first processor 1310 for slice 4 240 can be determined as 0.

[0068] According to embodiments, therefore, a processor for processing each slice can be determined based on a time taken to perform a task of processing a layer of each slice and a switching time for transmitting data between processors to process a plurality of slices through a plurality of processors.

[0069] Figure 3 is a block diagram for describing an internal configuration of the electronic device 1000 according to an embodiment.

[0070] Figure 4 is a block diagram for describing an internal configuration of the electronic device 1000 according to an embodiment.

[0071] Referring to Figure 3 , the electronic device 1000 can include a processor 1300, a memory 1700, and an outputter 1200. However, the electronic device 1000 can be implemented by components other than those shown in Figure 3 .

[0072] For example, as shown in Figure 4 , the electronic device 1000 according to some embodiments can include a user inputter 1100, a sensing unit 1400, a communicator 1500, and an A / V inputter 1600, in addition to the processor 1300, the memory 1700, and the outputter 1200.

[0073] The user inputter 1100 can represent a tool by means of which a user inputs data to control the electronic device 1000. For example, the user inputter 1100 can include a keypad, a dome switch, a touchpad (a contact type capacitive method, a pressure type resistive method, an infrared detection method, a surface acoustic wave conduction method, an integral equation type tension measurement method, a piezoelectric effect method, etc.), a jog wheel, a jog switch, etc.; however, embodiments are not limited thereto.

[0074] According to an embodiment, the user inputter 1100 can receive a user input required to execute an operation through the neural network model 110. According to an embodiment of the disclosure, the electronic device 1000 can process the neural network model 110 by using a plurality of processors, and output a result thereof according to a user input received from the user inputter 1100.

[0075] The outputter 1200 can output an audio signal, a video signal, or a vibration signal, and the outputter 1200 can include a display unit 1210, an audio output interface 1220, and a vibration motor 1230.

[0076] The display unit 1210 can display information processed in the electronic device 1000.

[0077] When the display unit 1210 and the touch panel make the touch screen in a layer structure, the display unit 1210 can be used as an input device as well as an output device. The display unit 1210 can include at least one of a liquid crystal display, a thin film transistor liquid crystal display, an organic light emitting diode, a flexible display, a three-dimensional display, and an electrophoretic display. In addition, the electronic device 1000 can include two or more display units 1210 according to implementation types of the electronic device 1000.

[0078] The audio output interface 1220 can output audio data received from the communicator 1500 or stored in the memory 1700.

[0079] The vibration motor 1230 can output a vibration signal. In addition, when a touch is input onto the touch screen, the vibration motor 1230 can output a vibration signal.

[0080] According to an embodiment, a result of the neural network model 110 processed by the processor 1300 can be output in various forms through the display unit 1210, the audio output interface 1220, and the vibration motor 1230.

[0081] The processor 1300 can control overall operations of the electronic device 1000. For example, the processor 1300 can control the user inputter 1100, the outputter 1200, the sensing unit 1400, the communicator 1500, the A / V inputter 1600, etc. substantially by running programs stored in the memory 1700. The electronic device 1000 can include a plurality of processors.

[0082] The processor 1300 can be configured to process instructions of a computer program by performing basic arithmetic, logical, and input / output operations. The instructions can be provided to the processor 1300 from the memory 1700, or can be received through the communicator 1500 and provided to the processor 1300. For example, the processor 1300 can be configured to run instructions according to program codes stored in a storage device such as a memory.

[0083] The processor 1300 according to an embodiment can allocate a plurality of layers included in the neural network model 110 to at least one slice, and allocate processors from among a plurality of processors included in the processor 1300 to process each slice. In addition, based on a result of the allocation, the processor 1300 can process the neural network model 110.

[0084] According to an embodiment, the processor for each slice can be allocated based on a processing time of the processor for the slice. The processing time according to an embodiment can include a switching time taken for data transmission between processors.

[0085] According to an embodiment, the processor 1300 can identify at least one layer included in at least one slice allocated to a first processor among a plurality of processors. In addition, the processor 1300 can identify at least one chunk each representing at least one piece of data input to the identified layer, data output from the identified layer, and data temporarily stored in the identified layer, and allocate a memory to store data corresponding to the identified chunk.

[0086] According to an embodiment, a memory for a current chunk can be allocated by determining whether a time period for using a previous chunk is terminated before data of the current chunk is generated according to a series of processing layers. In addition, a size of the allocated memory can be determined based on a maximum data size among data sizes of at least one chunk to which the same memory is allocated.

[0087] The sensing unit 1400 can detect a state of the electronic device 1000 or an environmental state of the electronic device 1000, and deliver detected information to the processor 1300.

[0088] The sensing unit 1400 can include at least one of a geomagnetic sensor 1410, an acceleration sensor 1420, a temperature / humidity sensor 1430, an infrared sensor 1440, a gyro sensor 1450, a location sensor 1460 (e.g., a global positioning system (GPS)), an atmospheric pressure sensor 1470, a near field sensor 1480, and a red-green-blue (RGB) sensor 1490 (an illuminance sensor); however, embodiments are not limited thereto.

[0089] According to an embodiment, information detected by the sensing unit 1400 can be used as input information of the neural network model 110 or can be used to update the neural network model 110. In addition to the above-described examples, information detected by the sensing unit 1400 can be used according to various methods of processing the neural network model 110.

[0090] The communicator 1500 can include one or more components that enable the electronic device 1000 to communicate with the server 2000 or an external device. For example, the communicator 1500 can include a short-range wireless communication unit 1510, a mobile communicator 1520, and a broadcast receiver 1530.

[0091] The short-range wireless communication unit 1510 can include a Bluetooth communication unit, a Bluetooth low energy (BLE) communication unit, a near field communication unit, a wireless local area network (WLAN) or Wi-Fi communication unit, a Zigbee communication unit, an infrared data association (IrDA) communication unit, a Wi-Fi direct (WFD) communication unit, an ultra wide band (UWB) communication unit, an Ant+ communication unit, etc.; however, embodiments are not limited thereto.

[0092] The mobile communicator 1520 can transmit or receive a wireless signal to or from at least one of a base station, an external terminal, and a server on a mobile communication network. The wireless signal can include various types of data according to transceiving audio call signals, video communication call signals, or text / multimedia messages.

[0093] The broadcast receiver 1530 can receive a broadcast signal and / or information related to a broadcast from an external through a broadcast channel. The broadcast channel can include a satellite channel and a ground wave channel. According to a type of implementation, the electronic device 1000 can not include the broadcast receiver 1530.

[0094] According to an embodiment, the communicator 1500 can transmit a result of processing the neural network model 110 to an external device. Alternatively, the communicator 1500 can receive information for processing the neural network model 110 from an external device.

[0095] An audio / video (A / V) inputter 1600 inputs an audio signal or a video signal, and can include a camera 1610, a microphone 1620, etc. The camera 1610 can acquire an image frame such as a still image or a video image via an image sensor in a video call mode or a photographing mode. The image captured by the image sensor can be processed via the processor 1300 or a separate image processor. The microphone 1620 can receive an input of an external audio signal, and process the audio signal into electric audio data.

[0096] According to an embodiment, the image data or the audio data obtained by the A / V inputter 1600 can be used as input information of the neural network model 110 or can be used to update the neural network model 110. In addition to the above-described example, the image data or the audio data can be used according to various methods of processing the neural network model 110.

[0097] The memory 1700 can store a program for processing and controlling the processor 1300, and also store data input to or output from the electronic device 1000.

[0098] The memory 1700 according to an embodiment can store information about the neural network model 110 and information about performance of the plurality of processors. For example, the information about performance of the plurality of processors can include information about processing speed of the processor for a layer, information about a layer that can be processed by each processor, and information about time taken to switch data between different processors. In addition to the above-described example, the information about performance of the plurality of processors can also include various types of information for allocating a slice to the plurality of processors.

[0099] The memory 1700 can include at least one type of storage medium such as a flash memory, a hard disk, a multimedia micro card, a card-type memory (e.g., a secure digital (SD) or extreme digital (XD) memory), a random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, and an optical disk.

[0100] The programs stored in the memory 1700 can be classified into a plurality of modules by function, and can be, for example, a user interface (UI) module 1710, a touch screen module 1720, and a notification module 1730, etc.

[0101] The UI module 1710 can provide a dedicated UI linked to the electronic device 1000, a graphic UI (GUI), etc. for each application. The touch screen module 1720 can sense a touch gesture of a user on the touch screen and transmit information about the touch gesture to the processor 1300. The touch screen module 1720 according to some embodiments can recognize and analyze a touch code. The touch screen module 1720 can be composed of a separate hardware including a controller.

[0102] Various sensors can be arranged inside or near the touch screen to detect a touch or proximity touch on the touch screen. The tactile sensor is an example of a sensor for detecting a touch on the touch screen. The tactile sensor can sense a touch of a specific object at a level of human feeling or higher than it. The tactile sensor can detect a variety of information such as roughness of a contact surface, hardness of a contact material, and temperature of a touch point.

[0103] Various touch gestures of a user can include a tap, a touch and hold, a double tap, a drag, a fanning, a flick, a drag and drop, a swipe, etc.

[0104] The notification module 1730 can generate a signal for notifying the electronic device 1000 of an event occurrence.

[0105] Figure 5 is a flowchart of a method for processing a neural network model using a plurality of processors according to an embodiment.

[0106] Referring to Figure 5 In operation 510, the electronic device 1000 can allocate a plurality of layers constituting the neural network model 110 to at least one slice. The electronic device 1000 according to an embodiment can allocate the plurality of layers to at least one slice by determining whether each layer is a slice point.

[0107] For example, when the first layer is determined to be a slice point, a new first slice can be created, and the first layer can be assigned to the first slice instead of a previous slice. The previous slice can indicate that at least one layer has been allocated to the slice before determining whether the first layer is a slice point. On the other hand, when the first layer is not determined to be a slice point, the first layer can be allocated to the previous slice.

[0108] According to an embodiment, after assigning the first layer to the first slice, when the second layer is determined to be a slice point, a second slice can be newly created, and the second layer can be assigned to the second slice. On the other hand, when the second layer is not determined to be a slice point, the second layer can be allocated to the first slice.

[0109] In operation 520, the electronic device 1000 can allocate a plurality of layers constituting the neural network model 110 to at least one slice, and then allocate each slice to a plurality of processors. The plurality of processors according to an embodiment can process the neural network model 110 by processing at least one layer included in the slice allocated to each processor.

[0110] According to an embodiment, the electronic device 1000 can allocate each slice to a plurality of processors based on a processing time taken by each processor to process the slice. The processing time according to an embodiment can include a time taken by each processor to process the slice, and also include a switching time taken to receive data for processing the slice from another processor.

[0111] In operation 530, the electronic device 1000 can process the neural network model 110 by using the plurality of processors based on the allocation result in operation 520.

[0112] According to an embodiment, when the processors for processing respective slices are allocated, a plurality of slices arranged in parallel in the neural network model 110 can be processed by the plurality of processors at the same time. Accordingly, according to an embodiment, the layers of the neural network model 110 are arranged in a row according to a topology classification method, and thus the layers can be processed faster than in a case where the layers of the neural network model 110 are processed sequentially.

[0113] Figure 6 is a flowchart of a method of allocating slices to a plurality of processors according to an embodiment.

[0114] Referring to Figure 6 In operation 610, the electronic device 1000 can obtain information about a slice S and a processor P.

[0115] In an embodiment, the slice S and the processor P can respectively represent one slice S among at least one slice and one processor P among a plurality of processors. According to an embodiment, in operation 610, a processor capable of processing a layer included in the slice S among a plurality of processors included in the electronic device 1000 can be determined as the processor P.

[0116] For example, the information about the slice S can include information about a task of at least one layer included in the slice S. In addition, the information about the processor P can include information about a performance of the processor P, such as information about a task that the processor P can perform, information about a time taken for the processor P to process a task, and information about a switching time taken for data to be transmitted between different processors. In addition to the above-described examples, the information about the slice S and the processor P can include various information that can be used to predict a processing time and an accuracy of a result of processing the slice S by the processor P.

[0117] In operation 620, when the slice S is processed by the processor P based on the information obtained in operation 610, the electronic device 1000 can predict an accuracy of a result of processing the slice S, and determine whether the predicted accuracy is less than or equal to a reference value.

[0118] According to an embodiment, when the predicted accuracy is less than or equal to the reference value, in operation 650, it can be determined whether there is a processor other than the processor P. The other processor can be a processor capable of processing the slice S among a plurality of processors of the electronic device 1000 that has not yet been determined whether to be allocated to the slice S.

[0119] According to an embodiment, in operation 660, the electronic device 1000 can identify another processor capable of processing the slice S. The electronic device 1000 can repeat operations 610 to 640 with respect to the identified processor.

[0120] In operation 630, the electronic device 1000 can predict a time taken for a task of the slice S to be processed by the processor P based on the information obtained in operation 610. For example, the electronic device 1000 can determine a processing time of the processor P for each type of task, and based on the determined processing time, the electronic device 1000 can determine a processing time of the processor P for a task of at least one layer included in the slice S. In addition to the above-described example, the processing time of the processor P for the slice S can be predicted in various ways.

[0121] In addition, in operation 640, the electronic device 1000 can estimate a switching time taken for input data of the slice S to be transmitted from another processor to the processor P based on the information obtained in operation 610.

[0122] For example, the input data of the slice S can be transmitted to the processor P from a processor that processed a previous slice from which the input data is output. The previous slice according to the embodiment can indicate a slice that outputs data to be input to the slice S. In addition, the switching time can indicate a time that elapses to transmit the input data of the slice S from the processor that processed the previous slice to the processor P. Meanwhile, when the processor that processed the previous slice is the same as the processor P, the switching time can be determined as zero.

[0123] According to the embodiment, the switching time can be estimated based on a time that elapses to transmit data between different processors of the electronic device 1000.

[0124] In operation 650, the electronic device 1000 can determine whether there is another processor that did not determine at least one of the accuracy, the processing time, and the switching time for the slice S in operations 620 to 640 among the processors of the electronic device 1000. When there is another processor, the electronic device 1000 can repeat operations 610 to 640 for the other processor identified in operation 660.

[0125] In addition, for the slice S, when there is no other processor that did not determine at least one of the accuracy, the processing time, and the switching time for the slice S in operations 620 to 640, the electronic device 1000 can assign the slice S to one of the plurality of processors in operation 670.

[0126] According to the embodiment, the processor for processing the slice S can be determined based on the processing time and the switching time for the slice S of each processor. For example, a processor corresponding to a minimum sum of the processing time and the switching time can be determined as the processor for processing the slice S.

[0127] According to the embodiment, in operation 670, a processor corresponding to an accuracy equal to or less than a reference value can not be selected as the processor for processing the slice S. Accordingly, even when the processing time or the switching time is short, a processor having low accuracy for a processing result can not be assigned as the processor for processing the slice S.

[0128] According to an embodiment of the disclosure, after assigning the processor to the slice S, operations 610 to 670 can be repeatedly performed for slices of processors that are not assigned except for the slice S.

[0129] Figure 7 FIG. 6 is a diagram illustrating an example of assigning a processor to a slice according to an embodiment.

[0130] Reference Figure 7Referring to FIG. 7, a graph 710 illustrating an example of assigning processors to slices according to processing time and switching time is shown. In the graph 710, a node represents time taken to process a task of a slice, and an arrow represents time taken to switch data from a previous node to a current node.

[0131] According to an embodiment, unlike the method of sequentially assigning processors to slices as shown in FIG. 7, a processor for processing a slice can be determined based on a path corresponding to a minimum sum of processing times at respective nodes. The processing time according to an embodiment can include switching time and time taken for a processor to process a slice at each node. Figure 6

[0132] For example, a plurality of nodes 721, 722, 723, 731, 732, 733, 741, 742, and 743 can be generated based on a path corresponding to a minimum sum of processing times among a plurality of paths, the plurality of nodes indicating that a combination of different slices and different processors is sequentially connected according to an arrangement order of the slices, and a slice arranged in series in the neural network model 110 among at least one slice is assigned to a plurality of processors.

[0133] Input data 711 input to slice 1 can be processed by a CPU as a basic processor. According to a direction of an arrow, the input data 711 can be switched to one of processors such as a CPU, a GPU, and an NPU. When the input data 711 is switched to any one processor, the input data 711 can be input to a layer included in slice 1 and processed by one of nodes 721, 722, and 723.

[0134] According to an embodiment, since accuracy of a result of processing slice 2 by an NPU is less than or equal to a reference value, or slice 2 includes a layer that the NPU cannot process, the NPU can be excluded in an operation of assigning a processor to slice 2.

[0135] According to an embodiment, as shown in the graph 710, a path for assigning a CPU, a GPU, and an NPU to slices 1 to 3 can be determined as one of a total of 36 paths. The processors can be respectively assigned to slices 1 to 3 according to a path having a minimum sum of processing times and switching times of respective nodes among the 36 paths.

[0136] Referring to FIG. 7, a graph 710 illustrating an example of assigning processors to slices according to processing time and switching time is shown. In the graph 710, a node represents time taken to process a task of a slice, and an arrow represents time taken to switch data from a previous node to a current node.

[0137] ​The GPUs 16 and 32 shown in the graph 750 represent 16-bit GPUs and 32-bit GPUs, respectively, and can be handled as different processors in one embodiment.

[0138] According to an embodiment, when the slices 752, 754, and 755 are assigned to different processors, they can be processed in parallel simultaneously. Thus, according to an embodiment, some layers arranged in parallel in the neural network model 110 can be processed in parallel simultaneously by multiple processors, and thus the processing speed can be further improved.

[0139] According to one embodiment, after the layers are aligned in series, the slices 755 and 756 to which the same processor is assigned can be processed by the NPU, so that the layers can be processed by the NPU simultaneously. For example, the layers can be aligned according to a topological sorting method, and then processed by the NPU simultaneously.

[0140] According to an embodiment of the present disclosure, at least one slice can be assigned to multiple processors according to a combination of the method of assigning a slice to a processor according to the graph 710 of Figure 7 and the method of assigning a slice to a processor according to the graph 720 of Figure 6 .

[0141] In the neural network model 110, a processor can be assigned to slices arranged in series, such as the slices 751, 752, and 753, based on a path corresponding to a minimum total sum of processing time among multiple paths generated for the multiple nodes, which indicates that the combination of different slices and different processors is connected according to the arrangement order of the slices, as shown in the graph 710 of Figure 7 .

[0142] Alternatively, as shown in the graph 720 of Figure 6 , a processor can be assigned to slices arranged in parallel in the neural network model 110, such as the slices 754, 755, and 756, based on the processing time, switching time, and accuracy of the processor for the slices.

[0143] Figure 8 is a diagram showing an example of assigning memory in a layer according to an embodiment.

[0144] According to an embodiment, a convolution layer among the layers can include tasks of repeatedly performing a 1x1 convolution 812, depth convolutions (DCs) 813 to 815, and a 1x1 convolution 816 from a channel 1 to a channel N, as shown in the graph 730 of Figure 8 . The tasks for the channel 1 can be performed in the operation 810, and the tasks for the channel 2 can be performed in the operation 820 after the operation 810 is performed. Likewise, the channel N can be performed in the operation 830 after the operations before the channel N are performed.

[0145] According to an embodiment, the channels can respectively correspond to a plurality of data having the same size among input data input to the layer. For example, in the first channel, first data among the plurality of data can be processed. In the second channel, second data can be processed. Accordingly, when a plurality of image data having the same size is input to the layer, the plurality of input data can be sequentially processed for the same number of channels as the number of the plurality of input data. According to an embodiment, in the convolution layer, the tasks can be sequentially performed. For example, after performing the 1x1 convolution 812 for the input data of all the channels, the DCs 813 to 815 can be performed based on the result of performing the 1x1 convolution 812. In addition, the 1x1 convolution 816 can be performed based on the result of performing the DCs 813 to 815.

[0146] According to an embodiment of the disclosure, the tasks can be sequentially performed from channel 1 to channel N, instead of sequentially performing the tasks as described above. For example, when there are 24 pieces of input data having a size of 56x56, there are 24 channels, and the tasks from channel 1 to channel 24 can be sequentially performed for each channel.

[0147] According to an embodiment, when the tasks are sequentially performed for each channel, a memory of 294 KB size for storing 24 pieces of data having a size of 56x56 811, 821, and 831 input to the convolution layer can be allocated. For example, when the input data is image data, a memory for storing a plurality of pieces of image data having the same size can be allocated. According to an embodiment, when the tasks are sequentially performed for each channel, instead of allocating a memory for storing data for all the channels, only a memory for storing data for one channel can be allocated. Accordingly, when the tasks are sequentially performed for each channel, compared to the case where the tasks are sequentially performed one after another, a memory space for data storage can be reduced.

[0148] In operation 810, the task of channel 1, that is, the 1x1 convolution 812, the DCs 813, 814, and 815, and the 1x1 convolution 816 can be sequentially performed. The 1x1 convolution 812 performed in operation 810 can be performed on first data among 24 pieces of data 811 having a size of 56x56 and first data among 24 pieces of data 812 having a size of 1x24.

[0149] According to an embodiment, in order to process the 1x1 convolution 812, instead of allocating all 24 pieces of data 812 having a size of 1x24 already allocated to the memory, only first bits among 24 bits of data 812 having a size of 1x24 used only in operation 810 can be allocated to the memory. The 24 pieces of data having a size of 1x24 for processing the 1x1 convolution 812 can be a value pre-stored in the electronic device 1000.

[0150] In operation 820, the second data 822 of size 1x24 can be stored in the same space as the data 812 allocated in operation 810 without additional memory allocation.

[0151] Accordingly, according to an embodiment, when 1x1 convolutions 812, 822, and 832 are performed on each of the 24 channels, instead of allocating 14 KB memory in which all 24 pieces of data of size 1x24 can be stored, only 0.009 KB memory in which one piece of data of size 1x24 can be stored can be allocated. According to an embodiment, the memory space allocated for processing the 1x1 convolution in the previous channel can be reused for the 1x1 convolution process in the current channel. Accordingly, the memory space for processing the layer in each channel can be reduced.

[0152] For memory allocation of data processed in the DC 813, 814, 815, memory can also be allocated for data processed in the DC 813 to 815, not all data. Reference numeral 813 denotes data obtained by performing the 1x1 convolution 812, and can include data of size 56x56. In addition, reference numeral 814 denotes kernel data of size 3x3 for the DC. The kernel data of size 3x3 can be a value stored in advance in the electronic device 1000 for processing the DC. Reference numeral 815 denotes data obtained as a result of processing the DC and can include data of size 56x56.

[0153] According to an embodiment, in order to process the DC 813, 814, and 815 in channel 1, 24.53 KB memory can be allocated for storing data 813 of size 56x56 (12.24 KB), kernel data 814 of size 3x3 (0.03 KB), and data 815 of size 56x56 (12.14 KB). In operation 820, in channel 2, data 823, 824, and 825 can be stored in the same space as the memory allocated for data 813, 814, and 815 in the previous operation 810 without additional memory allocation.

[0154] For memory allocation of data processed in the 1x1 convolution 816, memory can also be allocated for data processed in the 1x1 convolution 816, not all data. In 816, 144 pieces of data of size 1x1 can be used in the 1x1 convolution 816. Accordingly, 0.09 KB memory capable of storing 144 pieces of data of size 1x1 can be allocated. In operation 820, data 826 can be stored in the same space as the memory allocated for data 816 in operation 810 without additional memory allocation.

[0155] As a result of performing the 1x1 convolution 816, in 817, 24 pieces of data of which the size is 56x56 can be obtained. According to an embodiment of the disclosure, 294 KB of memory can be allocated in which the 24 pieces of data of which the size is 56x56 obtained in 817 can be stored. In operation 820, as in 817, when the 1x1 convolution 826 is performed in 827, 24 pieces of data of which the size is 56x56 can be obtained. Thereafter, also in the channels 3 to N, as a result of performing the 1x1 convolution, the DC, and the 1x1 convolution, 24 pieces of data of which the size is 56x56 can be obtained for each channel.

[0156] According to an embodiment, by combining the 24 pieces of data of which the size is 56x56 output from each channel, a final result of the convolution layer can be obtained.

[0157] Accordingly, a total of 612 KB of memory space can be allocated, which is a sum of at least one piece of data for a task in the channel 1 to be processed. According to an embodiment, in the channels 2 to N, since the tasks of the channels are sequentially performed, the 612 KB of memory space allocated in the channel 1 can be reused without additional memory allocation. Accordingly, all of the tasks of the channels 1 to N of the convolution layer can be processed using only the 612 KB of memory space.

[0158] According to an embodiment, the size of the memory space for processing a layer is not limited to the above-described example, and various sizes of memory space can be allocated and used according to the size of data to be processed.

[0159] In addition, Figure 8 An example of processing input data through a convolution layer is illustrated. However, according to an embodiment, the disclosure is not limited thereto, and a plurality of pieces of input data can be input to various types of layers other than the convolution layer. According to an embodiment, as in the case of allocating memory space for the above-described convolution layer, each of other types of layers can be allocated as much memory space as the size of memory required for processing a task in a first channel among a plurality of channels of the layer for processing a task in a plurality of channels of the layer. In channels other than the first channel, the memory allocated for the task in the first channel can be reused without additional memory allocation.

[0160] Figure 9 is a flowchart of a method of allocating memory for I / O data of a layer according to an embodiment.

[0161] Memory according to an embodiment can be allocated to a chunk respectively corresponding to I / O data of a layer. The neural network model 110 according to an embodiment can include a layer on which a task is performed and a chunk respectively corresponding to data input to / output from the layer.

[0162] According to an embodiment, not only a chunk for I / O data of a layer but also a chunk for intermediate data generated by an internal function of the layer can be identified. Accordingly, the electronic device 1000 can allocate a memory by further considering a chunk for intermediate data.

[0163] In an embodiment, memory allocation refers to an operation for allocating a space of a memory in which data is to be stored in advance during a compilation operation. During memory allocation according to an embodiment, after determining chunks to be allocated to the same memory space, a size of each memory space can be determined based on a data size of the chunks.

[0164] Reference Figure 9 In operation 910, the electronic device 1000 can identify a layer allocated to each processor. According to an embodiment, the electronic device 1000 can identify at least one layer included in at least one slice allocated to a first processor from among a plurality of processors according to a method of allocating a slice to a plurality of processors. The electronic device 1000 according to an embodiment can perform memory allocation according to an embodiment with respect to each processor.

[0165] In operation 920, the electronic device 1000 can determine an order for executing at least one layer allocated to one processor. According to an embodiment, the electronic device 1000 can determine an order for processing at least one layer according to a phase alignment method. The present disclosure is not limited to the above-described example, and an order for processing one layer can be determined in various ways.

[0166] In operation 930, the electronic device 1000 can identify an internal function included in each layer, and in operation 940, can identify a chunk including data temporarily stored in a layer including the internal function based on the identified internal function. According to an embodiment, a chunk input and output to each layer can be a chunk that has been identified according to information about a structure of the neural network model 110.

[0167] For example, when two or more internal functions are included in a layer, there can be a chunk for temporarily storing intermediate data between the two internal functions. In addition to the above-described example, a chunk other than a chunk for data input to and / or output from a layer through an internal function of the layer can be identified in operation 940.

[0168] According to an embodiment of the present disclosure, a chunk for temporarily storing data such as I / O data of a layer through an internal function of the layer needs to be allocated a memory for storing data when a task is performed. Accordingly, the electronic device 1000 can perform memory allocation considering a chunk for temporary data storage in a layer as well as a chunk for I / O data of the layer.

[0169] In operation 950, the electronic device 1000 can allocate a memory in which at least one of the chunk storing the I / O data for each layer and the chunk of the internal function for each layer identified in operation 940 is to be stored.

[0170] According to an embodiment, the memory for the current chunk can be allocated based on whether a time period for using a previous chunk is terminated before data of the current chunk is generated according to an order for processing each layer. For example, when the time period for using the previous chunk is terminated before the data of the current chunk is generated, the same memory allocated to the previous chunk can be allocated to the current chunk. The time period for using a chunk according to an embodiment can be determined based on a time period in which data of the chunk is used by at least one layer. For example, the time period for using a chunk can refer to a time period in which data of the chunk is read from or written to a memory space (e.g., a buffer) storing the data while a task of each layer is performed. In addition, the time period in which data of a chunk for which the time period for use is determined is used by at least one layer can be determined based on a lifetime of the chunk. For example, the lifetime of the chunk can be a preset value, or can be determined based on a time period for running a function of a layer in which the chunk is processed. In addition, the time period for using a chunk can be determined in various ways other than the above-described examples.

[0171] For example, when a task of the first layer is completed, the electronic device 1000 can no longer use the input data of the first layer and the data temporarily stored through the internal function of the first layer. Accordingly, the temporarily stored data and the output data of a second layer operating after the first layer can use a memory space in which the input data and the data temporarily stored through the internal function of the first layer are stored.

[0172] The electronic device 1000 according to an embodiment can determine a maximum data size among data sizes of at least one chunk to which the same memory is allocated, as a size of the memory.

[0173] For example, it can be determined that the same memory for indicating the chunk 3 of the temporarily stored data of the second layer and indicating the chunk 4 of the output data of the second layer is allocated to the chunk 1 indicating the input data of the first layer and the chunk 2 indicating the temporarily stored data of the first layer. For example, a first memory can be allocated to the chunk 1 and the chunk 3, and a second memory can be allocated to the chunk 2 and the chunk 4.

[0174] When the data size of the chunk 1 is 1 kb and the data size of the chunk 3 is 2 kb, a size of 2 kb, which is the largest data size among the chunks allocated to the first memory, can be allocated to the first memory. Also, when the data size of the chunk 2 is 4 kb and the data size of the chunk 4 is 5 kb, a size of 5 kb, which is the largest data size among the chunks allocated to the second memory, can be allocated to the second memory.

[0175] Figure 10 FIG. 1 is a diagram illustrating an example of identifying a chunk within a layer according to an embodiment.

[0176] Referring to Figure 10 From among the plurality of layers included in the neural network model 110, the internal functions "im2col" 1011 and "sgemm" 1013 can be identified with respect to the "conv2d_65" 1010 layer. Also, the chunk b2 1012 temporarily stored between the internal functions 1011 and 1013 can be identified.

[0177] The chunk b1 1020 and the chunk b3 1030 according to an embodiment are chunks indicating input data and output data of the "conv2d_65" 1010 layer, respectively.

[0178] Accordingly, according to an embodiment, for the "conv2d_65" 1010 layer, the chunk b1 1020, the chunk b2 1012, and the chunk b3 1030 can be identified, and memory allocation can be performed with respect to the identified chunks.

[0179] According to an embodiment, the memory allocated for each chunk can be allocated based on whether a time period for using a previous chunk is terminated before data of a current chunk is generated.

[0180] For example, from a time point at which processing of the "sgemm" 1013 is started after completion of processing of the "im2col" 1011, the chunk b1 1020 can not be used any more. The time period for using the chunk b1 1020 can be terminated before data of the chunk b3 1030 is generated. Accordingly, according to an embodiment, the same memory can be allocated to the chunk b1 1020 and the chunk b3 1030. For example, when processing the "sgemm" 1013, after the chunk b1 1020 previously stored in the same memory space to which the chunk b1 1020 is allocated is deleted, data of the chunk b3 1030 can be stored therein.

[0181] Also, the memory allocated to a chunk among chunks before the chunk b2 1012, for which a use time period is terminated before the chunk b2 1012 is generated, can be reallocated to the chunk b2 1012.

[0182] Figure 11 is a diagram showing an example of allocating memory to a chunk of a neural network model 110 including a chunk within a layer according to an embodiment.

[0183] According to an embodiment, chunks 1101, 1103, 1104, 1105, 1106, 1108, 1109, 1111, and 1115 representing I / O data of a plurality of layers can be identified. In addition, chunks 1102 and 1107 representing data that can be temporarily stored due to an internal function of each layer can be identified.

[0184] In Figure 11 , the hatched boxes represent layers on which a task can be performed, and the non-hatched boxes represent chunks indicating data.

[0185] According to an embodiment, when time periods in which data chunks 1101, col_buffer1 chunks 1102, and conv1 chunks 1103 are used overlap each other, they can be allocated to different memories. For example, data chunks 1101, col_buffer1 chunks 1102, and conv1 chunks 1103 can be allocated to b1 memory 1112, b2 memory 1114, and b3 memory 1113, respectively.

[0186] In addition, data chunks 1101 and col_buffer1 chunks 1102 can be used in the conv1 layer and can not be used after the relu1 layer that processes after the conv1 layer. Accordingly, a time period for using data chunks 1101 and col_buffer1 chunks 1102 can be terminated before generating relu1 chunks 1104. Relu1 chunks 1104 can be allocated to b1 memory 1112 or b2 memory 1114, which is the same memory as that for data chunks 1101 or col_buffer1 chunks 1102. In an embodiment, it is assumed that relu1 chunks 1104 are allocated to b1 memory 1112.

[0187] In addition, norm1 chunks 1105 can be allocated to a memory in which a chunk corresponding to a use time period that is terminated before generating norm1 chunks 1105 is stored. For example, norm1 chunks 1105 can be allocated to b2 memory 1114 or b3 memory 1113. In an embodiment, it is assumed that norm1 chunks 1105 are allocated to b2 memory 1114.

[0188] In addition, the pool1 blob 1106 can be allocated to the b1 memory 1112 or the b2 memory 1114, which is a memory in which a usage time period ends before the pool1 blob 1106 is generated. In an embodiment, it is assumed that the pool1 blob 1106 is allocated to the b1 memory 1112.

[0189] In addition, the col_buffer2 blob 1107 can be allocated to the b2 memory 1114 or the b3 memory 1113, which is a memory in which a usage time period ends before the col_buffer2 blob 1107 is generated. In an embodiment, it is assumed that the col_buffer2 blob 1107 is allocated to the b2 memory 1114.

[0190] In addition, the conv2 blob 1108 can be allocated to the b1 memory 1112 or the b3 memory 1113, which is a memory in which a usage time period ends before the conv2 blob 1108 is generated.

[0191] In addition, the relu2 blob 1109 can be allocated to the b1 memory 1112 or the b2 memory 1114, which is a memory in which a usage time period ends before the relu2 blob 1109 is generated. In an embodiment, it is assumed that the relu2 blob 1109 is allocated to the b1 memory 1112.

[0192] In addition, the norm2 blob 1115 can be allocated to the b2 memory 1114 or the b3 memory 1113, which is a memory in which a usage time period ends before the norm2 blob 1115 is generated. In an embodiment, it is assumed that the norm2 blob 1115 is allocated to the b2 memory 1114.

[0193] In addition, the pool2 blob 1111 can be allocated to the b1 memory 1112 or the b2 memory 1114, which is a memory in which a usage time period ends before the pool2 blob 1111 is generated. In an embodiment, it is assumed that the pool2 blob 1110 is allocated to the b1 memory 1112.

[0194] According to an embodiment, after each blob is allocated to the b1 memory 1112, the b2 memory 1114, or the b3 memory 1113 according to the above-described method, the sizes of the b1 memory 1112, the b2 memory 1114, and the b3 memory 1113 can be determined based on a maximum value among data sizes of the blobs allocated to the respective memories.

[0195] According to an embodiment, the blobs can have data size values as shown in Table 2 below.

[0196] Table 2

[0197]

[0198] According to an embodiment, the size of the b1 memory 1112 can be determined based on the maximum value (i.e., 1.11 MB) among the data sizes of the data chunk 1101, the relu1 chunk 1104, the pool1 chunk 1106, the relu2 chunk 1109, and the pool2 chunk 1111 allocated to the b1 memory 1112.

[0199] In addition, the size of the b2 memory 1114 can be determined based on the maximum value (i.e., 6.67 MB) among the data sizes of the col_buffer1 chunk 1102, the norm1 chunk 1105, the col_buffer2 chunk 1107, and the norm2 chunk 1110 allocated to the b2 memory 1114.

[0200] In addition, the size of the b3 memory 1113 can be determined based on the maximum value (i.e., 1.11 MB) among the data sizes of the colv1 chunk 1103 and the conv2 chunk 1108 allocated to the b3 memory 1113.

[0201] Accordingly, according to an embodiment, 8.89 MB of memory space can be used to store the illustrated chunks, which is the sum of the sizes of the b1 memory 1112, the b2 memory 1114, and the b3 memory 1113. Figure 11

[0202] When memory is allocated only for the chunks of I / O data without allocating memory for the chunks of internal data, memory for storing the chunks of internal data needs to be reallocated, and thus the memory allocation space can increase.

[0203] However, according to an embodiment, when allocating memory, not only the chunks for I / O data but also the chunks for internal data generated by internal functions of a layer are considered. Accordingly, a smaller memory space can be allocated.

[0204] Figure 12 is a diagram illustrating an example of processing a neural network model by a plurality of processors according to an embodiment.

[0205] Based on the method for allocating layers to a plurality of processors according to an embodiment, the layer 1 to the layer 4 1201, 1202, 1203, and 1204 can be processed by a CPU, a GPU, a GPU, and a CPU, respectively.

[0206] ​The memory can be allocated to the chunks of I / O data and internal data of the layer 1 1201 and the layer 4 1204 processed by the CPU based on the memory allocation method according to an embodiment. For example, at least one memory allocated to a chunk (e.g., a chunk including I / O data or internal data of the layer 1 1201) of the layer 1 1201 can be reallocated to a chunk of the layer 4 1204.

[0207] According to an embodiment, when there are a plurality of chunks of internal data in the same layer, the same memory can be allocated to the chunks.

[0208] According to an embodiment, the chunks 1 to 4 1205, 1206, 1207, and 1208 can exist in the layer 1 1201 for different internal data generated by different internal functions in the layer 1 1201. For example, output data implemented by the internal functions 1 to 4 arranged in series can be indicated by the chunks 1 to 4 1205, 1206, 1207, and 1208, respectively. In addition, data of the chunk 4 1208 can be used as input data of the internal function 5.

[0209] According to an embodiment, since the task of the internal function 3 of the chunk 3 1207 can be performed after the task of the internal function 1 of the chunk 1 1205 is completed, the period of time using the chunk 1 1205 can be terminated before the chunk 3 1207 is generated. Accordingly, the memory allocated to the chunk 1 1205 can be reallocated to the chunk 3 1207.

[0210] In addition, since the task of the internal function 4 of the chunk 4 1206 can be performed after the task of the internal function 4 of the chunk 4 1208 is completed, the period of time using the chunk 2 1206 can be terminated before the chunk 4 1208 is generated. Accordingly, the memory allocated to the chunk 2 1206 can be reallocated to the chunk 4 1208.

[0211] According to an embodiment, by using a plurality of processors, a neural network model can be processed more quickly and accurately.

[0212] One or more embodiments can be implemented by a computer-readable recording medium including computer-executable instructions, such as a program module executed by a computer. The computer-readable recording medium can be any available medium accessible by a computer, and examples thereof include all volatile media (e.g., RAM) and non-volatile media (e.g., ROM), and detachable and non-detachable media. Furthermore, examples of the computer-readable recording medium can include computer storage media and communication media. Examples of the computer storage media include all volatile and non-volatile media that have been implemented by any method or technology for storing information such as computer-readable instructions, data structures, program modules, and other data, and detachable and non-detachable media. The communication media typically include computer-readable instructions, data structures, or program modules, and include any information delivery media.

[0213] In addition, in the present specification, a "unit" can be a hardware component such as a processor or circuit, and / or a software component executed by a hardware component such as a processor.

[0214] Those of ordinary skill in the art will understand that various changes in form and detail can be made without departing from the spirit and scope of the present disclosure as defined by the following claims. Accordingly, it should be understood that the above-described embodiments are illustrative only, and not restrictive. For example, each component described as a single entity can be distributed and implemented, and components described as distributed can also be implemented in a combined form.

[0215] Therefore, the scope of the present disclosure is not limited by the specific embodiments, but is defined by the appended claims, and all differences within the scope of equivalents are to be interpreted as being included in the present disclosure.

Claims

1. A method for processing a neural network model using multiple processors, wherein the neural network model includes multiple layers, and the method comprises: determining at least one layer among the plurality of layers as a slicing point; assigning the plurality of layers to at least one slice based on the determined slice points, such that each slice includes one or more layers among the plurality of layers; For each slice, predicting, based on performance information related to each processor among the plurality of processors, a processing time and accuracy of a processing result of processing each slice by each processor, wherein the processing time includes a time elapsed for a current processor to process a task of a layer included in the current slice; allocating each slice of the at least one slice to a corresponding processor among the plurality of processors in units of slices based on the predicted processing time and the predicted accuracy; as well as The neural network model is processed by using the plurality of processors based on the distributed results.

2. The method according to claim 1, in, The slicing point is determined based on at least one of: whether each of the plurality of layers is a branch point of the plurality of layers; whether each of the plurality of layers is a point of combination of the plurality of layers; whether each of the plurality of layers includes tasks that can be processed by the same processor; and whether each of the plurality of layers includes tasks requiring high accuracy.

3. The method of claim 1, wherein Each slice of the at least one slice is assigned to a corresponding processor among the multiple processors based on a path corresponding to a minimum sum of the processing times among multiple paths generated as multiple nodes, wherein the multiple nodes indicate that combinations of different slices among the at least one slice and different processors among the multiple processors are connected according to an arrangement order of the at least one slice.

4. The method according to claim 1, wherein sequentially processing, for each of a plurality of channels having the same number as the number of pieces of input data, the plurality of pieces of input data to be input to a layer among the plurality of layers, and For the multiple channels of the layer, a memory size required for processing tasks in a first channel among the multiple channels is allocated.

5. The method of claim 1 , further comprising: identifying one or more layers included in at least one slice assigned to a first processor among the plurality of processors; identifying at least one blob indicative of at least one of: data input to the identified one or more layers, data output from the identified one or more layers, and data temporarily stored in the identified one or more layers; as well as Memory is allocated to store data of the at least one blob.

6. The method of claim 5, wherein: The allocation includes: determining an order for processing the identified one or more layers; and Memory for the current blob is allocated by determining whether a time period for using a previous blob expires before generating data for the current blob based on the sequence.

7. The method according to claim 5, wherein: The allocation includes: The size of the memory is determined based on a size of a maximum data among sizes of data of the at least one blob to which the same memory is allocated.

8. The method according to any one of claims 1 to 7, wherein The processing time also includes a switching time taken for each of the plurality of processors to receive data for processing a current slice from a previous processor that processed a previous slice among the plurality of processors.

9. An electronic device for processing a neural network model, the neural network model comprising a plurality of layers, and the electronic device comprising: a memory configured to store the neural network model; Multiple processors, configured as: determining at least one layer among the plurality of layers as a slicing point; assigning the plurality of layers to at least one slice based on the determined slice points, such that each slice includes one or more layers among the plurality of layers; For each slice, predicting, based on performance information related to each processor among the plurality of processors, a processing time and accuracy of a processing result of processing each slice by each processor, wherein the processing time includes a time elapsed for a current processor to process a task of a layer included in the current slice; allocating each slice of the at least one slice to a corresponding processor among the plurality of processors in units of slices based on the predicted processing time and the predicted accuracy; processing the neural network model based on the assigned results, and An outputter is configured to output a result of processing the neural network model.

10. The electronic device according to claim 9, in, The slicing point is determined based on at least one of: whether each of the plurality of layers is a branch point of the plurality of layers; whether each of the plurality of layers is a point of combination of the plurality of layers; whether each of the plurality of layers includes tasks that can be processed by the same processor; and whether each of the plurality of layers includes tasks requiring high accuracy.

11. The electronic device according to claim 9, wherein: Each slice of the at least one slice is assigned to a corresponding processor among the multiple processors based on a path corresponding to a minimum sum of the processing times among multiple paths generated as multiple nodes, wherein the multiple nodes indicate that combinations of different slices among the at least one slice and different processors among the multiple processors are connected according to an arrangement order of the at least one slice.

12. The electronic device according to claim 9, wherein: sequentially processing, for each of a plurality of channels having the same number as the number of pieces of input data, the plurality of pieces of input data to be input to a layer among the plurality of layers, and For the multiple channels of the layer, a memory size required for processing tasks in a first channel among the multiple channels is allocated.

13. The electronic device according to claim 9, wherein: The plurality of processors identify one or more layers included in at least one slice assigned to a first processor among the plurality of processors, identifying at least one blob indicative of at least one of: data input to the identified one or more layers, data output from the identified one or more layers, and data temporarily stored in the identified one or more layers, and Memory is allocated to store data of the at least one blob.

14. The electronic device according to claim 13, wherein: The plurality of processors determines an order for processing the identified one or more layers and allocates memory for a current blob by determining whether a time period for using a previous blob expires before generating data for the current blob based on the order.

15. The electronic device according to claim 13, wherein: The plurality of processors determine a size of a memory for storing data of the at least one blob based on a size of maximum data among sizes of data of the at least one blob to which the same memory is allocated.

16. The electronic device according to any one of claims 9 to 13, wherein: The processing time also includes a switching time taken for each of the plurality of processors to receive data for processing a current slice from a previous processor that processed a previous slice among the plurality of processors. 17 . A computer-readable recording medium configured to store one or more computer programs including instructions, which, when executed by a plurality of processors, cause the plurality of processors to control the processors to perform the method according to claim 1 .

Images