System and method for executing tasks by executing artificial neural networks through heterogeneous resources

Abstract

A system and method for executing tasks by executing an artificial neural network through heterogeneous resources is disclosed. The system includes: a model analyzer that receives an artificial neural network model and outputs subgraph information generated based on the artificial neural network model including at least one subgraph; a detector that outputs perception information about the plurality of heterogeneous resources; and a task manager that outputs a first request signal for executing a task for each layer to a first resource among the plurality of heterogeneous resources based on the subgraph information and the perception information, and outputs a second request signal for executing a task for each depth direction to a second resource among the plurality of heterogeneous resources.

Description

[0001] Korean Patent Application No. 10-2018-0134440, filed on November 5, 2018 in the Korean Intellectual Property Office and entitled “METHOD OF MANAGING TASKS IN ARTIFICIAL NEURAL NETWORK AND SYSTEM INCLUDING ARTIFICIAL NEURAL NETWORK SYSTEM USING THE METHOD” is hereby incorporated by reference in its entirety. Technical Field

[0002] Embodiments relate to a method and system for managing tasks of an artificial neural network, and more particularly, to a method and system for managing tasks to be processed by an artificial neural network driven by a plurality of heterogeneous resources. Background Art

[0003] Artificial neural network (ANN) represents a computing architecture that models the biological brain. Deep learning, machine learning, etc. can be implemented based on artificial neural networks. Recently, as the number of tasks that will be processed by artificial neural networks has increased significantly, there is a need to implement artificial neural networks to perform arithmetic processing efficiently. Summary of the invention

[0004] According to one embodiment, there is provided a system for managing tasks to be performed by executing an artificial neural network by multiple heterogeneous resources, the artificial neural network system comprising: a model analyzer, receiving an artificial neural network model and outputting subgraph information generated based on the artificial neural network model; a detector, outputting perception information about the multiple heterogeneous resources; and a task manager, outputting a first request signal for a first resource among the multiple heterogeneous resources to execute the task for each layer, and a second request signal for a second resource among the multiple heterogeneous resources to execute the task for each depth direction based on the subgraph information and the perception information.

[0005] According to one embodiment, a method for managing tasks to be performed by executing an artificial neural network by multiple heterogeneous resources is provided, the method comprising: receiving an artificial neural network model and outputting subgraph information generated based on the artificial neural network model; outputting perception information about the multiple heterogeneous resources; performing a partitioning operation of multiple tasks included in the subgraph information based on the subgraph information; and outputting a request signal for one or more processes in the multiple heterogeneous resources on which the partitioning operation is performed, based on the perception information.

[0006] According to one embodiment, a system for managing tasks to be performed by resources executing artificial neural networks is provided, the artificial neural network system comprising: a model analyzer, which receives an artificial neural network model and outputs subgraph information generated based on the artificial neural network model; and a task manager, which divides a plurality of tasks contained in the subgraph based on the subgraph information to execute tasks for each depth direction, provides a request signal to a resource that does not support batch mode to execute the divided plurality of tasks, and provides N request signals to execute the plurality of tasks, wherein N is the amount of input data. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Features will become apparent to those skilled in the art by describing in detail exemplary embodiments with reference to the accompanying drawings, in which:

[0008] Figure 1 An artificial neural network system according to an embodiment is shown;

[0009] Figure 2 shows the data flow of an artificial neural network system according to an embodiment;

[0010] Figure 3A A diagram for explaining a general resource is shown. Figure 3B It is a diagram used to explain the resources dedicated to a specific task;

[0011] Figure 4 A flowchart for explaining a method for managing the operation of an artificial neural network system according to an embodiment is shown;

[0012] Figure 5A and Figure 5B A diagram illustrating a number of transmission times of a request signal that varies according to a layout mode according to an embodiment is shown;

[0013] Figure 6 A diagram for explaining a method for processing a task in an artificial neural network model according to an embodiment is shown;

[0014] Figure 7 A flowchart for explaining a method for managing the operation of an artificial neural network system according to an embodiment is shown;

[0015] Figure 8 A flowchart for explaining a method for managing the operation of an artificial neural network system according to an embodiment is shown;

[0016] Fig. 9 A flowchart for explaining a method for managing the operation of an artificial neural network system according to an embodiment is shown; and

[0017] Fig.10 A flowchart for explaining a work management method performed by an artificial neural network system according to an embodiment is shown. DETAILED DESCRIPTION

[0018] Figure 1 An artificial neural network system according to an embodiment is shown. Figure 1 , the artificial neural network system 1 may include a plurality of applications APP_1 to APP_N, a framework 10 and an operating system (OS) 20. The artificial neural network system 1 may communicate with one or more resources RS.

[0019] According to one embodiment, multiple applications APP_1 to APP_N can be generated through various application program interfaces (APIs) and can execute artificial neural network models to perform various tasks. For example, multiple applications APP_1 to APP_N can request the framework 10 to execute artificial neural network models to perform data processing.

[0020] According to one embodiment, the plurality of resources RS_1 to RS_N may include heterogeneous resources RS_1 to RS_N that are different resources, respectively. The plurality of resources RS_1 to RS_N may include computing resources and communication resources, but are not limited thereto.

[0021] For example, computing resources may include various computing devices (e.g., central processing unit (CPU), graphics processing unit (GPU), application processor (AP), digital signal processor (DSP), field programmable gate array (FPGA), neural network processing unit (NPU), electronic control unit (ECU), image signal processor (ISP), etc.). Communication resources may include resources supporting internal communication (e.g., system bus, peripheral component interconnect (PCI), PCI Express (PCIe), serial advanced technology attachment (SATA), small computer system interface (SCSI), and serial attached small computer system interface (SCSI) (SAS)) bus and / or resources supporting external communication (such as universal serial bus (USB), Ethernet, WiFi, Bluetooth, near field communication (NFC), radio frequency identification (RFID), and mobile telecommunications, etc.).

[0022] The computing resources may include specific task-specific resources. The specific task-specific resources may include, for example, a network processing unit (NPU), a visual processing unit (VPU), etc. For example, the NPU may be a specific task-specific resource for performing an artificial neural network model processing task.

[0023] The heterogeneous resources RS_1 to RS_N are different types of resources. For example, the first resource RS_1 may be a CPU, the second resource RS_2 may be a GPU, and the Nth resource RS_N may be an NPU.

[0024] According to one embodiment, the framework 10 may include a model analyzer 100, a task manager 200, a detector 300, and a compiler 400. Each configuration included in the framework 10 may be implemented in software and / or hardware (e.g., electronic circuits). For example, the configuration included in the framework 10 may be implemented by a neural network platform (e.g., Evolutionary Deep Network (EDEN)), and / or may be implemented by one or more circuits.

[0025] According to one embodiment, the model analyzer 100 may receive data including an artificial neural network model received from a plurality of applications APP_1 to APP_N, and may obtain information about the artificial neural network model from the received data. For example, a model parser included in the model analyzer 100 may obtain information about the model.

[0026] The model analyzer 100 may acquire information about the model to construct a data structure represented in a graph structure. For example, a model builder included in the model analyzer 100 may configure the data structure.

[0027] The model analyzer 100 may change the data structure represented by the graph structure into a structure suitable for processing by the heterogeneous resources RS_1 to RS_N. Preferably, the model analyzer 100 may also perform replacement, merging and / or splitting of the data structure. For example, a model optimizer included in the model analyzer 100 may change the data structure.

[0028] The model analyzer 100 may send the data structure of the graph structure or the data structure of the subgraph structure to the task manager 200. According to one embodiment, the task manager 200 may generate a plurality of request signals for dividing and / or allocating tasks to a plurality of heterogeneous resources RS_1 to RS_N. For example, the task manager 200 may generate a first request signal, a second request signal, and a third request signal for executing tasks on the first resource RS_1, the second resource RS_2, and the third resource RS_3, respectively. In this case, the first request signal to the third request signal may be signals for requesting the first resource RS_1 to the third resource RS_3 to process tasks, respectively.

[0029] Specifically, the task manager 200 may generate a first request for processing at least a portion of the task contained in the data structure in the first resource RS_1, may generate a second request for processing another portion of the task contained in the data structure in the second resource RS_2, and may generate a request for processing another portion of the task contained in the data structure in the third resource RS_3.

[0030] According to one embodiment, the detector 300 may provide various types of awareness information to the task manager 200 to generate a request. For example, the awareness information may include at least one of information about each processable task of the plurality of resources RS_1 to RS_N and information about the task currently being executed by each of the plurality of resources RS_1 to RS_N, etc. The information about each processable task of the resources RS_1 to RS_N may include the type of each processable task of the resources RS_1 to RS_N, the processing time, the ability to process the task, etc. The information about the task currently being executed by each of the resources RS_1 to RS_N may include information, for example, the type of task being processed by each of the resources RS_1 to RS_N, the time required to complete the current task, the currently available capacity, the runtime information, etc.

[0031] Since the detector 300 provides the sensing information to the task manager 200, the task manager 200 may send a request signal to the compiler 400 (not shown) or the plurality of resources RS_1 to RS_N to allow the task manager 200 to process the most appropriate resource among the plurality of resources RS_1 to RS_N. That is, the task manager 200 may send a request signal to the compiler 400 when compilation is required, and may send a request signal to the resources RS_1 to RS_N when compilation is not required.

[0032] Compiler 400 can change a program written in a programming language into a language understood by a resource. For example, when task manager 200 sends a request signal for executing a task on a specific resource to compiler 400, compiler 400 can compile the task in response to the request signal to allow the task to be processed on the specific resource.

[0033] According to one embodiment, compiler 400 may include one or more subcompilers. Each subcompiler may be converted into a language that can be interpreted by each of heterogeneous resources RS_1 to RS_N. For example, compiler 400 may include a first subcompiler that can be converted into a language that can be interpreted by a first resource RS_1, and a second subcompiler that can be converted into a language that can be interpreted by a second resource RS_2. In other words, the first subcompiler compiles the task in response to receiving the first request signal so that it can be processed in the first resource RS_1, and the second subcompiler compiles the task in response to receiving the second request signal so that it can be processed in the second resource RS_2.

[0034] The operating system (OS) 20 may include various types of mobile operating systems, online operating systems, offline operating systems, etc., for example, Linux, Android based on Linux, Windows, iOS, Mac OS, Chromium OS, etc.

[0035] The memory for storing data or resources RS_1 to RS_N processed in the artificial neural network system 1 may be included inside or outside the artificial neural network system 1 or inside or outside each of the heterogeneous resources RS_1 to RS_N. The memory may be one of a volatile memory and a non-volatile memory. The non-volatile memory may include a flash memory, a phase change RAM (PRAM), a magnetic RAM (MRAM), a resistive RAM (RRAM), a ferroelectric RAM (FRAM), etc. The volatile memory may include a dynamic RAM (DRAM), a static RAM (SRAM), a synchronous DRAM (SDRAM), etc. In one embodiment, the memory may be one of a hard disk drive (HDD), a solid state drive (SSD), a compact flash (CF), a secure digital (SD), a micro secure digital (SD), an extreme digital (xD), a memory stick, etc. In addition, a direct memory access (DMA) may be included to control access to the memory, and the DMA may include, for example, a memory DMA (MDMA), a peripheral DMA (PDMA), a remote DMA (RDMA), a smart DMA (SDMA), etc.

[0036] Figure 2 FIG. 1 shows the data flow of the artificial neural network system 1 according to the embodiment. Figure 2 , although three applications APP_1 to APP_3 and two resources RS_1 and RS_2 are shown for ease of explanation, any number of applications may be used.

[0037] Reference Figure 2 , each of the applications APP_1 to APP_3 may send request information for executing an artificial neural network (ANN) model to execute instructions programmed in at least one of the applications APP_1 to APP_3. For example, the first application, the second application, and the third application APP_1 to APP_3 may send first request information, second request information, and third request information RI_1 to RI_3, respectively. The request command sent by at least one of the applications APP_1 to APP_3 may include information about the artificial neural network model and / or data about the artificial neural network model.

[0038] The model analyzer 100 may receive a request command from at least one of the applications APP_1 to APP_3 to analyze the ANN model. Figure 1As described above, the model analyzer 100 may obtain information about the model, may construct a data structure represented by a graph structure, or may change the structure into a structure suitable for processing by the heterogeneous resources RS_1 to RS_N. For example, the model analyzer 100 may process the artificial neural network model using a data structure having a graph structure or a subgraph structure suitable for processing by the heterogeneous resources RS_1 to RS_N to provide data about the graph or the subgraph to the task manager 200. In this case, the graph structure represents the entire graph structure of the artificial neural network model, and the subgraph structure represents a data structure that forms at least a part of the graph structure. For example, Figure 4 A subgraph structure as a part of the entire artificial neural network graph structure may be represented. According to one embodiment, when the model analyzer 100 does not need to change the graph structure, the model analyzer 100 may provide information about the data structure of the graph structure to the task manager 200. That is, the model analyzer 100 may provide the graph information or the subgraph information to the task manager 200.

[0039] At the same time, the detector 300 may provide the perception information AW to the task manager 200. Figure 1 As described above, the awareness information AW may include information about tasks that can be processed by each of the plurality of resources RS_1 to RS_N, information about currently executed tasks, runtime information, and the like.

[0040] The task manager 200 may receive graph information or subgraph information from the model analyzer 100 and may receive perception information AW from the detector 300. The task manager 200 may provide a request signal (RSG) requesting that resources RS_1 to RS_N be allowed to process tasks based on at least one of the graph information or subgraph information and the perception information AW.

[0041] According to one embodiment, when the task manager 200 determines based on the perception information AW that the first resource RS_1 is a resource for processing tasks for each layer (e.g., for operations in the layer direction), the task manager 200 may provide a first request signal RSG_1 to the first resource RS_1 to process the tasks for each layer. When the task manager 200 determines based on the perception information AW that the second resource RS_2 may be a resource for processing tasks for each depth direction (e.g., for operations in the depth direction), the task manager 200 may provide a second request signal RSG_2 to the second resource RS_2 to process the tasks for each depth direction.

[0042] According to one embodiment, the task manager 200 may provide a request signal (RSG) to the compiler 400. For example, the task manager 200 may send a first request signal RSG_1 to the first sub-compiler 410 for the first resource RS_1, and may send a second request signal RSG_2 to the second sub-compiler 420 for the second resource RS_2. That is, the task manager 200 may send a request signal (RSG) to the sub-compiler required for each resource RS.

[0043] As another example, the task manager 200 may provide the request signal (RSG) to the resource RS without passing through the compiler 400. Figure 1 As described above, when compilation is not required, a request signal (RSG) corresponding to each of the resources RS_1 to RS_N may be provided. For example, the task manager 200 may provide the first request signal RSG_1 directly to the first resource RS_1 and the second request signal RSG_2 directly to the second resource RS_2.

[0044] According to one embodiment, since the artificial neural network system 1 is configured to allow resources RS_1 to RS_N to perform tasks in the fastest manner based on graph information or subgraph information including information about resources RS_1 to RS_N and perception information AW, the processing speed of the artificial neural network system 1 and resources RS_1 to RS_N can be improved.

[0045] Figure 3A is a diagram for explaining general resources. Figure 3B A diagram used to explain the resources dedicated to a specific task.

[0046] Resources RS_1 to RS_N may include task-specific resources RS_D and general resources RS_G. Task-specific resources (RS_D) are computing resources (e.g., hardware designed to be suitable for processing specific tasks). For example, a neural network processing unit (NPU) may be a task-specific resource RS_D for performing computing processing tasks (e.g., convolution tasks, fully connected tasks, pooling tasks, flexible maximum tasks, etc.) according to an artificial neural network model. In contrast, general resources RS_G are resources for performing general task processing and may include CPUs, GPUs, DSPs, etc.

[0047] Reference Figure 3A , the general resource RS_G can process multiple input data to output multiple output data. That is, the general resource RS_G can support batch processing mode. In this case, the general resource RS_G can receive multiple input data at the same time, and can process the received data at the same time in the internal configuration of the general resource RS_G, or can process the received data using time division technology to output multiple output data.

[0048] Reference Figure 3B , the specific task dedicated resource RS_D cannot support the batch processing mode. That is, even if you want to process multiple input data in the specific task dedicated resource RS_D, since the specific task dedicated resource RS_D does not receive multiple input data, the data input at the same time can be limited to one.

[0049] Therefore, the processing speed decreases.

[0050] According to one embodiment, the artificial neural network system 1 may generate a recursive request signal for a specific task-specific resource RS_D that does not support the batch mode. Figure 4 Give a description.

[0051] Figure 4 1 is a flowchart for explaining the operation management method of the artificial neural network system 1 according to the embodiment. Figure 4 , as a data structure of an artificial neural network model composed of a graph structure, the artificial neural network system 1 can control each resource RS to execute each task (OP) included in the graph structure. The artificial neural network system 1 can divide the tasks (OP) into specific types of work groups 511 to 514. In addition, multiple operations S611 to S613 can be executed on the same or different resources RS, respectively.

[0052] In the processing of the general resource RS_G, the artificial neural network system 1 can provide a request signal (RSG) to the general resource RS_G. Figure 3A As in, the general resource RS_G may support a batch mode. For example, when there are six image objects to be processed using an artificial neural network model, one request signal (RSG) may be provided to the general resource RS_G, and one request signal (RSG) may include information about six input data (here, image objects).

[0053] Figure 5A and Figure 5B is a diagram for explaining the number of transmission times of a request signal according to an arrangement mode according to an embodiment.

[0054] In the artificial neural network system 1 according to the embodiment, the amount of input data may be changed according to the changed situation. Figure 5A and Figure 5B As shown in , when the artificial neural network system 1 is to identify foreground objects (OB) photographed by a camera attached to a user's vehicle, the number of objects (OB) may vary as the user's vehicle travels. For example, the objects OB may be identified as regions of interest. In this case, the artificial neural network system 1 may perform a task for identifying whether each object (OB) is a vehicle. Therefore, when Figure 5AWhen the number of objects (OBs) OB_1 to OB_6 shown in FIG. 1 is determined to be six, the number of input data is six, and the artificial neural network system 1 may send one request signal (RSG) including six input data to a resource RS supporting the batch mode (e.g., a general resource RS_G). However, the artificial neural network system 1 may send six request signals (RSG) to a resource RS not supporting the batch mode (e.g., a specific task-dedicated resource RS_D).

[0055] According to one embodiment, as the user's vehicle travels Figure 5B When the number of objects (OB) identified as shown in FIG. 1 increases to seven, the artificial neural network system 1 may send a single request signal RSG including the seven objects OB_1' to OB_7' to a resource RS supporting the batch mode (e.g., a general resource RS_G). On the other hand, the artificial neural network system 1 may send seven request signals (RSG) regarding each of the seven objects to a resource RS not supporting the batch mode (e.g., a specific task-specific resource RS_D).

[0056] Reference Figure 4 According to one embodiment, operation S611 may include tasks OP_00, OP_01, and OP_02 that are not supported by the specific task-specific resource RS_D, and the artificial neural network system 1 may generate a request signal (RSG) for processing some tasks OP_00, OP_01, and OP_02 of all tasks (OP) in the general resource RS_G to process operation S611 in the general resource RS_G. In this case, since the general resource RS_G does not perform tasks in each depth direction, the artificial neural network system 1 may provide the request signal (RSG) to the general resource RS_G to perform tasks for each layer.

[0057] Similarly, operation S613 may include tasks OP_20 and OP_21 that are not supported by the specific task-dedicated resource RS_D, and the artificial neural network system 1 may generate a request signal (RSG) for processing some tasks OP_20 and OP_21 of all tasks (OP) in the general resource RS_G to process operation S613 in the general resource RS_G.

[0058] According to one embodiment, the artificial neural network system 1 may provide the request signal (RSG) according to the result of performing operation S611 and the request signal (RSG) according to the result of performing operation S613, which are the same as or different from each other, to the general resource RS_G. For example, the artificial neural network system 1 may send the request signal (RSG) according to the result of performing operation S611 to the CPU, and may send the request signal (RSG) according to the result of performing operation S613 to the CPU or the DSP.

[0059] According to one embodiment, operation S612 may include tasks OP_11 to OP_17 supported by the specific task-specific resource RS_D, and the artificial neural network system 1 may generate a request signal (RSG) for processing some of the tasks OP_11 to OP_17 among all the tasks in the specific task-specific resource RS_D in operation S612. In this case, since the specific task-specific resource RS_D performs the task for each depth direction, the artificial neural network system 1 may provide the request signal RSG to the specific task-specific resource RS_D to perform the task for each layer or for each depth direction.

[0060] The artificial neural network system 1 may provide the request signal (RSG) to the specific task-specific resource RS_D as many times as the number of input data in the operation processed in the specific task-specific resource RS_D. That is, when the number of input data is N, the artificial neural network system 1 may provide the request signal (RSG) N times. Figure 3B As shown in , the specific task-specific resource RS_D cannot support the batch mode. For example, when the number of image objects to be processed by the artificial neural network model is six, six request signals (RSG) may be provided to the specific task-specific resource RS_D, and each of the six request signals (RSG) may include information about six input data (here, image objects). In this case, the task manager 200 may repeatedly send the request signal (RSG) to the specific task-specific resource RS_D in a recursive loop form during six times.

[0061] According to one embodiment, the task manager 200 may determine the resource RS for executing each task (OP) based on the information about the received neural network model. For example, the task manager 200 may receive subgraph information from the model analyzer 100, and may analyze the task (OP) included in the received information to determine the compatible resource RS for the task (OP). For example, the task manager 200 may determine that the task OP_01 is not executed by the NPU.

[0062] According to one embodiment, the artificial neural network system 1 may divide the tasks into a plurality of task (or work / task) groups. Specifically, the artificial neural network system 1 may divide the tasks OP_11 to OP_17 to be executed in a specific task-specific resource RS_D into at least one of the work groups 511 to 514. In the case of at least one operation S612 among the plurality of operations S611 to S613 that can perform processing for each depth direction, the artificial neural network system 1 may divide the tasks OP_11 to OP_17 to process each of the work groups 511 to 514 in parallel. For example, the work group 511 may include tasks OP_11 and OP_12, the work group 512 may include tasks OP_13 and OP_14, the work group 513 may include tasks OP_15 and OP_16, and the work group 514 may include task OP_17. That is, the artificial neural network system 1 may generate the work groups 511 to 514 so that the work processing can be performed in parallel.

[0063] Tasks OP_11, OP_13, and OP_15 may be processed in the same layer of the neural network model. In other words, executing a task for each layer may mean executing tasks OP_12, OP_14, OP_16, and OP_17 processed in the same layer after tasks OP_11, OP_13, and OP_15 processed in the same layer are executed. On the other hand, executing a task for each depth direction may mean executing tasks OP_13 and OP_14 placed in the same depth direction after tasks OP_11, OP_12 placed in the same depth direction are executed.

[0064] Figure 6 is a diagram for explaining a method for processing a task in an artificial neural network model according to an embodiment. Figure 6 , each of the first multilayer ST_1 and the second multilayer ST_2 may include three layers, and each of the third to fifth multilayers ST_3 to ST_5 may include four layers. However, the number of multilayers and the number of layers may be variously set.

[0065] Each layer can handle various tasks performed in artificial neural networks, such as convolution (CONV), full connection, pooling (POOL), flexible max, etc.

[0066] According to one embodiment, performing a task layer by layer means that the output of a specific layer is received as an input on a next layer of the specific layer and the next layer performs an operation. In addition, performing a task for each depth direction means that the task results of all layers included in a specific depth direction are processed as input values ​​of all layers in the next depth direction of the specific depth direction to perform the task in the next depth direction.

[0067] At the same time, refer again Figure 4According to one embodiment, the task manager 200 may generate an operation or a task / work group based on the perception information AW and the subgraph information. For example, when the available capacity of the CPU is insufficient, in operation S611, the task manager 200 may generate a request signal (RSG) to be processed by the GPU instead of the CPU based on the perception information AW.

[0068] The task manager 200 may generate more or fewer work groups based on the sub-graph information. Figure 4 As shown in , the task manager 200 can create three working groups, but if there are many parallel tasks (OPs) in the graph structure or subgraph structure, more working groups can be created. Figure 4 , 14 tasks (OPs) are shown for ease of illustration, and any number of tasks (OPs) may be used.

[0069] The task manager 200 may generate various numbers of work groups and various numbers of operations. The task manager 200 may generate a request signal (RSG) for dynamically allocating each operation to one or more of the heterogeneous resources RS_1 to RS_N.

[0070] Figure 7 1 is a flowchart for explaining the operation management method of the artificial neural network system 1 according to the embodiment. Figure 7 ,and Figure 4 Differently, the task manager 200 may generate eight working groups 521 to 524 and 531 to 534. Figure 4 Differently, the task manager 200 may classify tasks OP_10 to OP_18 into operation S621, and may classify tasks OP_30 to OP_38 into operation S622. Operation S621 may include tasks OP_30 to OP_38 performed by a specific task-specific resource RS_D.

[0071] According to one embodiment, in operation S621 and operation S622, the task manager 200 may send a request signal (RSG) to the specific task dedicated resource RS_D to perform tasks for each layer and for each depth direction. That is, the specific task dedicated resource RS_D may perform work for the work group 521, the work group 522, and the work group 523 for the depth direction during the first time period, and may perform work for the layer from the work group 522 and the work group 523 to the work group 524 during the second time period. Therefore, the specific task dedicated resource RS_D may perform work for the work group 521 and the work group 524 for each depth direction. This is because the specific task dedicated resource RS_D may perform tasks layer by layer and may also perform tasks depth by depth direction.

[0072] According to an example embodiment, the framework 10 may further include a model holder, and the task manager 200 may store request signals for some of the operations in the model holder. The model holder may be implemented as described above in Figure 1 . The task manager 200 may receive subgraph information from the model analyzer 100, and may determine that at least one of the tasks is equal to each other. For example, the task manager 200 may determine that each of the tasks OP_10 to OP_18 and each of the tasks OP_30 to OP_38 are tasks of the same kind as each other, and the order in which the tasks are processed is also the same. In this case, the task manager 200 may reuse the request signal of operation S612 without performing a separate generation of a request signal for operation S622. That is, the task manager 200 may store request signals for some of the operations. If the task manager 200 determines that the operation that generates the request signal is the same as some of the operations, the task manager 200 may load to output the stored request signal.

[0073] According to one embodiment, the task manager 200 may send a request signal (RSG) for executing each of the work groups 521 to 524 to one or more of the heterogeneous resources RS_1 to RS_N. For example, the task manager 200 sends a request signal (RSG) to each NPU so that the work group 521 is assigned to the first NPU, the work group 522 is assigned to the second NPU, and the work group 523 is assigned to the third NPU. The task manager 200 may determine that a specific task-specific resource RS_D performs work for each layer and / or each depth direction based on the perception information AW.

[0074] In this case, the task manager 200 may determine that the available capacity of the third NPU is sufficient based on the sensing information AW, and may allocate the work group 524 to one of the resources RS to which the previous layer is allocated. For example, when the third NPU to which the work group 523 is allocated is larger than the second NPU to which the work group 522 is allocated based on the sensing information AW, the task manager 200 may send a request signal (RSG) to the third NPU to process the work group 524. As another example, when the third NPU is occupied by another work group based on the sensing information AW, the task manager 200 may send a request signal (RSG) for processing the work group 524 to the second NPU having more available capacity.

[0075] According to one embodiment, when there are no more specific task-specific resources RS_D for processing the work group 524, the task manager 200 may send a request signal (RSG) to the general resource RS_G. When the available capacity of all specific task-specific resources RS_D communicating with the artificial neural network system 1 is insufficient, the request signal (RSG) may be sent to the general resource RS_G. For example, for the work groups 521 to 523, unlike the work group 524, there is no need to process the work group for each depth direction. Therefore, the task manager 200 may request the general resource RS_G to process the work groups 521 to 523 by providing the request signal (RSG) to the general resource RS_G.

[0076] According to one embodiment, when the number of input data is N, the task manager 200 may send a request signal (RSG) to a specific task-specific resource RS_D N times. For example, the task manager 200 may determine that the work group 521 is the tasks OP_11 and OP_12 to be processed by the first resource RS_1. Figure 5A When the task manager 200 selects the features of the six regions of interest of the object of the type shown in , the task manager 200 may send the request signal (RSG) to the first resource RS_1 6 times. In this case, the request signal (RSG) sent six times may be the request signal (RSG) stored in the model holder. In addition, the task manager 200 may use a recursive statement or a loop function to send the same request signal (RSG) to a specific resource RS six times.

[0077] Meanwhile, operation S621 and tasks OP_10 to OP_18 including the above-described tasks OP_10 to OP_18 may respectively have diagrams similar to operation S622 and tasks OP_30 to OP_38 including tasks OP_30 to OP_38, and thus, the above-described embodiments may also be similarly applied.

[0078] Figure 8 1 is a flowchart for explaining the operation management method of the artificial neural network system 1 according to the embodiment. Figure 8 The task manager 200 receives the subgraph information and may divide the work group in the same subgraph structure in various ways. For example, the task manager 200 may divide the work group into two in operation S621. Figure 7 In operation S621, the task manager 200 may divide the work groups 521 to 524 into four tasks. Figure 8 As shown in FIG. 5 , in operation S621 , the task manager 200 may divide the work group 541 and the work group 542 into two tasks.

[0079] According to one embodiment, the task manager 200 may provide a request signal (RSG) to process the divided work group 541 in the first NPU, and may provide a request signal (RSG) to process the work group 542 in the second NPU. In addition, task OP_10 and task OP_18 may be requested to be processed by the first NPU or the second NPU, respectively.

[0080] According to one embodiment, Figure 7 In operation S622, the task manager 200 may divide the work group 543 and the work group 544 to process all tasks layer by layer. The work groups 543 and 544 divided for each layer may be assigned to the general resource RS_G or the specific task-specific resource RS_D. For example, the task manager 200 may provide a request signal (RSG) to process the work group 543 in the third NPU, and may provide a request signal (RSG) to process the work group 544 in the fourth NPU. In other words, the task manager 200 may provide a request signal (RSG) to the specific task-specific resource RS_D to process the task for each layer.

[0081] Fig. 9 1 is a flowchart for explaining the operation management method of the artificial neural network system 1 according to the embodiment. Fig. 9 , the task manager 200 may receive the subgraph information and may provide a request signal (RSG) to allow one or more heterogeneous resources to process all tasks OP_40 to OP_48, OP_50 to OP_58, and OP_60 to OP_62 included in the subgraph information of each layer in each of operations S631 to S633. In addition, the task manager 200 may assign work groups 551 and 552 to process all tasks included in the subgraph layer by layer. For example, the task manager 200 may use the first NPU in operation S631, may use the second NPU in operation S632, and may send a request signal (RSG) to each resource RS of the first CPU for processing in operation S633.

[0082] In this case, the task manager 200 may provide a request signal (RSG) for processing a task of a specific operation in the same resource RS as a resource RS for processing a task of an operation adjacent to the specific operation based on the perception information AW. For example, since the task manager 200 receives the perception information AW to process the task for processing fingerprint recognition by the second NPU, the task manager 200 may determine that the available capacity of the second NPU is insufficient. According to the result of the determination, the task manager 200 does not send a request signal (RSG) to the second NPU, but may send a request signal (RSG) for a work group 552 for processing operation S632 on a resource RS for processing operation S631 or operation S633 adjacent to operation S632. For example, the task manager 200 may send a request signal (RSG) to the first NPU processing operation S631, or may send a request signal (RSG) to the first CPU processing operation S633.

[0083] According to the above embodiment, the task manager 200 may execute a task on the same resource as a neighboring operation, thereby improving the working efficiency among the plurality of heterogeneous resources RS_1 to RS_N.

[0084] Fig.10 : is a flowchart for explaining a work management method executed by the artificial neural network system 1 according to the embodiment.

[0085] In operation S710, the work management method may receive an artificial neural network model to provide information about a subgraph structure for modifying a graph structure of the artificial neural network model. In this case, the model analyzer 100 may optimize the graph structure of the artificial neural network into a subgraph structure through the various methods described above, and may provide subgraph information about the subgraph structure. For example, as used to explain Figure 1 As shown in the example of , the model parser, model builder, and model optimizer included in the model analyzer 100 may provide subgraph information.

[0086] In operation S720, perception information AW related to the state of heterogeneous resources RS_1 to RS_N may be provided. The perception information AW may include at least one of information about tasks to be processed by each of the heterogeneous resources RS_1 to RS_N and information about tasks performed by each of the heterogeneous resources RS_1 to RS_N. For example, the perception information may include information about tasks that can be processed by each of the multiple resources RS_1 to RS_N, information about tasks currently being executed, and the like. Information about tasks processed by each of the resources RS_1 to RS_N may include, for example, the type of tasks that can be processed by each of the resources RS_1 to RS_N, the processing time of the tasks, the capabilities required to process the tasks, and the like. Information about tasks currently being performed by each of the resources RS_1 to RS_N may include information, for example, the type of tasks being processed by each of the resources RS_1 to RS_N, the time required to complete the current task, the currently available capabilities, runtime information, and the like.

[0087] When the task manager 200 determines that one or more of the heterogeneous resources RS_1 to RS_N cannot execute the divided tasks based on the received awareness information AW, the task manager 200 may re-divide the plurality of tasks.

[0088] In operation S730 , a request signal (RSG) may be provided to one or more of the heterogeneous resources RS_1 to RS_N to perform a task based on the subgraph information and the awareness information AW.

[0089] According to one embodiment, when providing the request signal (RSG), the task manager 200 may divide the plurality of tasks contained in the subgraph based on the subgraph information, and may provide a different request signal (RSG) to process the divided tasks in each of the heterogeneous resources RS_1 to RS_N. In this case, the division may include at least one of a first division for dividing the plurality of tasks for each layer and a second division for dividing in a depth direction.

[0090] According to one embodiment, when providing a request signal (RSG), the task manager 200 may provide a first request signal (RSG) to one or more of the heterogeneous resources RS_1 to RS_N to perform the task for each layer based on the sub-graph information and the perception information AW, or may provide a second request signal (RSG) to another one of the heterogeneous resources RS_1 to RS_N to perform the task depth by depth.

[0091] According to one embodiment, the task manager 200 may repeatedly provide a request signal (RSG) by the number of received input data to process the task for each depth-wise partition in the specific resource RS. In this case, the specific resource RS may be a resource that does not support a batch mode.

[0092] In one example embodiment, according to a method for managing tasks of an artificial neural network driven by a plurality of heterogeneous resources and a system including the method, tasks are efficiently dispatched to the plurality of heterogeneous resources to increase operation processing speed.

[0093] In one example embodiment, a method is provided for efficiently allocating tasks to resources that perform layer-directional operation processing and resources that perform depth-directional operation processing, thereby increasing the operation processing speed of an electronic system.

[0094] In an example embodiment, resources that do not support batch mode also support dynamic batch mode, and operation processing of resources that do not support batch mode is efficiently performed.

[0095] According to functional blocks, units, modules and / or methods, embodiments are described and shown in the accompanying drawings. It will be understood by those skilled in the art that these blocks, units, modules and / or methods are physically implemented by electronic (or optical) circuits (such as, logic gates, discrete components, microprocessors, hard-wired circuits, memory elements, wiring connections, etc.), wherein electronic (or optical) circuits can be formed using semiconductor-based manufacturing techniques or other manufacturing techniques. In the case where blocks, units, modules and / or methods are implemented by microprocessors or the like, they can be programmed using software (e.g., microcode) to perform the various functions discussed herein, and can be optionally driven by firmware and / or software. Optionally, each block, unit, module and / or method can be implemented by dedicated hardware or can be implemented by a combination of dedicated hardware for performing some functions and processors (e.g., one or more programmed microprocessors and associated circuits) for performing other functions. In addition, without departing from the scope of the disclosure, each block, unit and / or module of the embodiment can be physically divided into two or more interactive and discrete blocks, units and / or modules. Furthermore, the blocks, units and / or modules of the embodiments may be physically combined into more complex blocks, units and / or modules without departing from the scope of the disclosure.

[0096] Example embodiments have been disclosed herein, and although specific terms are employed, they are used and interpreted only in a general and descriptive manner and not for purposes of limitation. In some cases, as will be apparent to one of ordinary skill in the art when submitting this application, unless otherwise specifically noted, features, characteristics, and / or elements described in conjunction with a particular embodiment may be used alone or in combination with features, characteristics, and / or elements described in conjunction with other embodiments. Therefore, it will be understood by those skilled in the art that various changes may be made in form and detail without departing from the spirit and scope of the invention as set forth in the claims.

Claims

1. A system for executing a task by executing an artificial neural network through multiple heterogeneous resources, the system comprising: A model analyzer receives an artificial neural network model and outputs subgraph information generated based on the artificial neural network model including at least one subgraph; A detector, outputting sensing information about the plurality of heterogeneous resources; as well as a task manager that outputs a first request signal for executing a task for each layer to a first resource among the plurality of heterogeneous resources based on the subgraph information and the perception information, and outputs a second request signal for executing a task for each depth direction to a second resource among the plurality of heterogeneous resources, The task manager further divides the plurality of tasks included in the at least one subgraph in response to the subgraph information. Among them, performing a task for each layer means that the output of a specific layer is received as an input on the next layer of the specific layer and the next layer performs an operation, Among them, executing a task for each depth direction means that the task results of all layers included in a specific depth direction are processed as input values ​​of all layers included in the next depth direction of the specific depth direction to execute the task in the next depth direction.

2. The system of claim 1, wherein: The task manager also outputs at least one of a first request signal and a second request signal to process the divided tasks through the plurality of heterogeneous resources.

3. The system of claim 2, wherein: The division of the task manager includes at least one of dividing the plurality of tasks for each layer as a first task and dividing the plurality of tasks for each depth direction as a second task.

4. The system of claim 3, wherein: The task manager also merges the second task into the first task or merges the first task into the second task.

5. The system of claim 2, wherein: The perception information includes at least one of information about tasks processable by each of the plurality of heterogeneous resources and information about tasks being executed by each of the plurality of heterogeneous resources.

6. The system of claim 2, wherein: The task manager repeatedly provides the second request signal based on the amount of received input data.

7. The system of claim 6, wherein: The number of received input data includes the number of objects corresponding to the region of interest recognized by the visual recognition.

8. The system of claim 2, wherein: When at least one of the plurality of heterogeneous resources is unable to execute the divided tasks based on the sensing information, the task manager divides the plurality of tasks again.

9. The system of claim 2, wherein: The task manager provides at least one of a first request signal and a second request signal to process the first task at the same resource as a second task at a next stage of the first task based on the sensing information.

10. The system of claim 1, further comprising: The compiler, in response to receiving at least one of a first request signal and a second request signal from the task manager, allows the plurality of heterogeneous resources to process the task.

11. The system of claim 10, wherein: The compiler includes a first sub-compiler and a second sub-compiler; The first sub-compiler compiles the task in response to receiving the first request signal so as to be processed in the first resource; as well as The second sub-compiler compiles the task in response to receiving the second request signal so as to be processed in the second resource.

12. The system of claim 1, wherein: Each of the first resource and the second resource performs an operation using an artificial neural network including a plurality of layers forming a depth direction, The first resource performs an operation using the output value from the first layer as the input value of the second layer; as well as The second resource performs an operation using the output value in the first depth direction as an input value in the second depth direction.

13. The system of claim 1, wherein: The first resource is a batch mode resource that supports multiple inputs at once. The second resource is a resource that does not support batch mode, and The second resource processes operations based on an artificial neural network model.

14. A method for executing a task by executing an artificial neural network through multiple heterogeneous resources, the method comprising: receiving an artificial neural network model, and outputting subgraph information generated based on the artificial neural network model; Outputting perception information about the plurality of heterogeneous resources; performing a division operation of a plurality of tasks included in the sub-graph information based on the sub-graph information; as well as outputting a request signal for processing the plurality of tasks on which the division operation is performed in one or more of the plurality of heterogeneous resources based on the sensing information, The step of outputting a request signal includes: based on the subgraph information and the perception information, outputting a first request signal for performing a task for each layer to a first resource among the plurality of heterogeneous resources; and outputting a second request signal for performing a task for each depth direction to a second resource among the plurality of heterogeneous resources, Among them, performing a task for each layer means that the output of a specific layer is received as an input on the next layer of the specific layer and the next layer performs an operation, Among them, executing a task for each depth direction means that the task results of all layers included in a specific depth direction are processed as input values ​​of all layers included in the next depth direction of the specific depth direction to execute the task in the next depth direction.

15. The method of claim 14, wherein: The step of outputting the request signal includes repeatedly providing a second request signal based on the amount of received input data.

16. The method of claim 14, wherein: The division operation includes at least one of a first division operation for dividing the plurality of tasks for each layer and a second division operation for dividing the plurality of tasks for each depth direction.

17. The method of claim 14, wherein: The perception information includes at least one of information about tasks processable by each of the plurality of heterogeneous resources and information about tasks being executed by each of the plurality of heterogeneous resources.

18. The method of claim 17, wherein: When at least one of the plurality of heterogeneous resources is unable to perform the divided tasks based on the sensing information, the step of outputting a request signal includes performing a division operation on the plurality of tasks again.

19. A system for executing a task by executing an artificial neural network through multiple heterogeneous resources, the system comprising: A model analyzer receives an artificial neural network model and outputs subgraph information generated based on the artificial neural network model including at least one subgraph; as well as a task manager that divides a plurality of tasks included in the at least one subgraph based on the subgraph information to execute a task for each depth direction and to execute a task for each layer, provides a first request signal to a resource that does not support the batch mode among the plurality of heterogeneous resources to execute the task for each depth direction, provides N times the first request signal to execute the task for each depth direction, and provides a second request signal to another resource among the plurality of heterogeneous resources to execute the task for each layer, wherein N is the number of input data, Among them, performing a task for each layer means that the output of a specific layer is received as an input on the next layer of the specific layer and the next layer performs an operation, Among them, executing a task for each depth direction means that the task results of all layers included in a specific depth direction are processed as input values ​​of all layers included in the next depth direction of the specific depth direction to execute the task in the next depth direction.

Images