Electronic device and control method therefor

Abstract

This electronic device comprises: a memory including instructions; a first processor including a first internal memory and a first core for performing an operation related to a neural network model; and a second processor including a second internal memory and a second core for performing an operation related to digital signal processing. The instructions, when executed by the first and second processors, cause the electronic device to: acquire information about an instruction sequence for performing an operation related to a neural network model; acquire information about memory usage and use of slots per hour used by the instruction sequence; acquire information about a data flow graph from the instruction sequence; acquire information about a plurality of candidate spill patterns for spilling, to the second internal memory, intermediate result values acquired through operations on the basis of the data flow graph; identify information about memory usage and use of slots due to the plurality of candidate spill patterns on the basis of the information about the memory usage and use of slots per hour used by the instruction sequence; and determine whether to spill, to the second internal memory, the intermediate result values acquired by means of the plurality of candidate spill patterns on the basis of the information about the memory usage and use of slots due to the plurality of candidate spill patterns.

Description

Electronic device and method of controlling the same

[0001] The present disclosure relates to an electronic device and a control method thereof, and more particularly, to an electronic device and a control method thereof for optimizing data transmission using on-chip memory in a processor including heterogeneous cores.

[0002] Recently, increased computing power has been required to process operations related to neural network models. In particular, this demand for improved computing performance has led electronic devices to utilize two or more heterogeneous cores to perform various operations. Multicore processors utilizing two or more heterogeneous cores offer the advantages of fast processing speeds and low power consumption.

[0003] In particular, recently, a single processor chip includes an artificial intelligence core (e.g., a Deep Learning Accelerator (DLA)) for processing operations related to artificial intelligence and a general-purpose core (e.g., a Digital Signal Processor (DPS)) for processing operations related to digital signal processing.

[0004] The above information may be provided as background art to aid in understanding the present disclosure. No claim or determination is made as to whether any of the above is applicable as prior art related to the present disclosure.

[0005] According to one embodiment of the present disclosure, an electronic device comprises: a memory including instructions; And a first processor including a first core for performing operations related to a first internal memory and a neural network model, and a second processor including a second core for performing operations related to a second internal memory and a digital signal processing; wherein the instructions, when executed by the first and second processors, cause the electronic device to obtain information about an instruction sequence for performing operations related to a neural network model, obtain information about memory usage per hour and slot usage used by the instruction sequence, obtain information about a data flow graph from the instruction sequence, obtain information about a plurality of candidate spill patterns for spilling intermediate result values ​​obtained through calculations based on the data flow graph into the second internal memory, identify information about memory usage and slot usage by the plurality of candidate spill patterns based on the information about memory usage per hour and slot usage used by the instruction sequence, and determine whether the memory usage by the plurality of candidate spill patterns is less than or equal to the second internal memory size and whether the plurality of candidate spill patterns are different from the second internal memory size. It is determined whether to spill intermediate result values ​​obtained by the plurality of candidate spill patterns into the second internal memory based on whether the slot usage of the pattern is limited.

[0006] The above instructions, when executed by the first and second processors, may cause the electronic device to identify a plurality of operators included in the instruction sequence, among the plurality of operators having the same feature map ID and including a store operator, a concatenation operator, and a load operator, as candidate spill patterns.

[0007] The instructions, when executed by the first and second processors, may cause the electronic device to sort the plurality of candidate spill patterns in a non-ascending order based on the number of times the candidate spill patterns are used, and to identify, for each of the plurality of candidate spill patterns sorted in the non-ascending order, whether to sequentially spill an intermediate result value obtained by the candidate spill pattern to the second internal memory.

[0008] The above instructions, when executed by the first and second processors, may cause the electronic device to preferentially sort candidate spill patterns having a larger feature map size among candidate spill patterns having the same number of uses among the plurality of candidate spill patterns, if there are candidate spill patterns having the same number of uses.

[0009] The instructions, when executed by the first and second processors, may cause the electronic device to identify an intermediate result obtained by the first candidate spill pattern to spill into the second internal memory if the memory usage by the first candidate spill pattern is less than or equal to the second internal memory size, and to identify an intermediate result obtained by the second candidate spill pattern to store in the memory if the memory usage by the second candidate spill pattern exceeds the second internal memory size.

[0010] The instructions, when executed by the first and second processors, may cause the electronic device to identify whether an on-chip communication slot for spilling data stored in the first internal memory to the second internal memory by another instruction at the time of executing the third candidate spill pattern already exists, thereby identifying whether slot utilization of the third candidate spill pattern is restricted.

[0011] The instructions, when executed by the first and second processors, may cause the electronic device to identify, if the on-chip communication slot does not exist at the time of executing the fourth candidate spill pattern, to spill an intermediate result obtained by the fourth candidate spill pattern into the second internal memory, and to identify, if the on-chip communication slot exists at the time of executing the fifth candidate spill pattern, to store an intermediate result obtained by the fifth candidate spill pattern in the memory.

[0012] The instructions, when executed by the first and second processors, may cause the electronic device to identify whether the plurality of candidate spill patterns are spills, change the storage location of intermediate result values ​​obtained by at least some of the plurality of candidate spill patterns to the second internal memory based on the identification result, and update information on memory usage per hour and slot usage used by the instruction sequence based on the change result.

[0013] The information about the above instruction sequence may include at least one of information about the time at which an instruction included in the instruction sequence is executed, information about an operator code of the instruction, information about data required for an operation in the instruction, and slot usage information of the instruction.

[0014] According to one embodiment of the present disclosure, a control method of an electronic device including a first processor including a first core for performing operations related to a memory, a first internal memory, and a neural network model, and a second processor including a second core for performing operations related to digital signal processing, the control method comprising: obtaining information about an instruction sequence for performing operations related to a neural network model; obtaining information about memory usage per hour and slot usage used by the instruction sequence; obtaining information about a data flow graph from the instruction sequence; obtaining information about a plurality of candidate spill patterns for spilling intermediate result values ​​obtained through an operation based on the data flow graph into the second internal memory; identifying information about memory usage and slot usage by the plurality of candidate spill patterns based on the information about the memory usage per hour and slot usage used by the instruction sequence; And a step of identifying whether to spill intermediate result values ​​obtained by the plurality of candidate spill patterns into the second internal memory based on whether the memory usage by the plurality of candidate spill patterns is less than or equal to the second internal memory size and whether slot usage of the plurality of candidate spill patterns is limited.

[0015] The step of obtaining information on the plurality of candidate spill patterns may identify, among the plurality of operators included in the instruction sequence, a plurality of operators having the same feature map ID and including a store operator, a concatenation operator, and a load operator as candidate spill patterns.

[0016] The control method further includes a step of sorting the plurality of candidate spill patterns in a non-ascending order based on the number of times the candidate spill patterns are used; and the step of identifying whether to spill to the second internal memory may identify, for each of the plurality of candidate spill patterns sorted in the non-ascending order, whether to sequentially spill an intermediate result value obtained by the candidate spill pattern to the second internal memory.

[0017] The above sorting step may, if there are candidate spill patterns with the same number of uses among the plurality of candidate spill patterns, preferentially sort candidate spill patterns with a larger feature map size among the candidate spill patterns with the same number of uses.

[0018] The step of identifying whether to spill to the second internal memory may include identifying the intermediate result value obtained by the first candidate spill pattern to spill to the second internal memory if the memory usage by the first candidate spill pattern is less than or equal to the second internal memory size, and identifying the intermediate result value obtained by the second candidate spill pattern to store in the memory if the memory usage by the second candidate spill pattern exceeds the second internal memory size.

[0019] The step of identifying whether to spill to the second internal memory may include identifying whether an on-chip communication slot for spilling data stored in the first internal memory to the second internal memory by another instruction at the time of executing the third candidate spill pattern already exists, thereby identifying whether slot usage of the third candidate spill pattern is restricted.

[0020] The step of identifying whether to spill to the second internal memory may include identifying the intermediate result value obtained by the fourth candidate spill pattern to spill to the second internal memory if the on-chip communication slot does not exist at the time of executing the fourth candidate spill pattern, and identifying the intermediate result value obtained by the fifth candidate spill pattern to store in the memory if the on-chip communication slot exists at the time of executing the fifth candidate spill pattern.

[0021] The processor may include: a step of identifying whether the plurality of candidate spill patterns are spills, and changing the storage location of intermediate result values ​​obtained by at least some of the plurality of candidate spill patterns to the second internal memory based on the identification result; and a step of updating information on memory usage per hour and slot usage used by the instruction sequence based on the change result.

[0022] The information about the above instruction sequence may include at least one of information about the time at which an instruction included in the instruction sequence is executed, information about an operator code of the instruction, information about data required for an operation in the instruction, and slot usage information of the instruction.

[0023] In connection with the description of the drawings, the same or similar reference numerals may be used for the same or similar components.

[0024] FIG. 1 is a block diagram showing the configuration of an electronic device according to one embodiment of the present disclosure.

[0025] FIG. 2 is a diagram illustrating data movement between components of an electronic device according to one embodiment of the present disclosure.

[0026] Figure 3 is a drawing for explaining the operation of a DLA core in a conventional NPU chip.

[0027] FIG. 4 is a diagram for explaining the operation of a DLA core in an NPU chip according to one embodiment of the present disclosure.

[0028] FIG. 5A is a diagram illustrating overflow when an intermediate result value obtained by a first core spills into a second internal memory according to one embodiment of the present disclosure.

[0029] FIG. 5b is a diagram for signing time slot usage information of instructions included in an instruction sequence according to one embodiment of the present disclosure.

[0030] FIGS. 6A to 6C are diagrams illustrating a method for obtaining an optimal spill plan with the greatest benefit according to one embodiment of the present disclosure.

[0031] FIG. 7 is a flowchart illustrating a process in which a compiler of an NPU allocates resources for an instruction sequence related to a neural network model according to one embodiment of the present disclosure.

[0032] FIG. 8 is a flowchart illustrating a process of updating an instruction sequence by identifying whether a plurality of candidate spill patterns included in the instruction sequence are spills, according to one embodiment of the present disclosure.

[0033] FIG. 9 is a diagram illustrating an instruction sequence according to one embodiment of the present disclosure.

[0034] FIG. 10A and FIG. 10B are diagrams illustrating information about memory usage and slot usage per hour of an instruction sequence according to one embodiment of the present disclosure.

[0035] FIG. 11 is a diagram illustrating a data flow graph corresponding to an instruction sequence according to one embodiment of the present disclosure.

[0036] FIG. 12 is a diagram illustrating a process for identifying a candidate spill pattern based on a data flow graph according to one embodiment of the present disclosure.

[0037] FIG. 13 is a diagram illustrating a process of sorting a plurality of candidate spill patterns based on the number of times the candidate spill patterns are used, according to one embodiment of the present disclosure.

[0038] FIG. 14 is a flowchart illustrating a process for identifying whether a candidate spill pattern is spilled, according to one embodiment of the present disclosure.

[0039] FIG. 15 is a diagram illustrating an updated instruction sequence according to one embodiment of the present disclosure.

[0040] The present embodiments may be modified and have various embodiments. Specific embodiments are illustrated in the drawings and described in detail in the detailed description. However, this is not intended to limit the scope to specific embodiments, but should be understood to encompass various modifications, equivalents, and / or alternatives of the embodiments of the present disclosure. In connection with the description of the drawings, similar reference numerals may be used for similar components.

[0041] In describing the present disclosure, if it is determined that a specific description of a related known function or configuration may unnecessarily obscure the gist of the present disclosure, a detailed description thereof will be omitted.

[0042] Additionally, the following embodiments may be modified in various other forms, and the scope of the technical concepts of the present disclosure is not limited to the following embodiments. Rather, these embodiments are provided to further faithfully and completely convey the technical concepts of the present disclosure to those skilled in the art.

[0043] The terminology used in this disclosure is solely for the purpose of describing specific embodiments and is not intended to limit the scope of the rights. Singular expressions include plural expressions unless the context clearly dictates otherwise.

[0044] In this disclosure, expressions such as “has,” “can have,” “includes,” or “may include” indicate the presence of a corresponding feature (e.g., a component such as a number, function, operation, or part), and do not exclude the presence of additional features.

[0045] In this disclosure, expressions such as “A or B,” “at least one of A and / or B,” or “one or more of A or / and B” can include all possible combinations of the listed items. For example, “A or B,” “at least one of A and B,” or “at least one of A or B” can all refer to (1) including at least one A, (2) including at least one B, or (3) including both at least one A and at least one B.

[0046] The expressions “first,” “second,” “first,” or “second,” etc., used in this disclosure can describe various components, regardless of order and / or importance, and are only used to distinguish one component from another, but do not limit the components.

[0047] When it is said that a component (e.g., a first component) is “(operatively or communicatively) coupled with / to” or “connected to” another component (e.g., a second component), it should be understood that said component may be directly coupled to said other component, or may be coupled via another component (e.g., a third component).

[0048] On the other hand, when it is said that a component (e.g., a first component) is "directly connected" or "directly connected" to another component (e.g., a second component), it can be understood that no other component (e.g., a third component) exists between said component and said other component.

[0049] The expression "configured to" as used in the present disclosure may be used interchangeably with, for example, "suitable for," "having the capacity to," "designed to," "adapted to," "made to," or "capable of." The term "configured to" may not necessarily mean only "specifically designed to" in terms of hardware.

[0050] Instead, in some contexts, the phrase "a device configured to" may mean that the device, in conjunction with other devices or components, is "capable of" performing A, B, and C. For example, the phrase "a processor configured (or set) to perform A, B, and C" may refer to a dedicated processor (e.g., an embedded processor) for performing those operations, or a general-purpose processor (e.g., a CPU or application processor) that can perform those operations by executing one or more software programs stored in a memory device.

[0051] In the embodiments, a 'module' or 'part' performs at least one function or operation, and may be implemented as hardware or software, or as a combination of hardware and software. Furthermore, a plurality of 'modules' or 'parts' may be integrated into at least one module and implemented as at least one processor, except for a 'module' or 'part' that needs to be implemented as a specific hardware.

[0052] Meanwhile, the various elements and areas in the drawings are schematically drawn. Therefore, the technical concept of the present invention is not limited by the relative sizes or spacing depicted in the attached drawings.

[0053] Hereinafter, the present disclosure will be described in more detail with reference to the drawings.

[0054] FIG. 1 is a block diagram illustrating a configuration of an electronic device according to one embodiment of the present disclosure. As illustrated in FIG. 1, the electronic device (100) may include memory (110) and processor(s) (120). The components, their relationships, and their functions illustrated in FIG. 1 are merely exemplary and do not limit the implementations described or claimed in this document. The electronic device (100) may be implemented as, but is not limited to, a mobile device, a user device, a multi-function device, a portable device, or a server.

[0055] The memory (110) may include one or more storage media (or one or more storage devices). For example, the memory (110) may include a memory assembly including one or more storage media. For example, the one or more storage media may include permanent memory (e.g., non-volatile memory) such as a hard drive, flash memory, read-only memory (ROM), semi-permanent memory (e.g., volatile memory) such as random access memory (RAM), any other suitable type of storage (or storage assembly), or any combination thereof. The memory (110) may be fixedly embedded within the electronic device (100) or incorporated into one or more suitable types of components (e.g., a subscriber identity module (SIM) card and / or a secure digital (SD) card) that can be repeatedly inserted into and removed from the electronic device (100).

[0056] For example, the memory (110) may store one or more software applications, such as an operating system (or system) software application, a firmware software application, a driver software application, a plug-in (e.g., add-in, add-on, and / or applet) software application, and / or any other suitable software applications. For example, the one or more software applications may include instructions executable by the processor (120). For example, the memory (110) may store instructions callable by an application programming interface (API). For example, the memory (110) may store instructions within a library.

[0057] The processor (120) may be implemented as one or more IC (integrated circuit (or circuitry)) chips and may perform various data processing. The processor (120) may include at least one electrical circuit and may individually or collectively perform distributed processing of instructions (or programs, data) stored in the memory (110). The processor (120) may include a processor assembly including one or more processing circuits. The processor (120) may include any processing circuit operative to control the performance and operations of one or more components (e.g., the memory (110), a microphone, a display, an image sensor, a communication circuit, a sensor, and / or a speaker) of the electronic device (100). For example, the processor (120) (e.g., an application processor (AP)) may be implemented as a system on chip (SoC) (e.g., a single chip or a chipset). For example, the processor (120) may be implemented with multiple cores (or at least one core circuit), multiple chips, or multiple chipsets. For example, the processor (120) may include one or more processing circuits. For example, the processor (120) may include one or more processing circuits configured to individually and / or collectively perform various functions of the present disclosure. As a non-limiting example, at least a portion of the processor (120) may be included in a first chip of the electronic device (100), and at least another portion of the processor (120) may be included in a second chip of the electronic device (100) that is different from the first chip of the electronic device (100).

[0058] The processor (120) can cause other components of the electronic device (100) to perform various operations by executing instructions stored in the memory (110).

[0059] In particular, the processor (120) may include one or more processors. Specifically, the one or more processors may include one or more of a central processing unit (CPU), a graphics processing unit (GPU), an accelerated processing unit (APU), a many integrated core (MIC), a digital signal processor (DSP), a neural processing unit (NPU), a hardware accelerator, or a machine learning accelerator. In one embodiment, the CPU (or central processing circuit) may be configured to control components of the processor (120) based on the execution of instructions stored in the memory (110) (e.g., volatile memory and / or non-volatile memory). In one embodiment, the GPU (or graphics processing circuit) may be configured to perform parallel operations (e.g., rendering). In one embodiment, the NPU (or neural processing circuit, or AI (artificial intelligence) chip) may be configured to perform operations (e.g., convolution computation) for an artificial intelligence model (or neural network model).

[0060] One or more processors may control one or any combination of other components of the electronic device and perform operations related to communication or data processing. The one or more processors may execute one or more programs or instructions stored in memory. For example, the one or more processors may perform a method according to an embodiment of the present disclosure by executing one or more instructions stored in memory. For example, the processor (110) may correspond to multiple processors that collectively perform multiple operations by dividing them among the processors.

[0061] When a method according to an embodiment of the present disclosure includes a plurality of operations, the plurality of operations may be performed by one processor or by a plurality of processors. That is, when a first operation, a second operation, and a third operation are performed by a method according to an embodiment, the first operation, the second operation, and the third operation may all be performed by the first processor, or the first operation and the second operation may be performed by the first processor (e.g., a general-purpose processor) and the third operation may be performed by the second processor (e.g., an artificial intelligence-specific processor).

[0062] One or more processors may be implemented as a single core processor including one core, or may be implemented as one or more multicore processors including multiple cores (e.g., homogeneous multicores or heterogeneous multicores). When one or more processors are implemented as a multicore processor, each of the multiple cores included in the multicore processor may include internal processor memory, such as cache memory or on-chip memory, and a common cache shared by the multiple cores may be included in the multicore processor. In addition, each of the multiple cores (or some of the multiple cores) included in the multicore processor may independently read and execute a program instruction for implementing a method according to an embodiment of the present disclosure, or all (or some) of the multiple cores may be linked to read and execute a program instruction for implementing a method according to an embodiment of the present disclosure.

[0063] When a method according to an embodiment of the present disclosure includes a plurality of operations, the plurality of operations may be performed by one core among the plurality of cores included in a multi-core processor, or may be performed by the plurality of cores. For example, when a first operation, a second operation, and a third operation are performed by a method according to an embodiment, the first operation, the second operation, and the third operation may all be performed by a first core included in the multi-core processor, or the first operation and the second operation may be performed by a first core included in the multi-core processor, and the third operation may be performed by a second core included in the multi-core processor.

[0064] In embodiments of the present disclosure, the processor (120) may mean a system on chip (SoC) in which one or more processors and other electronic components are integrated, a single-core processor, a multi-core processor, or a core included in a single-core processor or a multi-core processor, wherein the core may be implemented as a CPU, a GPU, an APU, a MIC, a DSP, an NPU, a hardware accelerator, or a machine learning accelerator, but embodiments of the present disclosure are not limited thereto.

[0065] In particular, the processor (120) may include a first processor (121) and a second processor (122), as illustrated in FIG. 1. In one embodiment, when the processor (120) is an NPU, the processor (120) may include heterogeneous first processors (121) and second processors (122) in one chip.

[0066] Here, the first processor (121) is an artificial intelligence processor for processing operations related to a neural network model, and may include a first internal memory (130) and a first core (140). The first internal memory (130) may be implemented as a cache memory, which is one or more different types of memory used to temporarily store data for a function or feature of the electronic device (100). In this case, the first internal memory (130) may be referred to as a local memory, a synchronous random-access memory (SRAM), or an on-chip scratchpad memory in addition to the cache memory. The first core (140) is a core for processing operations related to a neural network model. In one embodiment of the present disclosure, the first core (140) may be implemented as a DLA (Deep Learning Accelerator) core. The DLA is hardware designed to execute neural network models (or deep learning models). Unlike general-purpose cores (or processors) such as CPUs, it can be optimized to efficiently process neural network operations. The first core (140) can primarily accelerate and process operations related to neural network models, such as matrix multiplication and convolution operations. For example, the first core (140) can process operations such as the conv operator and the tconv operator.

[0067] The second processor (122) is a general-purpose processor for processing operations related to digital signal processing, and may include a second internal memory (135) and a second core (145). The second internal memory (135) may also be implemented as a cache memory. In this case, the second internal memory (135) may be referred to as a local memory, a synchronous random-access memory (SRAM), or an on-chip scratchpad memory in addition to the cache memory. The second core (145) is a general-purpose core for processing operations related to digital signal processing. In one embodiment of the present disclosure, the second core (145) may be implemented as a DSP (Digital Signal Processor) core. A DSP is a general-purpose processor designed for digital signal processing, and may be used in various fields such as audio signal processing, image processing, and communication systems. The second core (145) can mainly perform signal processing operations such as frequency conversion, filtering, compression, encryption, and decryption quickly and efficiently. For example, the second core (145) can process operations such as the softmax operator, the rsqrt operator, etc.

[0068] That is, each of the first and second processors (121, 122) may include an internal memory (130, 135) to reduce communication with the memory (110) (e.g., DRAM).

[0069] Meanwhile, it goes without saying that the processor (120) may include at least one more processor in addition to the first and second processors (121, 122).

[0070] In particular, data movement between the processor (120) and the memory (110) may be referred to as off-chip communication (or off-chip data transfer), and data movement within the processor (120) may be referred to as on-chip communication (or on-chip data transfer). For example, as illustrated in FIG. 2, communication (210) between the memory (110) and the first processor (121) and communication (220) between the memory (110) and the second processor (122) may be referred to as off-chip communication. Additionally, communication (230) between the first internal memory (130) and the first core (140), communication (240) between the second internal memory (135) and the second core (145), and communication between the first processor (121) (or the first internal memory (130) and the second processor (122) (or the second internal memory (135)) may be referred to as on-chip communication.

[0071] Off-chip communication, because data travels over a bus, can have high latency, low bandwidth, and external traffic (e.g., bus contention). In contrast, on-chip communication offers the advantages of lower latency, higher bandwidth, and less external traffic (e.g., no bus contention).

[0072] Previously, only communication (230, 240) between cores and internal memories corresponding to the cores was used in on-chip communication. Therefore, there was a problem that the internal memory corresponding to the idle core was wasted when other cores were in an idle state. Therefore, recently, it has become possible to efficiently use memory by spilling data into the internal memory corresponding to the heterogeneous cores through communication (250) between heterogeneous processors. Here, data spill may mean moving data stored in the internal memory corresponding to the current core to the internal memory corresponding to the external core. In one embodiment of the present disclosure, spill may include storing an intermediate result value output by the core in the internal memory of another core instead of storing it in the memory (110) (i.e., DRAM).

[0073] Referring to FIGS. 3 and 4, this is a drawing for explaining the operation of a DLA core in an NPU chip according to an embodiment of the prior art and the present disclosure.

[0074] Figure 3 is a drawing for explaining the operation of a DLA core in a conventional NPU chip.

[0075] Conventionally, in order for the DLA core (340) to perform operations related to a neural network model, the EM core (340) can load data on input tensors and weights stored in the DRAM (310) and store them in the SRAM corresponding to the DLA core (340). Then, the DLA core (340) performs operations based on the data on the input tensors and weights and then stores the obtained output tensor (or intermediate result, etc.) in the DRAM (310).

[0076] In this case, since data is transferred only through off-chip communication, there is a disadvantage of slow speed, and there is a problem of waste of SRAM (335) of the DSP (322) currently in an idle state.

[0077] FIG. 4 is a diagram for explaining the operation of a DLA core in an NPU chip according to one embodiment of the present disclosure.

[0078] In one or more embodiments, in order for the DLA core (340) to perform operations related to a neural network model, the DLA core (340) may load data on input tensors and weights stored in the DRAM (310) and store the data in the SRAM corresponding to the DLA core (340). In addition, the DLA core (340) may perform operations based on the data on the input tensors and weights and then, instead of storing the obtained output tensor (or intermediate result, etc.) in the DRAM (310), spill the data to the SRAM (335) of the DSP (322), as illustrated in FIG. 4.

[0079] Accordingly, operations related to neural network models can be performed more quickly.

[0080] However, when data is spilled through on-chip communication, problems may arise due to the size of internal memory corresponding to heterogeneous cores and the duplication of execution slots.

[0081] FIG. 5A is a diagram illustrating overflow when an intermediate result value obtained by a first core spills into a second internal memory according to one embodiment of the present disclosure.

[0082] For example, when the output tensor is spilled to the SRAM (335) by the DLA core (340) and the output tensor stored in the DRAM (310) is simultaneously loaded by the SRAM (335), as shown in FIG. 5A, the data sizes of the output tensor spilled by the DLA core (340) and the output tensor stored in the DRAM (310) may be larger than the size of the SRAM (335), resulting in memory overflow.

[0083] As another example, when spilling an output tensor to SRAM (335) by the DLA core (340), the size of the output tensor spilled by the DLA core (340) may be larger than the size of SRAM (335), causing memory overflow.

[0084] That is, there is a problem that memory usage may increase due to data spill, which may cause memory overflow.

[0085] FIG. 5b is a diagram for signing time slot usage information of instructions included in an instruction sequence according to one embodiment of the present disclosure.

[0086] As illustrated in Figure 5b, information about an instruction sequence may include information about the slot usage of the instruction. Here, a slot may refer to a resource that can be executed for a certain period of time. In particular, in a parallel computing environment, multiple instructions or tasks may be executed simultaneously. In this case, the time unit in which each instruction or task is executed may be referred to as a "slot."

[0087] For example, as illustrated in FIG. 5b, the DRAM->DLA slot and the DSP->DRAM slot may be used in the first time slot, the DRAM->DLA slot and the DLA->DRAM slot may be used in the second time slot, and the DRAM->DSP slot and the DLA->DSP slot may be used in the third time slot. Therefore, if the output tensor of the DLA core (340) is spilled to the SRAM (335) of the DSP (322) in the third time slot, a slot conflict problem may occur because the DLA->DSP slot is already in use in the third time slot.

[0088] Accordingly, according to one embodiment of the present disclosure, the electronic device (100) can identify whether data has spilled so that memory overflow does not occur and slot collisions do not occur. This will be described in detail later with reference to the drawings.

[0089] Additionally, when multiple intermediate result values ​​spill to internal memory corresponding to heterogeneous cores, there may be cases where only some of the multiple intermediate result values ​​need to be spilled due to the limited size of the internal memory.

[0090] At this time, the electronic device (100) can identify at least one spill plan (or a combination of spill plans) that can be spilled to the internal memory corresponding to the heterogeneous core among the plurality of intermediate result values, and can identify an optimal spill plan with the greatest benefit among the at least one spill plan. This will be described with reference to FIGS. 6A to 6C.

[0091] Specifically, when multiple intermediate result values ​​spill into internal memory corresponding to heterogeneous cores, the electronic device (100) can identify information about the size and number of times used of the multiple intermediate result values. For example, as illustrated in FIG. 6A, intermediate result value A (610) may have a size of 5 and a number of times used of 2, intermediate result value B (620) may have a size of 6 and a number of times used of 1, intermediate result value C (630) may have a size of 3 and a number of times used of 3, and intermediate result value D (640) may have a size of 2 and a number of times used of 1.

[0092] In addition, the electronic device (100) can identify the size of the internal memory corresponding to the heterogeneous core. For example, the electronic device (100) can identify the size of the internal memory corresponding to the heterogeneous core as 10.

[0093] The electronic device (100) can identify a spill plan such that the sum of the sizes of the intermediate result values ​​is smaller than the size of the internal memory corresponding to the heterogeneous core. For example, the electronic device (100) can identify a spill plan in which the sum of the sizes of the intermediate result values ​​is smaller than 10, which is the size of the internal memory. For example, as illustrated in FIG. 6b, the electronic device (100) can identify spill plan 1 as a combination of the intermediate result value B (620) and the intermediate result value C (630), and as illustrated in FIG. 6c, the electronic device (100) can identify spill plan 2 as a combination of the intermediate result value A (610), the intermediate result value C (630), and the intermediate result value D (640). It goes without saying that the electronic device (100) can identify other spill plans other than spill plans 1 and 2 (for example, a spill plan that is a combination of the intermediate result value A (610) and the intermediate result value C (630), etc.).

[0094] The electronic device (100) can identify a spill plan with the greatest benefit among multiple spill plans. In this case, the spill plan with the greatest benefit may be a reduction in off-chip data traffic. Here, the reduction in off-chip data traffic can be identified by the following mathematical expression 1.

[0095]

[0096] Therefore, the benefit of spill plan 1 may be 6*(1+1) + 3*(3+1)=24, as shown in FIG. 6b, and the benefit of spill plan 2 may be 5*(2+1)+3*(3+1)+2*(1+1)=31, as shown in FIG. 6c.

[0097] Accordingly, the electronic device (100) can identify spill plan 2, which is the spill plan with the greatest benefit among the multiple spill plans, and perform a spill operation.

[0098] Below, a method for identifying multiple spill patterns included in an instruction sequence and identifying whether or not multiple spill patterns are data spills is described.

[0099] FIG. 7 is a flowchart illustrating a process in which a compiler of an NPU allocates resources for an instruction sequence related to a neural network model according to one embodiment of the present disclosure.

[0100] In the following examples, the operations may be performed sequentially, but are not necessarily sequential. For example, the order of the operations may be changed, and at least two operations may be performed in parallel.

[0101] According to one embodiment, operations 710 to 760 may be understood to be performed in a processor (e.g., processor (120) of FIG. 1) of an electronic device (e.g., electronic device (100) of FIG. 1).

[0102] First, the electronic device (100) can obtain information about a neural network model (710). Here, the information about the neural network model may include information about the neural network model structure (e.g., the type and number of layers included in the neural network model, the input and output sizes of each layer, activation functions, etc.), information about weights, information about biases, information about the formats of input and output data, information about optimization and compiler settings, etc.

[0103] An electronic device (100) (particularly, a compiler of an NPU) can perform a parsing operation (720). Here, the parsing operation is a process in which the electronic device (100) understands the input program code and converts it into a data structure that can be internally understood. In particular, the electronic device (100) can analyze the structure of a program and check for syntax errors using a parser. Here, the electronic device (100) can analyze the data flow graph of a program or a neural network model to identify the edges (data flow) connected to each node.

[0104] The electronic device (100) can perform an optimization operation (730). Here, the optimization operation is an operation that transforms the analyzed program or neural network model so that it can be executed more efficiently. To achieve improved execution speed, optimized memory usage, and improved power efficiency, the electronic device (100) can perform constant folding, elimination of redundant code, and optimization of the operation order.

[0105] The electronic device (100) can perform a scheduling operation (740). Here, the scheduling operation determines the order in which operations of a program or neural network model will be executed. By defining when each operation will be performed on a specific core or execution unit, the electronic device (100) can maximize parallelism or optimize memory access patterns to optimize performance.

[0106] The electronic device (100) can perform an operation to identify whether or not to spill to remote on-chip memory (750). Here, since the electronic device (100) has determined the lifetime of instructions in the scheduling phase, it can identify the location where data (e.g., intermediate result values, etc.) will be stored before the allocation operation. In particular, the electronic device (100) can identify whether or not a plurality of candidate spill patterns included in an instruction sequence for processing an operation related to a neural network model are spilled, and update the instruction sequence. This will be described later with reference to FIGS. 8 to 15 .

[0107] An electronic device (100) can perform an allocation operation (760). The allocation operation is an operation that allocates hardware resources (e.g., memory, registers, cache, etc.). In other words, allocation is a process of mapping operations to specific hardware devices, such as a processor core or an NPU (Neural Processing Unit). This process can manage memory space or data buffers and enable efficient use of computational resources.

[0108] The electronic device (100) can understand the input neural network model by parsing it as described above by the NPU compiler, maximize execution efficiency through optimization and scheduling, and finally efficiently execute it by allocating it to actual hardware resources (770).

[0109] FIG. 8 is a flowchart illustrating a process of updating an instruction sequence by identifying whether a plurality of candidate spill patterns included in the instruction sequence are spills, according to one embodiment of the present disclosure.

[0110] In the following examples, the operations may be performed sequentially, but are not necessarily sequential. For example, the order of the operations may be changed, and at least two operations may be performed in parallel.

[0111] According to one embodiment, operations S810 to S890 may be understood to be performed in a processor (e.g., processor (120) of FIG. 1) of an electronic device (e.g., electronic device (100) of FIG. 1).

[0112] First, the electronic device (100) can obtain information about an instruction sequence (S810). Here, the instruction sequence may be a sequence of multiple instructions. In one or more embodiments, the information about the instruction sequence may include at least one of information about the execution time of an instruction included in the instruction sequence, information about the operator code of the instruction, information about data required for an operation in the instruction, and slot usage information of the instruction.

[0113] In one embodiment, the electronic device (100) may obtain information about an instruction sequence as illustrated in FIG. 9. Here, the storage location of the intermediate result value (or output tensor, intermediate tensor, etc.) obtained by the instruction sequence may be a memory (110) (i.e., DRAM).

[0114] The electronic device (100) can obtain information on the memory usage per hour and slot usage used by the instruction sequence (S820).

[0115] In one embodiment, the electronic device (100) can obtain information about the memory usage per hour used by an instruction sequence using a function as shown in Table 1 below.

[0116] Function AnalyzeMemoryUsage(Instructions):# Define an array that contains memory usage of a specific tick= array (same size with Instructions)for in Instructions:if writes on memory:= find_last_use() # Find the last instruction usingfrom in [.,_ .]:[] += size ofreturn

[0117] By the function described above, the electronic device (100) can determine whether each instruction performs a write operation on the memory while traversing the instruction sequence. Then, the electronic device (100) can find the last used instruction after the instruction performing each write operation. The electronic device (100) can calculate the memory usage corresponding to the time range between the instruction and the last used instruction and record the memory usage at the corresponding location. Then, the electronic device (100) can record the memory usage for the last used instruction. In this way, the electronic device (100) can obtain information on the memory usage per time used by the instruction sequence, as illustrated in FIG. 10A. In addition, the electronic device (100) can analyze the instruction sequence to obtain information on the slot usage per time used by the instruction sequence. In one embodiment, the electronic device (100) can obtain information on slot usage per hour, such as in FIG. 10b. For example, as illustrated in FIG. 10b, DRAM->DLA slots and DSP->DRAM slots may be used in a first time period, DRAM->DLA slots and DLA->DRAM slots may be used in a second time period, and DRAM->DSP slots and DLA->DSP slots may be used in a third time period.

[0118] An electronic device (100) can obtain information about a data flow graph from an instruction sequence (S830). The data flow graph may be a graph structure that represents the data flow of the instruction sequence with nodes and edges. Here, a node of the data flow graph may represent an operator. That is, each node represents an operation that performs a specific task, and may be an operation such as convolution, matrix multiplication, or an activation function, for example. An edge of the data flow graph is a line connecting nodes and may represent a data flow (or data dependency). An edge of the data flow graph may include information related to the shape of the data (e.g., a tensor or a multidimensional array) along with the direction of the data flow.

[0119] In one embodiment, the electronic device (100) may obtain the data flow graph illustrated in FIG. 11 through an instruction sequence as illustrated in FIG. 9. Meanwhile, in the case of an instruction for loading / storing a feature map (FM) from a DRAM, the corresponding feature map ID may be displayed in the data flow graph. In the case of data belonging to the same feature map, the feature map ID may be the same.

[0120] Meanwhile, a data flow graph can be obtained at step 830 of FIG. 8, but this is only one embodiment, and the data flow graph can be obtained in advance (e.g., in a parsing operation (710)).

[0121] The electronic device (100) can obtain information on multiple candidate spill patterns (S840). Here, the candidate spill pattern may be a pattern of instructions capable of performing data spill in an instruction sequence. Here, the candidate spill pattern may be referred to by various terms, such as a candidate spill combination or a candidate spill plan.

[0122] In one embodiment, the electronic device (100) may analyze a data flow graph to identify a plurality of operators included in an instruction sequence that have the same feature map ID, and include a store operator, a concatenation operator, and a load operator, as candidate spill patterns. In one embodiment, the electronic device (100) may identify "%4: store operator, %7: store operator, %8: concat operator, %9: load operator, %10: load operator" having the same feature map ID in the data flow graph of FIG. 11 as a candidate spill pattern (1210), as illustrated in FIG. 12.

[0123] In another embodiment, the electronic device (100) can analyze a data flow graph to identify a plurality of operators included in an instruction sequence that have the same feature map ID and include a store operator and a load operator as candidate spill patterns.

[0124] The electronic device (100) can identify information on memory usage and slot usage by a plurality of candidate spill patterns (S850). Specifically, the electronic device (100) can identify information on memory usage and slot usage by a plurality of candidate spill patterns based on information on memory usage and slot usage per hour using the instruction sequence obtained in operation S820 of FIG. 8.

[0125] The electronic device (100) can sort a plurality of candidate spill patterns based on the number of times each candidate spill pattern is used (S860). Specifically, the electronic device (100) can sort the plurality of candidate spill patterns in a non-ascending order based on the number of times each candidate spill pattern is used.

[0126] Specifically, the electronic device (100) can sort a plurality of candidate spill patterns based on the ratio of the benefit and cost through data spill. Here, the benefit through data spill may be the amount of reduction in off-chip data traffic that occurs when the intermediate result value by the candidate spill pattern is spilled to the second internal memory (135). The amount of reduction in off-chip data traffic may be the sum of 1) the size of the intermediate result value when the operator for storing the intermediate result value (i.e., the feature map) in DRAM is changed to on-chip memory communication and 2) the number of times the intermediate result value is used * the size of the intermediate result value when the operators for loading the intermediate result value are changed to on-chip memory communication. In addition, the cost through data spill may be the size of the intermediate result value stored in the on-chip memory (i.e., the internal memory corresponding to the heterogeneous core).

[0127] Therefore, the ratio of benefits and costs through data spill can be proportional to the number of uses, as calculated in mathematical expression 2 below.

[0128]

[0129] Accordingly, the electronic device (100) can sort the plurality of candidate spill patterns in ascending order according to the number of times the plurality of candidate spill patterns are used. That is, the electronic device (100) can sort the candidate spill patterns so that a higher number of times the candidate spill patterns are used gives them a higher priority, and can sort the candidate spill patterns so that a lower number of times the candidate spill patterns are used gives them a lower priority. For example, as illustrated in FIG. 13, the electronic device (100) can sort the first candidate spill pattern (1310) with a number of uses of 3 as the first priority, the second candidate spill pattern (132) with a number of uses of 2 as the second priority, and the third candidate spill pattern (133) with a number of uses of 1 as the third priority.

[0130] In one or more embodiments, if there are candidate spill patterns with the same number of uses among a plurality of candidate spill patterns, the electronic device (100) may preferentially sort candidate spill patterns with a larger feature map size among candidate spill patterns with the same number of uses.

[0131] The electronic device (100) can identify whether to spill intermediate results for each of a plurality of candidate spill patterns into the second internal memory (135) (S870). This will be described with reference to FIG. 14.

[0132] FIG. 14 is a flowchart illustrating a process for identifying whether a candidate spill pattern is spilled, according to one embodiment of the present disclosure.

[0133] First, the electronic device (100) can obtain information on a candidate spill pattern (S1410). Here, the information on the candidate spill pattern may include information on memory usage and slot usage by the candidate spill pattern identified in step S850.

[0134] The electronic device (100) can identify whether the memory usage by the candidate spill pattern is less than or equal to the size of the second internal memory (135) (S1420). Here, the memory usage by the candidate spill pattern may be the sum of the second internal memory usage (MemUsage[i]) at the time (i) of executing the candidate spill pattern and the size (FM_size) of the intermediate result value (or feature map) obtained by the candidate spill pattern.

[0135] If the memory usage by the candidate spill pattern is less than or equal to the second internal memory size (S1420-Y), the electronic device (100) can identify whether the same on-chip communication slot exists at the time of executing the candidate spill pattern (S1430). Here, the on-chip communication slot may be a slot that utilizes communication between heterogeneous processors within a single chip. For example, the on-chip communication slot may include a DLA->DSP slot and a DSP->DLA slot.

[0136] Specifically, the electronic device (100) can identify whether the slot usage of the candidate spill pattern is restricted. Here, the electronic device (100) can identify whether the slot usage of the third candidate spill pattern is restricted by identifying whether an on-chip communication slot for spilling data stored in the first internal memory to the second internal memory (135) already exists at the time of executing the candidate spill pattern. That is, the electronic device (100) can determine whether an on-chip communication slot identical to the candidate spill pattern exists by another instruction at the time of executing the candidate spill pattern.

[0137] If the same on-chip communication slot does not exist at the time of executing the candidate spill pattern (S1430-N), the electronic device (100) can identify the intermediate result obtained by the candidate spill pattern to spill into the second internal memory (135) (S1440).

[0138] If the memory usage by the candidate spill pattern exceeds the second internal memory size (S1420-N) or the same on-chip communication slot exists at the time of executing the candidate spill pattern (S1430-Y), the electronic device (100) can identify the intermediate result value obtained by the candidate spill pattern to be stored in the memory (110) (S1450).

[0139] After operation S1440 or operation S1450, the electronic device (100) can identify whether a next candidate spill pattern exists (S1460). That is, the electronic device (100) can identify whether to sequentially spill the intermediate result values ​​obtained by the candidate spill patterns into the second internal memory (135) by performing operations S1410 to S1430 for each of a plurality of candidate spill patterns sorted in ascending order.

[0140] If the next candidate spill pattern exists (S1460-Y), the electronic device (100) can perform operations S1410 to S1430 for the next candidate spill pattern to identify whether to spill the intermediate result value obtained by the next candidate spill pattern into the second internal memory (135).

[0141] Meanwhile, operations S1420 and S1430 of FIG. 14 can be expressed as functions as shown in Table 2 below.

[0142] Function constraint_violated(P):# P: patternviolated = Falsefor i in (P.start_tick, P.end_tick):if MemUsage[i] + P.FM_size > Memory size:violated = Truefor load in P.loads:if SlotUsage[load.tick].occupied(on-chip_load):violated = Truefor store in P.stores:if SlotUsage[store.tick].occupied(on-chip_store):violated = Truereturn violated

[0143] Referring back to FIG. 8, the electronic device (100) may change the storage location of the intermediate result value for at least some of the plurality of candidate spill patterns included in the instruction sequence to the second internal memory (135) according to the identification result (S880). Specifically, in the case of a candidate spill pattern identified as spilling the intermediate result value from among the plurality of candidate spill patterns to the second internal memory (135), the electronic device (100) may change the storage location of the intermediate result value to the second internal memory (135). In the case of a candidate spill pattern identified as spilling the intermediate result value from among the plurality of candidate spill patterns to the memory (110), the electronic device (100) may maintain the storage location of the intermediate result value as the memory (110).

[0144] For example, the electronic device (100) may change the storage location (and loading location) of the candidate spill pattern included in the instruction sequence to the second internal memory (135) (or on-chip memory), as illustrated in FIG. 15. For example, the electronic device (100) may change the storage location (or loading location) of "%4: store operator, %7: store operator, %8: concat operator, %9: load operator, %10: load operator" of the instruction sequence to the on-chip memory, as illustrated in FIG. 15.

[0145] The electronic device (100) may update information on memory usage and slot usage per hour based on the change result using an instruction sequence (S890). For example, the electronic device (100) may update information on memory usage and slot usage based on a function as shown in Table 3 below.

[0146] Function selection(sorted_patterns):For each pattern P in the sorted_patterns,if constraint_violated(P):continueConvertToOnChipCommunication(P) # Convert load / store to on-chip_load / storeUpdate(MemUsage) # Increase remote on-chip memory usageUpdate(SlotUsage) # Mark slots for on-chip_load / store

[0147] The electronic device (100) can allocate resources based on updated information about memory usage and slot usage. By the method described above, the electronic device (100) can efficiently process instructions related to a neural network model by using internal memory corresponding to heterogeneous cores, thereby improving the computational speed.

[0148] Meanwhile, the methods according to various embodiments of the present disclosure may be provided as included in a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read only memory (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) through an application store (e.g., Play Store™) or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product (e.g., a downloadable app) may be temporarily stored or temporarily generated in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0149] The methods according to various embodiments of the present disclosure may be implemented as software including commands stored in a machine-readable storage medium that can be read by a machine (e.g., a computer). The device is a device that can call commands stored in the storage medium and operate according to the called commands, and may include an electronic device (e.g., a TV) according to the disclosed embodiments.

[0150] Meanwhile, a device-readable storage medium may be provided in the form of a non-transitory storage medium. Here, the term "non-transitory storage medium" simply means a tangible device that does not contain signals (e.g., electromagnetic waves). This term does not distinguish between cases where data is permanently stored in the storage medium and cases where data is temporarily stored. For example, a "non-transitory storage medium" may include a buffer in which data is temporarily stored.

[0151] When the above instruction is executed by the processor, the processor may perform the function corresponding to the instruction directly or by using other components under the control of the processor. The instruction may include code generated or executed by a compiler or interpreter.

[0152] Although the preferred embodiments of the present disclosure have been illustrated and described above, the present disclosure is not limited to the specific embodiments described above, and various modifications may be made by a person having ordinary skill in the art to which the present disclosure pertains without departing from the gist of the present disclosure as claimed in the claims, and such modifications should not be understood individually from the technical idea or prospect of the present disclosure.

Claims

In electronic devices, memory containing instructions; and A first processor including a first internal memory and a first core for performing operations related to a neural network model, and a second processor including a second internal memory and a second core for performing operations related to digital signal processing; The above instructions, when executed by the first and second processors, cause the electronic device to: Obtain information about the instruction sequence for performing operations related to the neural network model, Obtain information about the memory usage and slot usage per hour used by the above instruction sequence, Obtain information about the data flow graph from the above instruction sequence, Obtain information on a plurality of candidate spill patterns for spilling intermediate result values ​​obtained through calculation based on the above data flow graph into the second internal memory, Identifying information about memory usage and slot usage by the plurality of candidate spill patterns based on information about memory usage and slot usage per hour used by the above instruction sequence, An electronic device that identifies whether to spill intermediate result values ​​obtained by the plurality of candidate spill patterns into the second internal memory based on whether the memory usage by the plurality of candidate spill patterns is less than or equal to the second internal memory size and whether slot usage of the plurality of candidate spill patterns is limited. In the first paragraph, The above instructions, when executed by the first and second processors, cause the electronic device to: An electronic device that identifies a plurality of operators included in the above instruction sequence, each having the same feature map ID and including a store operator, a concatenation operator, and a load operator, as a candidate spill pattern. In the first paragraph, The above instructions, when executed by the first and second processors, cause the electronic device to: Sort the plurality of candidate spill patterns in non-ascending order based on the number of times the candidate spill patterns are used, An electronic device that identifies whether to sequentially spill intermediate results obtained by the candidate spill patterns into the second internal memory for each of the plurality of candidate spill patterns sorted in ascending order. In the third paragraph, The above instructions, when executed by the first and second processors, cause the electronic device to: An electronic device that, when there are candidate spill patterns with the same number of uses among the plurality of candidate spill patterns, gives priority to sorting candidate spill patterns with a larger feature map size among candidate spill patterns with the same number of uses. In the first paragraph, The above instructions, when executed by the first and second processors, cause the electronic device to: If the memory usage by the first candidate spill pattern is less than or equal to the second internal memory size, the intermediate result value obtained by the first candidate spill pattern is identified to be spilled to the second internal memory, An electronic device that identifies an intermediate result value obtained by the second candidate spill pattern to be stored in the memory when the memory usage by the second candidate spill pattern exceeds the second internal memory size. In the first paragraph, The above instructions, when executed by the first and second processors, cause the electronic device to: An electronic device that identifies whether an on-chip communication slot for spilling data stored in the first internal memory to the second internal memory by another instruction already exists at the time of executing the third candidate spill pattern, thereby identifying whether the slot usage of the third candidate spill pattern is restricted. In paragraph 6, The above instructions, when executed by the first and second processors, cause the electronic device to: If the on-chip communication slot does not exist at the time of executing the fourth candidate spill pattern, the intermediate result obtained by the fourth candidate spill pattern is identified to be spilled to the second internal memory, An electronic device that identifies the presence of the on-chip communication slot at the time of executing the fifth candidate spill pattern so as to store the intermediate result obtained by the fifth candidate spill pattern in the memory. In the first paragraph, The above instructions, when executed by the first and second processors, cause the electronic device to: Identifying whether there is a spill for the plurality of candidate spill patterns, and changing the storage location of the intermediate result values ​​obtained by at least some of the plurality of candidate spill patterns to the second internal memory according to the identification result, An electronic device that updates information about memory usage and slot usage per hour used by the instruction sequence based on the change result. In the first paragraph, Information about the above instruction sequence is: An electronic device comprising at least one of information about the time at which an instruction included in the above instruction sequence is executed, information about an operator code of the instruction, information about data required for an operation in the instruction, and slot usage information of the instruction. A method for controlling an electronic device comprising a first processor including a first core for performing operations related to a memory, a first internal memory and a neural network model, and a second processor including a second internal memory and a second core for performing operations related to digital signal processing, A step of obtaining information about an instruction sequence for performing an operation related to a neural network model; A step of obtaining information on memory usage and slot usage per hour used by the above instruction sequence; A step of obtaining information about a data flow graph from the above instruction sequence; A step of obtaining information on a plurality of candidate spill patterns for spilling intermediate result values ​​obtained through calculation based on the data flow graph into the second internal memory; A step of identifying information on memory usage and slot usage by the plurality of candidate spill patterns based on information on memory usage and slot usage per hour used by the instruction sequence; and A control method comprising: a step of identifying whether to spill intermediate result values ​​obtained by the plurality of candidate spill patterns into the second internal memory based on whether the memory usage by the plurality of candidate spill patterns is less than or equal to the second internal memory size and whether slot usage of the plurality of candidate spill patterns is limited; In Article 10, The step of obtaining information about the above multiple candidate spill patterns is: A control method for identifying a plurality of operators included in the above instruction sequence, which have the same feature map ID and include a store operator, a concatenation operator, and a load operator, as candidate spill patterns. In Article 10, The above control method is, Further comprising a step of sorting the plurality of candidate spill patterns in a non-ascending order based on the number of times the candidate spill patterns are used; The step of identifying whether to spill to the second internal memory is as follows: A control method for identifying whether to sequentially spill intermediate result values ​​obtained by the candidate spill patterns into the second internal memory for each of the plurality of candidate spill patterns sorted in ascending order. In paragraph 12, The above sorting steps are: A control method for preferentially sorting candidate spill patterns having a larger feature map size among candidate spill patterns having the same number of uses among the plurality of candidate spill patterns. In Article 10, The step of identifying whether to spill to the second internal memory is as follows: If the memory usage by the first candidate spill pattern is less than or equal to the second internal memory size, the intermediate result value obtained by the first candidate spill pattern is identified to be spilled to the second internal memory, A control method for identifying an intermediate result value obtained by the second candidate spill pattern to be stored in the memory when the memory usage by the second candidate spill pattern exceeds the second internal memory size. In Article 10, The step of identifying whether to spill to the second internal memory is as follows: A control method for identifying whether the slot usage of the third candidate spill pattern is restricted by identifying whether an on-chip communication slot for spilling data stored in the first internal memory to the second internal memory by another instruction already exists at the time of executing the third candidate spill pattern.

Images