Neural processing device and method of operation of a neural processing device

Abstract

A neural processing device and a method of operating a neural processing device are disclosed. The neural processing device includes a first memory configured to store general data, a second memory distinguished from the first memory and having a capacity smaller than that of the first memory, a bandwidth control path configured to reconfigure a memory bandwidth based on a control signal for a memory user to use one of the first memory and the second memory, and a control logic configured to calculate a target capacity of data for a target user among the memory users and generate the control signal based on a result of comparing the target capacity with the capacity of the second memory to store the data of the target user in the second memory, the target user among the memory users being determined based on a layer configuration of an artificial neural network.

Description

[0001] This application claims the benefit of Korean Patent Application No. 10-2021-0028311, filed on March 3, 2021, with the Korean Intellectual Property Office, the entire disclosure of which is incorporated herein by reference for all purposes. Technical Field

[0002] The following description relates to the neural processing device and its operation. Background Technology

[0003] For a neural processing unit (NPU), a single integrated memory can be used. In this case, as the number of operators sharing the memory increases, the number of memory clients may also increase. Therefore, the incidence of bank conflicts may increase, thus impairing NPU performance. However, increasing the number of memory banks to prevent such conflicts increases power and / or memory area overhead. Furthermore, fixing the memory size of the banks may result in limited and inefficient memory usage and increased area overhead. Therefore, a memory operation method that can effectively utilize memory while preventing bank conflicts is desired. Summary of the Invention

[0004] This summary is provided to present, in a simplified form, the selection of concepts that will be further described in the detailed embodiments below. This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to help determine the scope of the claimed subject matter.

[0005] In one general aspect, a neural processing device includes: a first memory configured to store general data; a second memory, separate from the first memory and having a capacity smaller than that of the first memory; a bandwidth control path configured to reconfigure memory bandwidth based on a control signal for a memory user to use one of the first and second memories; and control logic configured to: calculate a target capacity for data of a target user among the memory users, and generate a control signal based on a comparison of the target capacity with the capacity of the second memory to store the target user data in the second memory, wherein the target user among the memory users is determined based on a layer configuration of an artificial neural network.

[0006] The control logic can also be configured to: in response to the target capacity being greater than the capacity of the second memory, generate a control signal to store the target user's data in the first memory, and reduce the bandwidth of the first memory for data of at least one remaining user in the memory users that excludes the target user.

[0007] The control logic can also be configured to generate a control signal in response to the target capacity being less than or equal to the capacity of the second memory, to store the target user's data in the second memory, and to increase the bandwidth of the first memory for data of at least one remaining user in the memory users who have excluded the target user.

[0008] The control logic can also be configured to generate a control signal in response to the capacity of the first memory being greater than the data capacity of the memory user, to operate the bandwidth control path in normal mode, in which the first memory is used for the data of the target user, the target user being determined based on the layer type and layer configuration of the artificial neural network.

[0009] The control logic can also be configured to generate a control signal in response to the capacity of the first memory being less than or equal to the data capacity of the memory user, to operate the bandwidth control path in an enhanced mode, in which the enhanced mode uses the second memory for the data of the target user, the target user being determined based on the layer type and layer configuration of the artificial neural network.

[0010] When the artificial neural network includes convolutional layers and the target user is a first user associated with weights among the memory users, the control logic can also be configured to: store the weights in a second memory based on the input channels and kernel size of the convolutional layer, and increase the bandwidth of the first memory for a third user among the memory users, the third user being associated with a portion and / or the output feature map.

[0011] When the artificial neural network includes fully connected layers and the target user is a second user associated with the feature map, the control logic can also be configured to: store the feature map data in a second memory and increase the bandwidth of the first memory for the first user in the memory users associated with the weights.

[0012] When the artificial neural network includes convolutional layers and the target user is a third user associated with a portion and a weight, the control logic can also be configured to: store the data of the portion and a weight in a second memory based on the size of the feature map of the convolutional layer, and increase the bandwidth of the first memory for the first user in the memory users, the first user being associated with the weight.

[0013] Each memory user can be associated with any of the input feature map, weights, and partial and / or output feature maps.

[0014] The bandwidth control path may include twelve input ports and nine output ports connected to a first memory. The twelve input ports include: four first input ports for a first user associated with weights among memory users, four second input ports for a second user associated with an input feature map among memory users, and four third input ports for a third user associated with a partial sum among memory users.

[0015] The four first input ports can be connected to a 4-to-1 multiplexer. The remaining three first input ports, excluding one of the four first input ports, can be connected to the three third input ports and three 2-to-1 multiplexers for the third user, respectively. The four second input ports for the second user can be directly connected to four of the nine output ports. The second user is associated with the input feature map.

[0016] When the bandwidth of the first memory for the first user is limited, the control logic can also be configured to sequentially provide selection signals to the 4-to-1 multiplexer via a polling method, the 4-to-1 multiplexer being connected to the four first input ports.

[0017] When the bandwidth of the first memory for the first user increases, the control logic can also be configured to provide a selection signal to one of the four first input ports, which is not connected to the three third input ports via a 2-to-1 multiplexer.

[0018] In another general aspect, a method of operating a neural processing device includes a first memory and a second memory, the second memory having a capacity smaller than that of the first memory. The method includes: calculating a target capacity for data of a target user among memory users, the target user being determined based on a layer configuration of an artificial neural network; comparing the target capacity with the capacity of the second memory; based on the comparison result, determining a target memory between the first and second memories in which the data of the target user will be stored; and generating control signals for a bandwidth control path to reconfigure memory bandwidth for the memory users to store the data of the target user in the target memory.

[0019] The step of determining the target memory may include: determining a first memory as the target memory in response to the target capacity being greater than the capacity of a second memory. The step of generating a control signal may include: generating a control signal to store the target user's data in the first memory and reducing the bandwidth of the first memory for data of at least one remaining user among the memory users who have excluded the target user.

[0020] The step of determining the target memory may include: determining the second memory as the target memory in response to the target capacity being less than or equal to the capacity of the second memory. The step of generating a control signal may include: generating a control signal to store the target user's data in the second memory, and increasing the bandwidth of the first memory for data of at least one remaining user among the memory users who have excluded the target user.

[0021] The operation method may further include: comparing the capacity of the first memory with the data capacity of the memory user. The step of generating a control signal may include: in response to the comparison result that the capacity of the first memory is greater than the data capacity of the memory user, generating a control signal to operate a bandwidth control path in a normal mode, whereby the first memory is used for data of a target user, the target user being determined based on the layer type and layer configuration of the artificial neural network.

[0022] The step of generating a control signal may further include: generating a control signal in response to the capacity of the first memory being less than or equal to the data capacity of the memory user, to operate the bandwidth control path in an enhanced mode, wherein the enhanced mode uses the second memory for the data of the target user, the target user being determined based on the layer type and layer configuration.

[0023] When an artificial neural network includes convolutional layers and the target user can be a first user associated with weights, the step of generating control signals may include: storing the weights in a second memory based on the input channels and kernel size of the convolutional layers; and increasing the bandwidth of the first memory for a third user among the memory users, the third user being associated with a portion and / or the output feature map.

[0024] When the artificial neural network includes fully connected layers and the target user is a second user associated with the feature map, the step of generating control signals may include: storing the feature map data in a second memory; and increasing the bandwidth of the first memory for the first user in the memory, the first user being associated with weights.

[0025] When the artificial neural network includes convolutional layers and the target user is a third user associated with a subset of weights, the step of generating control signals may include: storing the subset of weights in a second memory based on the size of the feature map of the convolutional layer; and increasing the bandwidth of the first memory for a first user among the memory users, the first user being associated with weights.

[0026] Each of the memory users can be associated with any of the input feature map, weights, and partial and / or output feature maps.

[0027] A non-transitory computer-readable storage medium that stores instructions, which, when executed by a processor, cause the processor to perform the above-described operation method.

[0028] In another general aspect, a neural processing device includes: a first memory; a second memory having a capacity smaller than that of the first memory; a bandwidth control path configured to reconfigure memory bandwidth based on a control signal for memory users to use either the first memory or the second memory; and control logic. The control logic is configured to: calculate a target capacity for data of a target user among the memory users, the target user being determined based on a layer configuration of an artificial neural network; compare the target capacity with the capacity of the second memory; and, in response to the target capacity being less than or equal to the capacity of the second memory, generate a control signal to store the target user's data in the second memory and increase the bandwidth of the first memory for data of at least one remaining user among the memory users excluding the target user.

[0029] The control logic can also be configured to: in response to the target capacity being greater than the capacity of the second storage bank, generate a control signal to store the target user's data in the first storage bank, and reduce the bandwidth of the first storage bank for data of at least one remaining user in the memory users who have excluded the target user.

[0030] Other features and aspects will become clear from the following detailed description, drawings, and claims. Attached Figure Description

[0031] Figure 1 An example of memory used in a neural processing unit (NPU) is shown.

[0032] Figure 2 An example of a roofline model is shown.

[0033] Figure 3 An example of a neural processing device is shown.

[0034] Figure 4A and Figure 4B It shows Figure 3 An example of the internal structure of the bandwidth control path is shown.

[0035] Figure 5 An example of the data path and control path for the bandwidth control path is shown.

[0036] Figure 6 An example of the operation flow of a neural processing device is shown.

[0037] Figure 7 An example of the operation and roofline model is shown in the case where the first user associated with the weights uses the second memory.

[0038] Figure 8 An example of operation and roofline model is shown in the case where a second user uses a second memory associated with a feature map.

[0039] Figure 9 An example of the operation and roofline model is shown in the case where a second memory is used by a third user in part and in association with it.

[0040] Figure 10 Another example of a neural processing device is shown.

[0041] Figure 11 Another example of the operation flow of a neural processing device is shown.

[0042] Figure 12A and Figure 12B An example of a neural processing device using a first memory and a second memory for data from a target user is shown.

[0043] Figure 13 An example of a bandwidth control bus is shown.

[0044] Figure 14 An example of the operation flow of a neural processing device is shown.

[0045] Throughout the accompanying drawings and detailed embodiments, the same reference numerals denote the same elements. The drawings may not be drawn to scale, and for clarity, illustration, and convenience, the relative sizes, proportions, and depictions of elements in the drawings may be exaggerated. Detailed Implementation

[0046] The following detailed embodiments are provided to aid the reader in gaining a comprehensive understanding of the methods, apparatus, and / or systems described herein. However, various changes, modifications, and equivalents of the methods, apparatus, and / or systems described herein will become apparent upon understanding this disclosure. For example, the order of operations described herein is merely illustrative and is not limited to those set forth herein, but may be altered as will become apparent upon understanding this disclosure, except for operations that must occur in a specific order. Furthermore, for clarity and conciseness, descriptions of known features may be omitted.

[0047] The features described herein may be embodied in different forms and should not be construed as limited to the examples described herein. Rather, the examples described herein are provided merely to illustrate some of the many feasible ways of implementing the methods, apparatus, and / or systems described herein that will become clear upon understanding the disclosure of this application.

[0048] The terminology used herein is for the purpose of describing particular examples only and is not intended to limit this disclosure. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. As used herein, the term “and / or” includes any one and any combination of any two or more of the associated listed items. As used herein, the terms “comprising,” “including,” and “having” indicate the presence of the stated features, quantities, operations, elements, components, and / or combinations thereof, but do not exclude the presence or addition of one or more other features, quantities, operations, elements, components, and / or combinations thereof.

[0049] Additionally, terms such as first, second, A, B, (a), (b), etc., may be used herein to describe components. Each of these terms is not intended to define the nature, order, or sequence of the corresponding component, but only to distinguish the corresponding component from other components. Throughout the specification, when an element such as a layer, region, or substrate is described as being “on,” “connected to,” or “bonded to” another element, it may be directly “on,” directly “connected to,” or directly “bonded to” the other element, or there may be one or more other elements in between. Conversely, when an element is described as being “directly on,” “directly connected to,” or “directly bonded to” another element, there may be no other elements in between.

[0050] Unless otherwise defined, all terms used herein (including technical and scientific terms) shall have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains and as commonly understood after understanding this disclosure. Unless expressly defined herein, terms (such as those defined in a general dictionary) shall be interpreted as having the same meaning as they have in the context of the relevant field and in this disclosure, and shall not be interpreted in an idealized or overly formal sense.

[0051] Furthermore, in the description of the exemplary embodiments, descriptions will be omitted where a detailed description of a structure or function known after understanding the disclosure of this application would lead to a vague interpretation of the exemplary embodiments. Hereinafter, the examples will be described in detail with reference to the accompanying drawings, in which the same reference numerals always denote the same elements.

[0052] Figure 1 An example of memory used in a neural processing unit (NPU) is shown. (See reference...) Figure 1 The diagram shows an on-chip storage device 111 for the NPU110.

[0053] The NPU 110 can refer to an artificial intelligence (AI) processor, which connects to multiple neurons via synapses to exchange signals between them and is configured to process a large number of operations simultaneously, and can be optimized for AI algorithm operations. The NPU 110 can also be referred to as a neural network processing device or a neural processing device, and can use multiple layers of hardware nodes (e.g., neurons) to learn and process information on its own.

[0054] The NPU 110 can process multiple operations in real time (such as simultaneous matrix operations) and derive optimal values ​​through self-learning based on accumulated data.

[0055] The NPU 110 may store input feature maps (or indicated as IFM) and / or weights in storage device 111, which serves as internal storage for the NPU 110. The input feature maps and / or weights are read from storage device 111 via input feature map acquirer 112 and / or weight acquirer 113. The NPU 110 may temporarily store the results of partial sums (or indicated as PSUM) generated during operation in storage device 111 via PSUM acquirer 114. Optionally, the NPU 110 may store output feature maps (or indicated as OFM) in storage device 111 via OFM writer and use the output feature maps in the next layer.

[0056] The NPU 110 can reuse internally stored information for operations in the Multiply-Accumulate (MAC) unit 116. Internally stored information may include, for example, input feature maps, weights, parts and results, and output feature maps, but examples of internally stored information are not limited to these.

[0057] Storage device 111 may have a relatively higher processing speed than dynamic random access memory (DRAM) 130, which has high power consumption and low processing speed. Storage device 111 may have a capacity smaller than that of DRAM 130. Storage device 111 may be configured as, for example, static random access memory (SRAM) and / or flash memory, but examples of storage device 111 are not limited thereto.

[0058] Figure 2 An example of a roofline model is shown. Figure 2 The memory-bound algorithm 1 and processor-bound algorithm 2 in the roofline model are shown.

[0059] A roofline model is an intuitive visual performance model that provides estimated performance values ​​for a given computational core or application running on an architecture such as a multi-core or accelerator processor by representing inherent hardware constraints as well as the potential effects and priorities of optimization. The roofline model can indicate the potential performance values ​​achievable through optimization.

[0060] Reference Figure 2 The roofline model shown exhibits performance that gradually increases with increasing computational intensity, then converges at a certain point. For example, in the region where Algorithm 1 exists, efficiency is observed to increase with increasing memory bandwidth at 100% MAC utilization. Algorithm 1's efficiency increases with increasing memory bandwidth, but after a computational intensity of 256 ops / byte, its efficiency no longer increases even with further increases in memory bandwidth. Computational intensity can be represented by operations per byte of DRAM traffic and is used to determine the most suitable architecture for a given computational core or to determine methods for optimizing the core.

[0061] Before Algorithm 1 reaches a point where its computational intensity is 256 ops / byte and its efficiency converges, performance can increase as memory accesses decrease. As mentioned above, this situation can be described as memory-constrained when the time required to complete a given computation or computation query is primarily determined by the amount of memory required to store the task data.

[0062] Conversely, in the regions where Algorithm 2 exists, it is observed that despite the increase in memory bandwidth, efficiency does not increase, and the performance of the processor itself determines efficiency. In the regions where Algorithm 2 exists, the importance of memory access can be reduced, so memory no longer limits performance. For example, efficiency can be improved by increasing the number of MACs or increasing the MAC (computation) speed. As mentioned above, when the time required to complete a given computation or computation query is primarily determined by the performance of the processor configured to perform the computation or computation operation, this situation can be described as computationally limited or processor-limited.

[0063] The operation of a neural processing device with memory-limited efficiency will be described in detail below.

[0064] Figure 3 An example of a neural processing device is shown. (See reference) Figure 3 The neural processing device 300 includes a first memory 310, a second memory 330, a bandwidth control module 350, and control logic 390.

[0065] The first memory 310 may store general-purpose data therein. The first memory 310 may also be referred to as main memory or general-purpose memory because it stores all the datasets of the artificial neural network. In the following text, the terms "first memory," "main memory," and "general-purpose memory" will be used interchangeably. The first memory 310 includes multiple SRAM entities. The first memory 310 may be divided into various portions that can be configured to be independently accessible multiple banks of memory. The first memory 310 includes bank arbitration logic 315, which is configured to arbitrate access requests to banks of memory in the first memory 310 and, where possible, allow memory accesses to be delivered to banks in a sequential order. Control signals for the bank arbitration logic 315 may be transmitted from outside the neural processing device 300 or generated internally within the neural processing device 300. According to an example, the first memory 310 may be a temporary RAM.

[0066] The second memory 330 is distinguishable from the first memory 310 and has a smaller capacity than the first memory 310. The second memory 330 includes multiple SRAM entities and is configured with multiple memory banks as in the first memory 310. The second memory 330 may also be referred to as auxiliary memory or dedicated memory because it operates as dedicated memory for specific operations (e.g., storing partial sums and results generated during operations). In the following text, the terms "second memory," "auxiliary memory," and "dedicated memory" will be used interchangeably.

[0067] Similar to the first memory 310, the second memory 330 includes bank arbitration logic 335, which is configured to arbitrate requests for access to the banks of the second memory 330 and, where possible, allow memory accesses to be transferred to the banks in a sequential manner.

[0068] The bandwidth control module 350 includes a bandwidth (or indicated as BW or B / W) control path 355 (or indicated as a B / W controlled path), multiple 2-to-1 multiplexers 351, 352 and 353, and a 3-to-1 multiplexer 357.

[0069] The bandwidth control path 355 can reconfigure the memory bandwidth for memory users 360, 370, and 380 based on control signals from control logic 390, allowing memory users 360, 370, and 380 to use either the first memory 310 or the second memory 330. Memory users 360, 370, and 380 may include, for example, a user 360 associated with an input feature map, a user 370 associated with weights, and a user 380 associated with a partial and / or output feature map, but examples are not limited thereto. The user 360 associated with the input feature map may be a user performing operations of acquiring, reading, calculating, and storing the input feature map. The user 370 associated with weights may be a user performing operations of acquiring, reading, calculating, and storing the weights. The user 380 associated with a partial and / or output feature map may be a user performing operations of acquiring, reading, calculating, and storing the result of the partial sum or the output feature map.

[0070] Control logic 390 can calculate the target capacity of data for a target user, which is determined from memory users 360, 370, and 380 based on the layer configuration of an artificial neural network. The layer configuration can be interpreted as the form of the layer configuration based on the type of artificial neural network (e.g., whether the artificial neural network is formed by fully connected layers, convolutional layers, or recurrent layers). The target user can be interpreted as the user whose data in memory users 360, 370, and 380 will be the target user to be stored in the second memory 330.

[0071] Control logic 390 can generate a control signal based on the result of comparing the target capacity with the capacity of the second memory 330, so that the target user's data is stored in the second memory 330.

[0072] In response to a target capacity greater than the capacity of the second memory 330, control logic 390 may generate a control signal causing the target user's data to be stored in the first memory 310, thereby limiting the bandwidth of the first memory 310 for data of at least one remaining user among memory users 360, 370, and 380 who have excluded the target user. Conversely, in response to a target capacity less than or equal to the capacity of the second memory 330, control logic 390 may generate a control signal causing the target user's data to be stored in the second memory 330, thereby increasing the bandwidth of the first memory 310 for data of at least one remaining user among memory users 360, 370, and 380 who have excluded the target user.

[0073] When the bandwidth of the first memory 310 is sufficient, the control logic 390 may allow all datasets to be stored in the first memory 310. However, when the bandwidth of the first memory 310 is insufficient, the control logic 390 may use the second memory 330 as a dedicated auxiliary memory for small-sized data based on the layer type and layer configuration of the artificial neural network.

[0074] In the following text, it will be described with reference to Figure 4 how the bandwidth control path 355 reconfigures the memory bandwidth for the memory user based on control signals. Additionally, reference will be made to... Figure 5 Describe the data path and control path of the bandwidth control module 350.

[0075] Figure 4A and Figure 4B It shows Figure 3 An example of the internal structure of the bandwidth control path 355 is shown. Figure 5 It shows Figure 3 The example shown is of the data path and control path of the bandwidth control module 350.

[0076] Reference Figure 4A and Figure 5 Memory users (e.g., user 0, user 1, and user 2) can connect to the memory arbitration logic (e.g., via a bandwidth control path) Figure 3 The memory arbitration logic 315 shown is illustrated. Figure 5 As shown, the memory users (user 0, user 1, and user 2) can correspond to the user associated with the weight, the user associated with the input feature map or input, and the user associated with the part and request, respectively.

[0077] For example, if the capacity of the first memory is greater than the sum of the data capacities of the memory users (user 0, user 1, and user 2), the control logic can generate a control signal to cause the bandwidth control path to operate in normal mode. In normal mode, the first memory is used for the data of the target user and the data of other users. The target user is determined based on the layer type and layer configuration of the artificial neural network. In other words, in normal mode, all memory users use the first memory.

[0078] Conversely, when the capacity of the first memory is less than or equal to the sum of the data capacities of the memory users, the control logic can generate a control signal causing the bandwidth control path to operate in an enhanced mode. This enhanced mode uses the second memory for the data of a target user, determined based on the layer type and configuration of the artificial neural network. The control logic can configure one of the memory users (user 0, user 1, and user 2) (e.g., the target user) to use the second memory and increase the bandwidth of the first memory for the remaining users excluding the target user.

[0079] As described above, the bandwidth control path can be used to reconfigure the memory bandwidth for memory users (user 0, user 1, and user 2) based on control signals. For example, four ports can be assigned to each of the memory users (user 0, user 1, and user 2).

[0080] Reference Figure 4A and Figure 5 The neural processing device includes 12 input ports connected to a data acquisition unit and 9 output ports 530 connected to a storage arbitration logic of a first memory. The 12 input ports include four first input ports for requests from a first user 510 in the memory, four second input ports for requests from a second user 515 in the memory, and four third input ports for requests from a third user 520 in the memory, wherein the first user 510 is associated with weights, the second user 515 is associated with an input feature map or input, and the third user 520 is associated with partial sums.

[0081] The lines connected to the four first input ports for requests from the weight-related first user 510 can be connected to a 4-to-1 multiplexer 540. Here, the remaining three first input ports, excluding one of the four first input ports for the weight-related first user 510, can be connected to the three third input ports for the partial and related third user 520, and three 2-to-1 multiplexers 550, 560, and 570, respectively.

[0082] Additionally, the four second input ports of the second user 515 associated with the input feature map can be directly connected to four of the nine output ports 530.

[0083] For example, in the case where all memory users (user 0, user 1, and user 2) use the first memory (or main memory), such as Figure 4A As shown in the left portion, the first user or user 0 (i.e., user 0) among the memory users (user 0, user 1, and user 2) is used. Figure 5 The bandwidth of the first memory of the first user (510) associated with the weight can be limited (or reduced) from 4 to 1.

[0084] The control logic can provide selection signals to a 4-to-1 multiplexer 540 connected to all four first input ports of the first user 510 associated with the weight via a round-robin method, rotating sequentially (e.g., 0→1→2→3→0→1…). Selection signals for the 4-to-1 multiplexer 540 can be provided by a 2-to-1 multiplexer 580 connected to control signal 590. Three 2-to-1 multiplexers 550, 560, and 570 can select requests from a third user 520 associated with the weighted user based on control signal 590 and connect the selected requests to output port 530.

[0085] For example, in the memory users (user 0, user 1, and user 2), user 2 (i.e., Figure 5 In cases where a third user (520) in part and related to the third user uses a second memory (or auxiliary memory or dedicated memory), such as Figure 4A As shown in the right-hand portion, the first memory is for the first user or user 0 among the memory users (user 0, user 1, and user 2). Figure 5 The bandwidth of the first user (510) with weighted correlation can be increased from 1 to 4.

[0086] In this example, the control logic may provide a selection signal to one of the four first input ports (first line) connected to the 4-to-1 multiplexer 540 that is not connected to the 2-to-1 multiplexers 550, 560, and 570. The 2-to-1 multiplexers 550, 560, and 570 may each select the lines of the remaining three first input ports (first line) excluding the stated first input port and connect them to the output port 530. In this example, user 2 or a related third user 520 uses the second memory, therefore requests may not be transmitted to user 2 or the related third user 520.

[0087] Figure 4B It shows the relationship with Figure 4A This is another example that differs from the previous one. Figure 4B In this example, all memory users (user 0, user 1, and user 2) can use main memory. In this example, the bandwidth of main memory for all memory users (user 0, user 1, and user 2) can be limited (or reduced) from 4 to 3. For example, if user 2 of the memory users (user 0, user 1, and user 2) uses dedicated memory, as... Figure 4B As shown in the right-hand section, the control logic can increase the bandwidth for the first user (e.g., user 0) and the second user (e.g., user 1) from 3 to 4.

[0088] Figure 6 An example of the operation flow of a neural processing device is shown. (Refer to...) Figure 6The neural processing device can reconfigure the memory bandwidth for data targeting a target user by performing operations 610 to 660, which will be described below. For ease of description, it will be based on the target user being a user who is partly and related to the user (e.g., ...). Figure 5 To describe the situation involving some and related third-party users (520) Figure 6 Examples, but Figure 6 Examples are not limited to this. For ease of description Figure 6 For example, data from some and related users will be indicated as PSUM.

[0089] In operation 610, the neural processing device calculates the required capacity of PSUM by verifying the layer configuration of the artificial neural network.

[0090] In operation 620, the neural processing device determines whether the auxiliary memory is large enough to store PSUM.

[0091] If, in operation 620, it is determined that the size of the auxiliary memory is sufficient to store the PSUM, the neural processing device uses the auxiliary memory to store the PSUM in operation 630, and in operation 640, it uses a bandwidth control path to increase the bandwidth of the main memory for users other than the target user (e.g., users related to weights). The neural processing device prefers to store the PSUM in the auxiliary memory, which increases the bandwidth of the main memory for the remaining users besides those related to the target user, thereby preventing performance degradation that could occur due to insufficient bandwidth in the main memory.

[0092] Conversely, if operation 620 determines that the size of the auxiliary memory is insufficient, the neural processing device may use the main memory to store the PSUM in operation 650, and use a bandwidth control path in operation 660 to limit (or reduce) the bandwidth of the main memory for the user (e.g., the user with weights).

[0093] Figure 7 An example of the operation and roofline model is shown when a first user, associated with weights, uses a second memory. (See reference...) Figure 7 The diagram shows an input feature map 710 and an output feature map 730 of an artificial neural network including convolutional layers, as well as a roofline model 750 with partial and related users.

[0094] For example, an artificial neural network may include convolutional layers of size 192×32×1×1. In this example, as shown in input feature map 710 and output feature map 730, 192 bytes may be the output channel size, 32 bytes may be the input channel size, and the kernel size may be 1×1. In this example, the weight size may be 12 kilobytes (kbytes).

[0095] Figure 7 The artificial neural network shown, comprising layers with small input channels and kernel size, can have a small number of accumulation iterations. A weighted sum of the input feature map 710 in the artificial neural network can be obtained, and an output channel can be generated. The input channel size of the input feature map 710 can be 32 bytes, so outputs such as the output feature map 730 can be generated only 32 times and then stored in memory. In this case, the computational intensity associated with the partial sum can be, for example, 16 ops / byte (=2ops×32×1×1 / 4 bytes), where 2 ops corresponds to multiplication + accumulation (MAC), and 32 corresponds to the number of input channels accumulated during the computation of a single output pixel. Furthermore, because... Figure 7 The example uses a 1×1 convolution, so 1×1 corresponds to the number of MACs in the kernel direction. Additionally, the 4 bytes correspond to the number of bytes assumed to be needed to compute a single output pixel in the partial sum.

[0096] For example, in the case of a computational intensity of 16 ops / byte in the roofline model 750, the artificial neural network architecture can be memory-constrained. In this example, the neural processing device can determine the user's use of secondary memory to associate the weights.

[0097] Neural processing devices can improve performance by increasing the bandwidth of the first memory for the remaining users (e.g., users associated with the output feature map and / or parts thereof) other than the weighted users who exclusively use the second memory.

[0098] The neural processing device can store weights in a second memory and, based on the input channels and kernel size of the convolutional layer, increase the memory bandwidth of the first memory for users (or third users) who are partially and related to or related to the output feature map.

[0099] Figure 8 An example of operation and roofline model is shown when a second user uses a second memory associated with a feature map. (See reference...) Figure 8 This illustrates a roofline model 800 of a user associated with weights, where a second user uses a second memory associated with the feature map.

[0100] For example, an artificial neural network may include fully connected layers of 2048×1024, and the size of the input feature map may be 2 kilobytes, while the size of the output feature map may be 1 kilobyte. In this example, the weights of each fully connected layer may be very large, and the size of the feature map may be very small, so the neural processing device can determine that a user associated with the small-sized feature map (e.g., a second user) uses a second memory.

[0101] In fully connected layers, loops may not exist, so weight reuse may not occur. Therefore, the computational intensity can be 2 ops / byte. With a computational intensity of 2 ops / byte in the roofline model 800, the architecture of the artificial neural network can be memory-constrained. In this case, the neural processing device can improve performance by storing the feature map data in a second memory and increasing the memory bandwidth of the first memory for the first user associated with the weights among the memory users.

[0102] Figure 9 An example of operation and roofline model is shown in the case where a second memory is used by a partial and associated third user. (Refer to...) Figure 9 The diagram shows the input feature map 910 and output feature map 930 of an artificial neural network including convolutional layers, as well as the roofline model 950 of the user associated with the weights.

[0103] For example, an artificial neural network may include a 448×384×3×3 convolutional layer, where 448 bytes can be the output channel size, 384 bytes can be the input channel size, and 3×3 can be the size of the weight kernel 920. With the input feature map 910 and output feature map 920 being 8×8 in size and 4 bytes in number, the target capacity for the partial sum of data could be 96 kilobytes (=448×8×8×4), allowing layers with smaller feature maps to have fewer weight reuses. The computational intensity of the weights could be 128 ops / byte (=2×8×8).

[0104] With a computational intensity of 128 ops / byte in the roofline model 950, the artificial neural network architecture can be memory-constrained. In this case, the neural processing device can improve performance based on the size of the feature maps of the convolutional layers by storing partial sums of data in a second memory and increasing the memory bandwidth of the first memory for the first user associated with the weights in the memory user.

[0105] Figure 10 Another example of a neural processing device is shown. (See reference) Figure 10 The neural processing device 1000 includes a first memory 1010, a second memory 1030, and a bandwidth control module 1050.

[0106] The first memory 1010 can store, for example, feature maps, weights, partial sums, and other results.

[0107] The second memory 1030 may be independent of the first memory 1010 and has a limited size. The second memory 1030 may be, for example, a dedicated memory for partial sums. The result of a partial sum may be selectively stored in either the first memory 1010 or the second memory 1030.

[0108] The bandwidth control module 1050 can determine which memory to store the portion and the result by reconfiguring the bandwidth of the memories 1010 and 1030 via an interface (i.e., bandwidth control path 1055), with the bandwidth control path 1055 selecting one of the two memories 1010 and 1030 based on the control signal of the control logic 1040.

[0109] Control logic 1040 can generate control signals such that the first memory 1010 is used when a small amount of memory bandwidth is required, and the second memory 1030 is used as a dedicated memory for partial sums when a large amount of memory bandwidth is required. Therefore, performance degradation that may occur due to conflicts between memory banks can be prevented while using memories 1010 and 1030 more efficiently.

[0110] For example, users (e.g., users 1070 and 1090) who read or write partial sums can be connected to the first memory 1010 or the second memory 1030 via bandwidth control path 1055 based on control signals from control logic 1040. In this example, the remaining users 1060 and 1080, other than users 1070 and 1090 connected to the second memory 1030, can receive greater bandwidth provided by the first memory 1010 via bandwidth control path 1055.

[0111] For example, when only the first memory 1010 is used, the bandwidth control module 1050 can allocate the bandwidth of the first memory 1010 based on control signals and provide the allocated bandwidth to all users 1060, 1070, 1080, and 1090 connected to the bandwidth control path 1055. Unlike this example, when users 1070 and 1090 use the second memory 1030, the bandwidth control module 1050 can provide the bandwidth of the first memory 1010 to the remaining users 1060 and 1080, excluding those associated with users 1070 and 1090, via the bandwidth control path 1055.

[0112] Figure 11 Another example of the operation flow of a neural processing device is shown. (See reference...) Figure 11 The neural processing device can determine the memory to be used from main memory and dedicated memory by performing operations 1110 to 1150, which will be described below. The neural processing device may include memory units, which include main memory and dedicated memory for partial and partial storage.

[0113] In operation 1110, the neural processing device determines whether the bandwidth of the main memory is insufficient to perform operations associated with the layers of the artificial neural network.

[0114] In operation 1120, when it is determined in operation 1110 that the bandwidth of the main memory is insufficient to perform the operations associated with the layers of the artificial neural network, the neural processing device determines whether the size of the dedicated memory is sufficient to store the partial sum results. In operation 1130, when it is determined in operation 1120 that the size of the dedicated memory is insufficient to store all the partial sum results, the neural processing device performs tiling on the outputs (or output matrices) of the artificial neural network layers until the dedicated memory becomes available. The neural processing device may repeatedly determine whether the size of the dedicated memory is sufficient to store the tiled outputs (or output matrices). In operation 1140, when it is determined in operation 1120 that the size of the dedicated memory is sufficient, the neural processing device uses the dedicated memory to store the partial sum results or the tiled outputs of the layers.

[0115] In operation 1150, when it is determined in operation 1110 that the bandwidth of the main memory is sufficient to perform operations associated with the layers of the artificial neural network, the neural processing device uses the main memory to perform operations associated with the layers of the artificial neural network.

[0116] Figure 12A and Figure 12B Examples of neural processing devices using a first memory for data of a target user are shown, as well as examples of neural processing devices using a second memory for data of a target user. Figure 13 An example of a bandwidth control bus is shown.

[0117] Reference Figure 12A This illustrates the connection relationships between bandwidth control paths 1230 and users 1240, 1250, 1260, and 1270 when the neural processing device uses the first memory 1210 for data from the target user. (Refer to...) Figure 12B The diagram illustrates the connection between the bandwidth control path 1230 and users 1240, 1250, 1260, and 1270 when the neural processing device uses both the first memory 1210 and the second memory 1220.

[0118] For example, the first memory 1210 may have a capacity of 1 megabyte (Mbyte). For example, the second memory 1220 may have a capacity of 48 kilobytes.

[0119] For example, when the size of the output feature map is greater than 16×16, where 16×16 corresponds to the height and width of the pixel unit, and the neural processing device generates 32 output channels at a time, each pixel requires 6 bytes to perform partial summation, the capacity of the second memory 1220 can be 48 kilobytes (=32×16×16×6).

[0120] In this example, the target capacity of the data for the target user may be greater than the capacity of the second memory 1220 (e.g., 48 kilobytes), therefore the neural processing device can generate a control signal causing the bandwidth control path 1230 to operate in the normal mode using the first memory 1210. Figure 13 As shown in the left portion 1310, the bandwidth control path 1230 can transmit user data to the first memory 1210 based on control signals used for operation in normal mode, by reducing the four channels of user 1240 (as a weighted user) to one, while maintaining four channels for user 1250 (as a partial and reading user). In this example, the second memory 1220 may not be used.

[0121] For example, when the size of the output feature map is less than or equal to 16×16, the target capacity of the data for the target user can be less than the capacity of the second memory 1220 (e.g., 48 kilobytes). Therefore, the neural processing device can generate control signals to cause the bandwidth control path 1230 to operate in an enhanced mode using the second memory 1220. Figure 13 As shown in the right-hand portion 1330, the bandwidth control path 1230 can, based on control signals for operation in enhanced mode, transfer data of user 1240 (as a weighted user) to the first memory 1210, while maintaining four channels for user 1240. In this example, the four channels of user 1250 (as a portion of the user's data) can be connected to the second memory 1220.

[0122] For example, when the total number of all memory users is 20 (= 8 IFM reads + 4 weighted reads + 4 PSUM reads + 4 OFM writes), the number of users in the first memory 1210 can be 17 (= 8 IFM reads + 1 weighted read + 4 PSUM reads + 4 PSUM writes) in normal mode and decreases to 12 (= 8 IFM reads + 4 weighted reads) in enhanced mode.

[0123] Reference Figure 12BWeighted read user 1240 and partial read user 1250 can be connected to bandwidth control path 1230. Input feature map read user 1270 can be connected only to first memory 1210, and partial write user 1260 can be selectively connected to first memory 1210 or second memory 1220.

[0124] For example, in such Figure 12A In the case shown where only the first memory 1210 is used, the bandwidth control bus can allocate the bandwidth of the first memory 1210 and provide the allocated bandwidth to partial and read users 1250 and weighted read users 1240, so the amount to be allocated to weighted read users 1240 can be reduced.

[0125] For another example, in such Figure 12B In the case shown, where both the first memory 1210 and the second memory 1220 are used, the bandwidth control bus can provide sufficient bandwidth to the weighted reading user 1240 through the first memory 1210.

[0126] For example, when the spatial dimension of the output of a neural processing device that fully reuses weights in the spatial direction is greater than a predetermined standard, the number of weight reuses may increase. Therefore, the weight reading user 1240 may not require a large bandwidth. However, when the spatial dimension of the output is smaller than the standard, the number of weight reuses may decrease, and thus, due to the reduced number of weight reuses, the weight reading user 1240 may require a larger bandwidth. In this case, the neural processing device can use the second memory 1220 to prevent performance degradation that may occur due to insufficient bandwidth of the first memory 1210.

[0127] Figure 14 An example of the operation flow of a neural processing device is shown. (Refer to...) Figure 14 The neural processing device can generate control signals for a bandwidth control path by performing operations 1410 to 1440, which will be described below. The neural processing device may include a first memory and a second memory having a smaller capacity than the first memory.

[0128] In operation 1410, the neural processing device calculates a target volume of data for a target user, which is determined from memory users based on the layer configuration of an artificial neural network. A memory user may be associated with at least one of the input feature map, weights, and portions / or output feature maps, but examples are not limited thereto.

[0129] In operation 1420, the neural processing device compares the target capacity calculated in operation 1410 with the capacity of the second memory.

[0130] In operation 1430, the neural processing device, based on the comparison result of operation 1420, determines a target memory from between the first memory and the second memory in which the target user's data will be stored. In response to the comparison result of operation 1420, if the target capacity is greater than the capacity of the second memory, the neural processing device may determine the first memory as the target memory. Conversely, in response to the comparison result of operation 1420, if the target capacity is less than or equal to the capacity of the second memory, the neural processing device may determine the second memory as the target memory.

[0131] In operation 1440, the neural processing device generates a control signal for a bandwidth control path to reconfigure the memory bandwidth of the memory user, such that the target user's data is stored in the target memory determined in operation 1430.

[0132] For example, if operation 1430 determines that the first memory is the target memory, the neural processing device may generate a control signal causing the target user's data to be stored in the first memory in operation 1440, thereby limiting (or reducing) the bandwidth of the first memory for data of at least one remaining user in the memory users who have excluded the target user. Conversely, if operation 1430 determines that the second memory is the target memory, the neural processing device may generate a control signal causing the target user's data to be stored in the second memory in operation 1440, thereby increasing the bandwidth of the first memory for data of at least one remaining user in the memory users who have excluded the target user.

[0133] Optionally, the neural processing device may compare the capacity of the first memory with the sum of the data capacities of the memory users. In operation 1440, in response to the comparison result that the capacity of the first memory is greater than the sum of the data capacities of the memory users, the neural processing device may generate a control signal causing the bandwidth control path to operate in a normal mode that uses the first memory for the data of a target user, the target user being determined based on the layer type and layer configuration of the artificial neural network. Conversely, in operation 1440, in response to the comparison result that the capacity of the first memory is less than or equal to the sum of the data capacities of the memory users, the neural processing device may generate a control signal causing the bandwidth control path to operate in an enhanced mode that uses the second memory for the data of a target user, the target user being determined based on the layer type and layer configuration of the artificial neural network.

[0134] For example, in an artificial neural network that includes convolutional layers and where the target user is a first user associated with weights, in operation 1440, the neural processing device may store the weights in a second memory based on the input channels and kernel size of the convolutional layers. In this example, the neural processing device may increase the memory bandwidth of the first memory for a third user among the memory users, who is associated with a portion and / or the output feature map.

[0135] For example, when the artificial neural network includes fully connected layers and the target user is a second user associated with the feature map, in operation 1440, the neural processing device can store the feature map data in a second memory. In this example, the neural processing device can increase the memory bandwidth of the first memory for the first user among the memory users associated with the weights.

[0136] In yet another example, where the artificial neural network includes convolutional layers and the target user is a third user associated with the partial sum, in operation 1440, the neural processing device may store the partial sum data in a second memory based on the size of the feature maps of the convolutional layers. In this example, the processing device may increase the memory bandwidth of the first memory for the first user among the memory users associated with the weights.

[0137] Regarding Figure 1 , Figure 3 , Figure 5 , Figure 7 , Figure 9 , Figure 10The neural processing device, NPU 110, storage device 111, input feature map acquirer 112, weight acquirer 113, PSUM acquirer 114, OFM writer 115, DRAM 130, neural processing devices 300 and 1000, first memory 310 and 1010, second memory 330 and 1030, bandwidth control module 350 and 1050, control logic 390 and 1040, and other devices, apparatuses, units, modules, and components described in Figure 12 are implemented by hardware components. Examples of hardware components that can be used to perform the operations described in this application include, where appropriate, controllers, sensors, generators, drivers, memories, comparators, arithmetic logic units, adders, subtractors, multipliers, dividers, integrators, and any other electronic components configured to perform the operations described in this application. In other examples, one or more of the hardware components performing the operations described in this application are implemented by computing hardware (e.g., by one or more processors or computers). A processor or computer may be implemented by one or more processing elements, such as logic gate arrays, controllers and arithmetic logic units, digital signal processors, microcomputers, programmable logic controllers, field-programmable gate arrays, programmable logic arrays, microprocessors, or any other means or combination of means configured to respond to and execute instructions in a defined manner to achieve a desired result. In one example, the processor or computer includes or is connected to one or more memories storing instructions or software executed by the processor or computer. Hardware components implemented by the processor or computer can execute instructions or software (such as an operating system (OS) and one or more software applications running on the OS) to perform the operations described in this application. The hardware components may also access, manipulate, process, create, and store data in response to the execution of instructions or software. For the sake of brevity, the singular terms “processor” or “computer” may be used in the description of the examples described in this application; however, in other examples, multiple processors or computers may be used, or a processor or computer may include multiple processing elements, or multiple types of processing elements, or both. For example, a single hardware component or two or more hardware components may be implemented by a single processor, or two or more processors, or a processor and a controller. One or more hardware components may be implemented by one or more processors, or processors and controllers, and one or more other hardware components may be implemented by one or more other processors, or additional processors and additional controllers. One or more processors, or processors and controllers, may implement a single hardware component, or two or more hardware components.The hardware components can be any one or more with different processing configurations, examples of which include a single processor, a discrete processor, a parallel processor, a single instruction single data (SISD) multiprocessing, a single instruction multiple data (SIMD) multiprocessing, multiple instruction single data (MISD) multiprocessing, and multiple instruction multiple data (MIMD) multiprocessing.

[0138] Figures 1 to 14 The methods for performing the operations described in this application, as shown, are executed by computing hardware (e.g., by one or more processors or a computer), which is implemented as described above to execute instructions or software to perform the operations performed by the methods described in this application. For example, a single operation or two or more operations may be executed by a single processor, or two or more processors, or a processor and a controller. One or more operations may be executed by one or more processors, or a processor and a controller, and one or more other operations may be executed by one or more other processors, or additional processors and additional controllers. One or more processors or a processor and a controller may execute a single operation or two or more operations.

[0139] Instructions or software for controlling a processor or computer to implement hardware components and perform the methods described above are written as computer programs, code segments, instructions, or any combination of computer programs, code segments, and instructions to individually or collectively instruct or configure the processor or computer to operate as a machine or special-purpose computer to perform operations performed by the hardware components and methods described above. In one example, the instructions or software include machine code (such as machine code generated by a compiler) that is directly executed by the processor or computer. In another example, the instructions or software include high-level code that is executed by the processor or computer using an interpreter. Those skilled in the art can readily write instructions or software based on the block diagrams and flowcharts shown in the accompanying drawings and the corresponding descriptions in the specification, which disclose algorithms for performing operations performed by the hardware components and methods described above.

[0140] Instructions or software used to control a processor or computer to implement hardware components and perform the methods described above, along with any associated data, data files, and data structures, are recorded, stored, or fixed in, or on, one or more non-transitory computer-readable storage media. Examples of non-transitory computer-readable storage media include read-only memory (ROM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disc storage, hard disk drive (HDD), solid-state drive (SSD), flash memory, card-type storage (such as multimedia microcards or cards (e.g., Secure Digital (SD) or Extreme Digital (XD))), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage, hard disk, solid-state disk, and any other device configured to store instructions or software and any associated data, data files and data structures in a non-transitory manner and to provide instructions or software and any associated data, data files and data structures to a processor or computer so that the processor or computer can execute the instructions.

[0141] While this disclosure includes specific examples, it will be clear upon understanding this disclosure that various changes in form and detail may be made in these examples without departing from the spirit and scope of the claims and their equivalents. The examples described herein are to be considered descriptive only and not for limiting purposes. The description of features or aspects in each example is to be applied to similar features or aspects in other examples. Suitable results may be achieved if the described techniques are performed in a different order, and / or if the components in the described system, architecture, apparatus, or circuit are combined in a different manner, and / or replaced or supplemented by other components or their equivalents.

[0142] Therefore, the scope of the disclosure is not limited by the specific embodiments, but by the claims and their equivalents, and all variations within the scope of the claims and their equivalents should be interpreted as included in the disclosure.

Claims

1. A neural processing device, comprising: The first memory is configured to store general-purpose data; The second memory is separate from the first memory and has a smaller capacity than the first memory. The bandwidth control path is configured to reconfigure the memory bandwidth based on control signals so that a memory user can use one of the first memory and the second memory. as well as The control logic is configured to: calculate a target capacity for data of a target user in the memory user, and generate a control signal based on a comparison of the target capacity with the capacity of a second memory, to store the target user data in the second memory, wherein the target user in the memory user is determined based on a layer configuration of an artificial neural network, wherein the layer configuration represents the form of the configuration of layers based on a type of artificial neural network.

2. The neural processing device as claimed in claim 1, wherein, The control logic is also configured as follows: In response to the target capacity being greater than the capacity of the second memory, a control signal is generated to store the target user's data in the first memory and to reduce the bandwidth of the first memory for data of at least one remaining user in the memory users that excludes the target user.

3. The neural processing device as claimed in claim 1, wherein, The control logic is also configured as follows: In response to the target capacity being less than or equal to the capacity of the second memory, a control signal is generated to store the target user's data in the second memory and to increase the bandwidth of the first memory for data of at least one remaining user in the memory users who have excluded the target user.

4. The neural processing device as claimed in claim 1, wherein, The control logic is also configured as follows: In response to the fact that the capacity of the first memory is greater than the sum of the data capacities of the memory users, a control signal is generated to operate the bandwidth control path in normal mode, in which the first memory is used for the data of all memory users.

5. The neural processing device as claimed in claim 1, wherein, The control logic is also configured as follows: In response to the fact that the capacity of the first memory is less than or equal to the sum of the data capacity of the memory users, a control signal is generated to operate the bandwidth control path in enhanced mode, which uses the second memory for the target user's data.

6. The neural processing device according to any one of claims 1 to 5, wherein, When the artificial neural network includes convolutional layers and the target user is the first user associated with the weights among the memory users, the control logic is also configured as follows: The weights are stored in a second memory based on the input channels and kernel size of the convolutional layer, and the bandwidth of the first memory is increased for a third user in the memory users, who are associated with the output feature map or partial sum, where the partial sum represents the result of the partial sum generated during the operation.

7. The neural processing device according to any one of claims 1 to 5, wherein, When the artificial neural network includes fully connected layers and the target user is a second user associated with the feature map, the control logic is also configured as follows: The feature map data is stored in a second memory, and the bandwidth of the first memory is increased for a first user among the memory users, who are associated with weights.

8. The neural processing device according to any one of claims 1 to 5, wherein, When the artificial neural network includes convolutional layers and the target user is a third user associated with a part of the network, the control logic is also configured as follows: Based on the size of the feature map of the convolutional layer, the partial sum data is stored in a second memory, and the bandwidth of the first memory is increased for a first user among the memory users, the first user being associated with the weights. Here, "partial sum" represents the result of the partial sum generated during the operation.

9. The neural processing device according to any one of claims 1 to 5, wherein, Each memory user in the memory user group is associated with any one of the input feature map, weights, and output feature map or partial sum. Here, partial sum represents the result of partial sums generated during the operation.

10. The neural processing device according to any one of claims 1 to 5, wherein, The bandwidth control path includes twelve input ports and nine output ports connected to the first memory. The twelve input ports include: four first input ports for a first user associated with weights among the memory users; four second input ports for a second user associated with the input feature map among the memory users; and four third input ports for a third user associated with partial sums among the memory users. Here, "partial sum" represents the result of the partial sum generated during the operation.

11. The neural processing device of claim 10, wherein, The four first input ports are connected to a 4-to-1 multiplexer. The remaining three first input ports, after excluding one of the four, are respectively connected to three third input ports for a third user and three 2-to-1 multiplexers. The four second input ports for the second user are directly connected to four of the nine output ports.

12. The neural processing device of claim 11, wherein, When the bandwidth of the first memory for the first user is limited, the control logic is also configured as follows: Selection signals are sequentially provided to the 4-to-1 multiplexer using a polling method.

13. The neural processing device of claim 11, wherein, When the bandwidth of the first memory for the first user increases, the control logic is also configured to: A selection signal is provided to one of the four first input ports, which is not connected to the three third input ports via a 2-to-1 multiplexer.

14. A method of operating a neural processing device, the neural processing device comprising a first memory and a second memory, the second memory having a capacity smaller than that of the first memory, the method of operating the device comprising: Calculate the target capacity of data for target users in the memory users, which is determined based on the layer configuration of the artificial neural network, where the layer configuration represents the form of the layer configuration based on the type of artificial neural network; Compare the target capacity with the capacity of the second memory; Based on the comparison results, a target memory is determined from the first memory and the second memory, in which the target user's data will be stored; and Generate control signals for the bandwidth control path to reconfigure the memory bandwidth for the memory user based on the control signals, so as to store the target user's data in the target memory.

15. The operating method as described in claim 14, wherein, The steps for determining the target memory include: In response to the target capacity being greater than the capacity of the second memory, the first memory is designated as the target memory. The steps for generating control signals include: A control signal is generated to store the target user's data in the first memory and to reduce the bandwidth of the first memory for data of at least one remaining user in the memory that excludes the target user.

16. The operating method as described in claim 14, wherein, The steps for determining the target memory include: In response to the target capacity being less than or equal to the capacity of the second memory, the second memory is determined as the target memory. The steps for generating control signals include: A control signal is generated to store the target user's data in the second memory and to increase the bandwidth of the first memory for data of at least one remaining user in the memory users who have excluded the target user.

17. The operating method as described in claim 14, further comprising: The capacity of the first memory is compared with the sum of the data capacity of the memory users. The steps for generating control signals include: In response to the comparison result that the capacity of the first memory is greater than the sum of the data capacities of the memory users, a control signal is generated to operate the bandwidth control path in normal mode, in which the first memory is used for data of all memory users.

18. The operating method as described in claim 17, wherein, The steps for generating control signals also include: In response to the fact that the capacity of the first memory is less than or equal to the sum of the data capacity of the memory users, a control signal is generated to operate the bandwidth control path in enhanced mode, which uses the second memory for the target user's data.

19. The method of operation as described in any one of claims 14 to 18, wherein, When the artificial neural network includes convolutional layers and the target user is a first user associated with the weights, the steps for generating control signals include: The weights are stored in a second memory based on the input channels and kernel size of the convolutional layer; and Increase the bandwidth of the first memory for a third user among the memory users, the third user being associated with the output feature map or a portion thereof. Here, "partial sum" represents the result of the partial sum generated during the operation.

20. The method of operation as described in any one of claims 14 to 18, wherein, When the artificial neural network includes fully connected layers and the target user is a second user associated with the feature map, the steps for generating control signals include: The feature map data is stored in a second memory; and Increase the bandwidth of the first memory for the first user among the memory users, the first user being associated with a weight.

21. The method of operation as described in any one of claims 14 to 18, wherein, When the artificial neural network includes convolutional layers and the target user is a third user associated with and partially associated with the target user, the steps for generating control signals include: The partial sum data is stored in a second memory based on the size of the feature map of the convolutional layer; and Increase the bandwidth of the first memory for the first user among the memory users, the first user being associated with a weight. Here, "partial sum" represents the result of the partial sum generated during the operation.

22. The method of operation as described in any one of claims 14 to 18, wherein, Each memory user is associated with any one of the input feature map, weights, and output feature map or partial sum. Here, "partial sum" represents the result of the partial sum generated during the operation.

23. A non-transitory computer-readable storage medium storing instructions that, when executed by a processor, cause the processor to perform the method of operation according to any one of claims 14 to 22.

24. A neural processing device, comprising: First storage; The second storage unit has a capacity smaller than that of the first storage unit; The bandwidth control path is configured to reconfigure the memory bandwidth based on control signals so that a memory user can use either the first memory bank or the second memory bank. as well as The control logic is configured as follows: The target capacity for data for a target user within the memory user group is calculated. The target user group is determined based on the layer configuration of an artificial neural network, where the layer configuration represents the form of the layer configuration based on the type of artificial neural network. Compare the target capacity with the capacity of the second memory bank, and In response to the target capacity being less than or equal to the capacity of the second storage unit, a control signal is generated to store the target user's data in the second storage unit and to increase the bandwidth of the first storage unit for data of at least one remaining user in the memory users who have excluded the target user.

25. The neural processing device of claim 24, wherein, The control logic is also configured to: generate a control signal in response to the target capacity being greater than the capacity of the second storage bank, to store the target user's data in the first storage bank, and to reduce the bandwidth of the first storage bank for data of at least one remaining user in the memory users who have excluded the target user.

26. The neural processing device of claim 25, wherein, When the artificial neural network includes convolutional layers and the target user is the first user associated with the weights among the memory users, the control logic is also configured as follows: The weights are stored in a second memory bank based on the input channels and kernel size of the convolutional layer, and the bandwidth of the first memory bank is increased for a third user in the memory bank, who is associated with the output feature map or partial sum, where the partial sum represents the result of the partial sum generated during the operation.

27. The neural processing device of claim 25, wherein, When the artificial neural network includes fully connected layers and the target user is a second user associated with the feature map, the control logic is also configured as follows: The feature map data is stored in a second storage unit, and the bandwidth of the first storage unit is increased for a first user among the memory users, who are associated with weights.

28. The neural processing device of claim 25, wherein, When the artificial neural network includes convolutional layers and the target user is a third user associated with a part of the network, the control logic is also configured as follows: Based on the size of the feature map of the convolutional layer, the partial sum data is stored in the second storage, and the bandwidth of the first storage is increased for the first user in the memory users, who are associated with weights. Here, "partial sum" represents the result of the partial sum generated during the operation.

Images