Electronic device and control method of electronic device

Abstract

A memory of an electronic device stores three-dimensional input data including (i) an input value, (ii) first kernel information, and (iii) second kernel information. A processor includes multiplication modules corresponding to channels and performs a convolution operation based on the input value and weights through the multiplication modules. Based on the deep convolution operation, the processor of the electronic device controls an input selection module to (a) configure the input value to correspond to a first channel among the channels and (b) input the input value to two or more multiplication modules among the multiplication modules. The processor inputs the weights, obtains intermediate values, and sums up the intermediate values corresponding to positions of the kernels, respectively, among the intermediate values through a first intermediate value accumulation module, and obtains an output value based on a result of each sum.

Description

Technical Field

[0001] This disclosure relates to electronic devices and methods for controlling electronic devices, and more specifically, to electronic devices capable of efficiently performing convolution operations and methods for controlling electronic devices. Background Technology

[0002] Recently, developments in the field of artificial intelligence (AI) have accelerated the development of neural network accelerators or deep learning chipsets for effectively implementing and executing AI functions.

[0003] In the case of neural network accelerators used to perform convolution operations, techniques are needed for a single neural network accelerator to efficiently handle both 3D convolution operations and deepwise convolution operations.

[0004] However, in the case of neural network accelerators with hardware structures for parallel processing in the input channel direction according to the prior art, when performing 3D convolution operations, the operation can be performed by using all operators, while when performing depthwise convolution operations, even if the amount of computation required is relatively small compared to the case of performing 3D convolution operations, there is a problem that operators may not be effectively utilized. Summary of the Invention

[0005] [Technical Issues]

[0006] One aspect of this disclosure is to provide an electronic device capable of efficiently performing convolution operations using a parallel hardware architecture of a neural network accelerator, and a method for controlling the electronic device.

[0007] [Technical Solution]

[0008] According to one aspect of this disclosure, an electronic device includes: a memory configured to store three-dimensional input data, the three-dimensional input data including (i) a plurality of input values ​​divided based on a plurality of channels, (ii) first kernel information about a kernel including a plurality of weights for each of the plurality of channels, and (iii) second kernel information generated by converting a plurality of weights configured in a two-dimensional matrix form for each of the plurality of channels into a three-dimensional matrix form. A processor includes a plurality of multiplication modules corresponding to the plurality of channels. The processor is configured to perform convolution operations based on the plurality of input values ​​and the plurality of weights through the plurality of multiplication modules. The processor is also configured to: based on the fact that the convolution operation is a depthwise convolution operation, control the input selection module (a) to configure multiple input values ​​to correspond to the first channel among multiple channels, and (b) input the multiple input values ​​to two or more multiplication modules among multiple multiplication modules, input the first set of weights corresponding to the first channel one by one to the two or more multiplication modules based on the second kernel information, perform multiplication operations with each weight among multiple weights for each of the multiple input values ​​through the two or more multiplication modules to obtain multiple intermediate values ​​based on each multiplication operation result, and sum the intermediate values ​​corresponding to the kernel positions among the multiple intermediate values ​​through the first intermediate value accumulation module to obtain multiple output values ​​based on each summation result.

[0009] Each of the multiple input values ​​corresponding to the first channel is an input to the input selection module in each preset cycle, and the input selection module is configured to send each of the multiple input values ​​to two or more multiplication modules in each preset cycle.

[0010] The two or more channels include a first channel and at least one channel adjacent to the first channel, and the number of the two or more multiplication modules corresponds to the number of weights included in the first kernel information.

[0011] The core is a two-dimensional core, and the processor also includes a buffer that stores intermediate values ​​corresponding to rows of the core among a plurality of intermediate values, and the processor obtains a plurality of output values ​​by summing the intermediate values ​​stored in the buffer corresponding to each position of the core through a first intermediate value accumulation module.

[0012] The processor is also configured to obtain multiple output values ​​by performing convolution operations in parallel using two or more multiplication modules corresponding to the number of multiple weights included in the kernel.

[0013] The processor is also configured to: based on the fact that the convolution operation is a three-dimensional convolution operation, control the input selection module to bypass the input values ​​input to the input selection module to multiple multiplication modules, and input the second set of weights corresponding to each of the multiple multiplication modules to the multiple multiplication modules based on the first core information.

[0014] The processor also includes a second intermediate value accumulation module to sum the intermediate values ​​of each of the multiple channels obtained through the multiple multiplication modules.

[0015] According to another aspect of this disclosure, a method for controlling an electronic device includes: performing a convolution operation based on three-dimensional input data through a plurality of multiplication modules corresponding to a plurality of channels, the three-dimensional input data including (i) a plurality of input values ​​divided according to the plurality of channels, (ii) first kernel information including a kernel with respect to a plurality of weights for each of the plurality of channels, and (iii) second kernel information generated by converting a plurality of weights configured in a two-dimensional matrix form for each of the plurality of channels into a three-dimensional matrix form. Since the convolution operation is a depthwise convolution operation, the method further includes controlling an input selection module to (a) configure a plurality of input values ​​corresponding to a first channel among the plurality of channels, and (b) input the plurality of input values ​​to two or more of the plurality of multiplication modules; inputting a first set of weights corresponding to the first channel one by one into the two or more multiplication modules based on the second kernel information; performing a multiplication operation with each weight among the plurality of weights for each of the plurality of input values ​​through the two or more multiplication modules to obtain a plurality of intermediate values ​​based on each multiplication result; and summing the intermediate values ​​corresponding to the kernel positions among the plurality of intermediate values ​​through a first intermediate value accumulation module to obtain a plurality of output values ​​based on each summation result.

[0016] Each of the multiple input values ​​corresponding to the first channel is an input to the input selection module in each preset cycle, and the input selection module sends each input value to two or more multiplication modules in each preset cycle.

[0017] The two or more channels include a first channel and at least one channel adjacent to the first channel, and the number of the two or more multiplication modules corresponds to the number of weights included in the first core.

[0018] The method further includes obtaining multiple output values ​​by summing the intermediate values ​​stored in the buffer corresponding to each position of the core using a first intermediate value accumulation module. The buffer is configured to store the intermediate values ​​corresponding to rows of the core from among the multiple intermediate values.

[0019] The method also includes obtaining multiple output values ​​by performing convolution operations in parallel using two or more multiplication modules corresponding to the number of multiple weights included in the kernel.

[0020] The method further includes: based on the fact that the convolution operation is a three-dimensional convolution operation, controlling the input selection module to bypass multiple input values ​​input to the input selection module to multiple multiplication modules; and based on the first kernel information, inputting a second set of weights corresponding to each of the multiple multiplication modules to the multiple multiplication modules.

[0021] According to another aspect of this disclosure, a non-transitory computer-readable recording medium includes a program for performing a control method for an electronic device. The electronic device performs a convolution operation based on three-dimensional input data through a plurality of multiplication modules corresponding to a plurality of channels. The three-dimensional input data includes (i) a plurality of input values ​​divided according to the plurality of channels, (ii) first kernel information about the kernel including weights for each of the plurality of channels, and (iii) second kernel information generated by converting a plurality of weights configured in a two-dimensional matrix form for each of the plurality of channels into a three-dimensional matrix form. The method for controlling the electronic device includes: based on the fact that the convolution operation is a depthwise convolution operation, controlling an input selection module such that multiple input values ​​corresponding to a first channel among multiple channels are input into all two or more multiplication modules; based on second kernel information, inputting a first set of weights corresponding to the first channel one by one into two or more multiplication modules; performing a multiplication operation with each weight among multiple weights for each of the multiple input values ​​through two or more multiplication modules to obtain multiple intermediate values ​​based on each multiplication result; and summing the intermediate values ​​corresponding to the kernel positions among the multiple intermediate values ​​through a first intermediate value accumulation module to obtain multiple output values ​​based on each summation result.

[0022] According to another aspect of this disclosure, a method for accelerating the computation of convolution operations is provided by using a parallel hardware architecture of a neural network accelerator that includes multiple multiplication modules and an input selection module. The method includes: receiving three-dimensional input data by multiple multiplication modules, the three-dimensional input data including: (i) multiple input values ​​divided based on multiple channels, (ii) first kernel information about the kernel including multiple weights for each of the multiple channels, and (iii) second kernel information generated by converting multiple weights corresponding to the multiple channels through the multiple multiplication modules based on the three-dimensional input data, including: based on the convolution operation being a depthwise convolution operation, controlling an input selection module to: (a) configure multiple input values ​​corresponding to a first channel among the multiple channels, and (b) input the multiple input values ​​to two or more of the multiple multiplication modules; inputting a first set of weights corresponding to the first channel one by one to the two or more multiplication modules based on the second kernel information; performing a multiplication operation with each weight among the multiple weights for each of the multiple input values ​​through the two or more multiplication modules to obtain multiple intermediate values ​​based on each multiplication operation result; summing the intermediate values ​​corresponding to the kernel positions among the multiple intermediate values ​​through a first intermediate value accumulation module to obtain multiple output values ​​based on each summation result, and sending the obtained multiple output values ​​to a device connected to a neural network accelerator. Attached Figure Description

[0023] The above and other aspects, features and advantages of certain embodiments of the present disclosure will become clearer from the following description taken in conjunction with the accompanying drawings, in which:

[0024] Figure 1 The configuration of an electronic device according to an embodiment of the present disclosure is shown;

[0025] Figure 2 Multiple modules according to embodiments of the present disclosure, as well as the input and output data of the multiple modules, are shown;

[0026] Figures 3 to 4B A portion of the depthwise convolution operation process according to an embodiment of the present disclosure is shown;

[0027] Figure 5 Several modules for performing two-dimensional convolution operations according to this disclosure are shown;

[0028] Figures 6 to 8 A portion of a two-dimensional convolution operation according to various embodiments of the present disclosure is shown;

[0029] Figure 9 Multiple modules for performing three-dimensional (3D) convolution operations according to embodiments of the present disclosure are shown;

[0030] Figure 10 and Figure 11A portion of a 3D convolution operation according to various embodiments of the present disclosure is shown; and

[0031] Figure 12 A method for controlling an electronic device according to an embodiment of the present disclosure is shown. Detailed Implementation

[0032] This disclosure can have various modifications and includes various embodiments, some of which are shown in the accompanying drawings and described in detail in the detailed description. However, this disclosure is not intended to limit the embodiments described herein, but rather to include various modifications, equivalents, and / or substitutions. In the context of the accompanying drawings, similar reference numerals may be used for similar components.

[0033] In describing this disclosure, well-known functions or structures have not been described in detail, as such details would obscure the disclosure unnecessarily. Furthermore, the embodiments described below can be modified in various different forms, and the scope of the technical concepts of this disclosure is not limited to the following embodiments. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of this disclosure to those skilled in the art.

[0034] The terminology used in this disclosure is for the purpose of describing particular embodiments only and is not intended to limit the scope of the claims. The singular expression includes multiple representations unless the context clearly indicates otherwise.

[0035] In the exemplary embodiments of this disclosure, the terms “having,” “may have,” “including,” and “may include” indicate the presence of an element of a corresponding feature (e.g., a numerical value, function, operation, or part thereof) and do not exclude the presence of additional features.

[0036] In the specification, the terms “A or B”, “at least one of A and / or B”, or “one or more of A and / or B” can include all possible combinations of the items listed together. For example, the term “at least one of A and / or B” includes (1) including at least one A, (2) including at least one B, or (3) including both at least one A and at least one B.

[0037] Furthermore, the expressions “first”, “second”, etc., used in this disclosure can indicate various components regardless of the order and / or importance of the components, and can be used to distinguish one component from other components without limiting the corresponding components.

[0038] When any component (e.g., the first component) is (operably or communicatively) coupled / coupled to another component (e.g., the second component) or connected to another component (e.g., the second component), it should be understood that any component can be directly coupled / coupled to another component, or can be coupled / coupled to another component through another component (e.g., the third component).

[0039] On the other hand, when any component (e.g., the first component) is "directly coupled to" or "directly connected to" another component (e.g., the second component), it should be understood that other components (e.g., the third component) do not exist between the directly coupled components.

[0040] The expression “configured as” as used in this disclosure may be used interchangeably with other expressions, such as “suitable for,” “capable of,” “designed to,” “adapted to,” “made of,” and “capable of.” The term “configured as” does not necessarily mean that the device is “specifically designed for” in terms of hardware.

[0041] Conversely, in some cases, the expression "device, configured to" can refer to, for example, that the device is "capable" of performing operations together with another device or component. For example, the phrase "processor, configured to perform A, B, and C" can refer to, for example, a dedicated processor (e.g., an embedded processor) for performing the corresponding operations, or a general-purpose processor (e.g., a central processing unit (CPU) or application processor) that can perform the corresponding operations by executing one or more software programs stored in a memory device.

[0042] Terms such as “module,” “unit,” and “part” can refer to, for example, an element that performs at least one function or operation, and such an element can be implemented as hardware or software, or a combination of hardware and software. Furthermore, except when each of a plurality of “modules,” “units,” “parts,” etc., needs to be implemented in separate hardware, components can be integrated into at least one module or chip and implemented in at least one processor.

[0043] It should be understood that various elements and areas in the figures may not be shown to scale. Therefore, the scope of this disclosure is not limited to the relative sizes or spacing drawn from the figures.

[0044] In the following, embodiments according to this disclosure will be described in detail with reference to the accompanying drawings so that those skilled in the art to which this disclosure pertains may readily perform them.

[0045] Figure 1 This is a block diagram illustrating the configuration of an electronic device according to an embodiment of the present disclosure; Figure 2 This is a block diagram illustrating multiple modules and their input and output data according to embodiments of the present disclosure; Figures 3 to 4B This is a diagram illustrating a portion of the depthwise convolution operation process according to an embodiment of the present disclosure. Key terms will first be described to describe the present disclosure, and references will be made below. Figures 1 to 4B Embodiments of this disclosure are described.

[0046] Electronic device 100 is a device that performs convolution operations. Specifically, electronic device 100 can obtain output data by performing convolution operations based on the input values ​​included in the input data and the weights of the kernel.

[0047] "Input data" can be three-dimensional data comprising input values ​​differentiated according to multiple channels. Specifically, input data can be a three-dimensional matrix comprising multiple input values ​​divided according to rows, columns, and depth, and can be divided into multiple channels corresponding to each depth. The term "input data" can be replaced by the term "input feature map," etc., and the term "input value" can be replaced by the term "input activation value."

[0048] A "kernel" can be a matrix that includes multiple weights used to perform multiplication with the input values. Specifically, depending on the type of convolution operation to be performed, the kernel can be constructed as a matrix, either one-dimensional, two-dimensional, or three-dimensional. The size of the kernel can be determined by its horizontal length (i.e., the number of columns), vertical length (i.e., the number of rows), and depth (i.e., the overall depth), and the multiple weights included in the kernel can be divided according to the multiple channels corresponding to the depth of the kernel. The term "kernel" can be replaced by terms such as filter or mask.

[0049] "Convolution operation" refers to the operation of multiplying the input values ​​included in the input data and the weights included in the kernel, and then summing the results of each multiplication. Specifically, convolution operations can include three-dimensional (3D) convolution operations and depthwise convolution operations. 3D convolution operation refers to the convolution operation that obtains 3D output data by using 3D input data and a 3D kernel, while depthwise convolution operation refers to the convolution operation that obtains 3D output data by using 3D input data and a one-dimensional kernel or a two-dimensional kernel. The detailed calculation process based on each type of convolution operation will be described below together with the description of embodiments according to this disclosure. Convolution operations can be performed by neural network models such as convolutional neural networks (CNNs). However, the types of neural network models to which this disclosure can be applied are not limited to these.

[0050] "Output data" can be a 3D matrix comprising multiple output values ​​partitioned by rows, columns, and depth, and can be divided into multiple channels corresponding to the respective depths. The rows, columns, and depth of the output data do not correspond to the rows, columns, and depth of the input data, and the rows, columns, and depth of the output data can vary depending on the kernel size, stride, padding, etc., used for the convolution operation. The term "output data" can be replaced by terms such as "output feature map," and the term "output value" can be replaced by the term "output activation value."

[0051] like Figure 1 As shown, the electronic device 100 according to this disclosure may include a memory 110 and a processor 120. However, as Figure 1 The configuration shown is merely illustrative, except that... Figure 1 In addition to the features shown, new configurations can be added or some configurations can be omitted.

[0052] At least one instruction for the electronic device 100 may be stored in the memory 110. Furthermore, an operating system (OS) for driving the electronic device 100 may be stored in the memory 110. According to various embodiments, the memory 110 may store various software programs or applications for operating the electronic device 100. The memory 110 may include semiconductor memory, such as flash memory, magnetic storage media (such as a hard disk), etc.

[0053] Specifically, the memory 110 can store various software modules for operating the electronic device 100, and the processor 120 can control the operation of the electronic device 100 by executing the various software modules stored in the memory 110. That is to say, the memory 110 can be accessed by the processor 120, and the processor 120 can perform data reading, recording, modification, deletion, updating, etc.

[0054] Memory 110 may include non-volatile memory 110A and volatile memory 110B. Non-volatile memory 110A retains stored information even during power interruption, while volatile memory 110B requires continuous power to retain stored information. For example, non-volatile memory 110A may be implemented using at least one of one-time programmable ROM (OTPROM), programmable ROM (PROM), erasable programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), mask ROM, or flash memory ROM, while volatile memory 110B may be implemented using at least one of dynamic RAM (DRAM), static RAM (SRAM), or synchronous dynamic RAM (SDRAM). In this disclosure, the term memory may be used to include memory 110, ROM, RAM in processor 120, or memory cards (e.g., micro-secure digital (SD) cards, memory sticks) mounted to electronic device 100.

[0055] In various embodiments according to this disclosure, memory 110 may store input data, output data, information about weights, etc. Although information about weights can be simply stored in... Figure 2 However, information about weights can be stored in memory 110 in the form of first core information and second core information.

[0056] "First kernel information" refers to kernel-related information including the weights of each of the multiple channels. For example, first kernel information could be kernel-related information including weights in a 3×3 (horizontal × vertical) matrix for each of the multiple channels. "Second kernel information" refers to kernel-related information that has been transformed such that the weights of each of the multiple channels in the first kernel information are arranged in the directions of the multiple channels. For example, second kernel information could be kernel-related information generated by transforming the weights configured in a 3×1 (horizontal × vertical) matrix for each of the multiple channels into a 1×1×3 (horizontal × vertical × depth) multi-channel form.

[0057] In other words, the first kernel information refers to the term for kernel information in a typical form used in convolution operations, that is, kernel information consisting of matrices for each of the multiple channels. The second kernel information refers to the term for kernel information generated by transforming the kernel information consisting of matrices for each of the multiple channels into a multi-channel kernel form, so as to perform depthwise convolution operations in parallel in the convolution operations according to this disclosure.

[0058] Various information required to achieve the purpose of this disclosure can be stored in memory 110, and the information stored in memory 110 can be received from external devices or updated by user input.

[0059] The processor 120 controls the overall operation of the electronic device 100. Specifically, the processor 120 is connected to the configuration of the electronic device 100 including the memory 110 as described above, and controls the overall operation of the electronic device 100 by executing at least one instruction stored in the memory 110 as described above.

[0060] The processor 120 can be implemented in various ways. For example, the processor 120 can be implemented as at least one of an application-specific integrated circuit (ASIC), an embedded processor, a microprocessor, hardware control logic, a hardware finite state machine (FSM), a digital signal processor (DSP), etc. In addition, the processor 120 may include at least one of a central processing unit (CPU), a graphics processing unit (GPU), a main processing unit (MPU), etc.

[0061] Processor 120 can load data required to perform various operations from non-volatile memory 110A to volatile memory 110B. Loading refers to the operation of loading data stored in non-volatile memory 110A and storing it in volatile memory 110B so that processor 120 can access it. Volatile memory 110B can be implemented as a component included in processor 120, as a component of processor 120, but this is merely an embodiment and can also be implemented as a component separate from processor 120.

[0062] Specifically, one or more processors 120 according to the present disclosure may be implemented. Processor 120 may include a neural network accelerator for effectively controlling the operation of a convolutional neural network model, and a central processing unit (CPU) for controlling the operation of various configurations including the neural network accelerator. Furthermore, the neural network accelerator may include multiple microprocessor units (MPUs), and the multiple MPUs may include multiple modules for implementing one or more embodiments according to the present disclosure.

[0063] like Figure 2 As shown, the multiple modules may include an input selection module 121, multiple multiplication modules 122, an intermediate value accumulation module 123, etc. The multiple modules according to this disclosure can be implemented as hardware modules included in the processor 120, or as software modules according to the embodiments.

[0064] "Input selection module 121" refers to a module used to send input values ​​to multiple multiplication modules 122 in different ways according to the type of convolution operation. When an input value included in the input data is received, input selection module 121 can send the input value to multiple multiplication modules 122 based on the type of convolution operation. Input selection module 121 can be implemented as a software module or a hardware module included in processor 120. In the following description, for convenience, the case where input selection module 121 is implemented as a hardware module included in processor 120 will be described.

[0065] "Multiple multiplication module 122" refers to a module used to perform multiplication operations between input values ​​and weights. Specifically, when an input value is received from the input selection module 121 and the processor 120 receives the weights, the multiple multiplication module 122 can multiply the received input value and the weights to send an intermediate value, which is the result of the multiplication, to the intermediate value accumulation module 123.

[0066] "Intermediate value accumulation module 123" refers to a module that obtains the output value by summing intermediate values ​​obtained during the convolution operation. Specifically, when multiple intermediate values ​​are received from multiple multiplication modules 122, the intermediate value accumulation module 123 can obtain and output the output value based on the multiple intermediate values. The intermediate value accumulation module 123 may include a first intermediate value accumulation module 123-1 used in the depthwise convolution process (e.g., as shown in the image). Figure 9 (as shown) and the second intermediate value accumulation module 123-2 used in the 3D convolution process (e.g., as shown) Figure 9 (As shown).

[0067] Here, the depth direction process using the first intermediate value accumulation module 123-1 will be described first, and reference will be made to... Figures 9 to 11 This describes a 3D convolution process using a second intermediate value accumulation module 123-2. The intermediate value accumulation module 123 may include at least one register and at least one buffer. (See above reference...) Figures 5 to 8 An example related to the buffer is described.

[0068] The processor 120 can identify the type of convolution operation to be performed and control multiple modules in different ways depending on whether the convolution operation to be performed is a 3D convolution operation or a depthwise convolution operation. In one embodiment, the processor 120 can use first core information when performing a 3D convolution operation and use second core information when performing a depthwise convolution operation. Referring below... Figures 3 to 4B Describe an example related to depthwise convolution operations.

[0069] Specifically, Figure 3This is a diagram illustrating the process of obtaining the output value based on the input values ​​and weights at time points T0 to T4 when performing a one-dimensional convolution operation within a depthwise convolution operation. Figure 4A and Figure 4B This indicates when executing as Figure 3 The diagram illustrates the operation of circuits comprising multiple modules according to this disclosure. Specifically, in Figures 3 to 4B The description provides an example of a case where multiple weights of three correspond to the first channel.

[0070] The processor 120 can input input values ​​with the same row and column but different depths from the input values ​​included in the input data to multiple MPUs in each preset cycle. However, in the case of depthwise convolution operations, unlike the 3D convolution operations described below, the process of adding the intermediate values ​​of the input values ​​from different channels to obtain an output value is not necessary. Therefore, in describing an embodiment of depthwise convolution operations, the operation process corresponding to the input value of the first channel among the multiple channels will be mainly described.

[0071] When performing a depthwise convolution operation, the processor 120 can control the input selection module 121 such that multiple input values ​​corresponding to a first channel among a plurality of channels are input to all two or more of the plurality of multiplication modules 122. Specifically, the number of two or more multiplication modules 122 may correspond to the number of weights included in the kernel. In the following description of this disclosure, the term "two or more multiplication modules 122" is used to specify the multiplication module 122 used in the depthwise convolution operation according to this disclosure among the plurality of multiplication modules 122.

[0072] In other words, the processor 120 can control the input selection module 121 such that whenever each of the plurality of inputs corresponding to the first channel is input to the input selection module 121, the input value is input to all the multiplication modules 122 as many as the number of weights included in the kernel. Conversely, in related art, in depthwise convolution operations, the input value corresponding to the first channel is input to the multiplication module 122 corresponding to the first channel.

[0073] For example, as Figure 3 and Figure 4A During the operation process of time T2, if F2, one of the multiple input values ​​corresponding to the first channel, is input to the input selection module 121, the processor 120 can control the input selection module 121 so that F2 is not only input to the multiplication module 122 corresponding to the first channel, but also to the multiplication module 122 corresponding to each of the second and third channels adjacent to the first channel.

[0074] The processor 120 can input multiple weights corresponding to the first channel one by one into two or more multiplication modules 122 based on the second core information.

[0075] In other words, according to the present disclosure, the processor 120 can input multiple weights corresponding to the first channel one by one into two or more multiplication modules 122 based on the second core information of the core, which has been converted into a multi-channel form. Conversely, in related technologies, in depthwise convolution operations, multiple weights corresponding to the first channel are only input into the multiplication module 122 corresponding to the first channel.

[0076] For example, such as Figure 3 and Figure 4A During the operation at time T2, the processor 120 can input each of W0, W1, and W2 (corresponding to multiple weights of the first channel) one by one into each of the three multiplication modules 122 corresponding to each of the first, second, and third channels. The processor 120 can input weight W0 into the multiplication module 122 corresponding to the first channel, weight W1 into the multiplication module 122 corresponding to the second channel, and weight W2 into the multiplication module 122 corresponding to the third channel.

[0077] Multiple weights (W0, W1, and W2) can be constructed in a 1*1*3 (horizontal*vertical*depth) multi-channel form and stored in memory 110 as second core information.

[0078] As described above, when multiple input values ​​and multiple weights are input to multiple multiplication modules 122, the processor 120 can perform a multiplication operation with each of the multiple weights on each of the multiple input values ​​through the multiple multiplication modules 122 to obtain multiple intermediate values ​​based on the result of each multiplication operation.

[0079] For example, as in Figure 3 and Figure 4A During the operation at time T2, all three multiplication modules 122 corresponding to the first, second, and third channels have an input value F2. When weights W0, W1, and W2 are input to each of the three multiplication modules 122, the multiplication module 122 corresponding to the first channel performs a multiplication operation between F2 and W0 to obtain an intermediate value F2*W0, the multiplication module 122 corresponding to the second channel performs a multiplication operation between F2 and W1 to obtain an intermediate value F2*W1, and the multiplication module 122 corresponding to the third channel performs a multiplication operation between F2 and W2 to obtain an intermediate value F2*W2.

[0080] Reference Figure 4B Combination Figure 3Similar to the calculation process at time T2, the three multiplication modules 122 corresponding to each of the first, second, and third channels can obtain multiple intermediate values ​​based on the calculation results of each of F3*W0, F3*W1, and F3*W2 at time T3 after T2, and can also obtain multiple intermediate values ​​based on the corresponding calculation results of F4*W0, F4*W1, and F4*W2 at time T4 after T3.

[0081] As described above, when multiple intermediate values ​​are obtained, the processor 120 can sum the intermediate values ​​corresponding to each position of the core among the multiple intermediate values ​​through the intermediate value accumulation module 123 to obtain multiple output values ​​based on each summation result. Specifically, the processor 120 can obtain multiple output values ​​based on the summation of intermediate values ​​by using the first intermediate value accumulation module 123-1 in the intermediate value accumulation module 123. "Intermediate values ​​corresponding to the positions of the core respectively" refers to intermediate values ​​obtained based on the multiplication result, which is obtained by multiplying multiple weights included in the core with multiple input values ​​corresponding to the multiple weights in the core, while moving the core sequentially according to a predetermined interval (i.e., stride) on the matrix of input data.

[0082] For example, when as described above in Figure 3 and Figure 4A After the calculation process is executed at time T2, Figure 3 and Figure 4B When the operation process of T3 and T4 is executed sequentially, when the position of the core corresponds to the input values ​​F2, F3 and F4 through the first intermediate value accumulation module 123-1, the processor 120 can obtain the output value O2 by summing the intermediate values ​​F2*W0, F3*W1 and F4*W2.

[0083] The output value O2 is obtained by summing the intermediate value F2*W0 obtained at time T2, the intermediate value F3*W1 obtained at time T3, and the intermediate value F4*W2 obtained at time T4. However, in the case of output values ​​O0 and O1, this value can be obtained by a similar method. If the intermediate value F0*W0 is obtained at time T0 by multiple multiplication modules 122, and if the intermediate values ​​F1*W0 and F1*W1 are obtained at time T1, the output value O0 can be obtained by summing the intermediate values ​​F0*W0, F1*W1, and F2*W2 at time T2, and the output value O1 can be obtained by summing the intermediate values ​​F1*W0, F2*W1, and F3*W2 at time T3.

[0084] Intermediate values ​​obtained by multiple multiplication modules 122 can be temporarily stored in registers included in intermediate value accumulation module 123, and the intermediate values ​​stored in the registers can be used to obtain the final output value. Figure 4AEach of “A” and “B” indicates a register used to store intermediate values ​​obtained by the multiplication module 122 corresponding to the first channel and the multiplication module 122 corresponding to the second channel. Figure 4A The “C” indicates the output value, which is the result of adding the intermediate value stored in register A and the intermediate value stored in register B to the intermediate value obtained by the multiplication module 122 corresponding to the third channel.

[0085] In the configuration included in the first intermediate value accumulation module 123-1, the registers included in the accumulation module may be referred to as the so-called partial summation register (PSR), and in the configuration included in the first intermediate value accumulation module 123-1, the configuration shown as the "+" sign may be referred to as the so-called partial summation adder (PSA), as a summer included in the intermediate value module.

[0086] Processor 120 can obtain intermediate values ​​F0*W1, F0*W2, and F0*W0 at time T0. Processor 120 can also obtain intermediate values ​​F1*W2, F1*W0, and F1*W1. However, intermediate values ​​F0*W1, F0*W2, and F1*W2 do not correspond to intermediate values ​​corresponding to core locations and do not need to be obtained.

[0087] As described above, for ease of description, the above description has been based on the calculation process corresponding to the first channel among multiple channels. However, the operation process according to the embodiment described above can also be applied to other channels besides the first channel among the multiple channels. Therefore, the electronic device 100 can obtain output data including the output value of the entire input data.

[0088] According to an embodiment, first kernel information for 3D convolution operations and second kernel information for depthwise convolution operations are pre-built and stored in memory 110. With only the first kernel information stored in memory 110, processor 120 can transform the weights of each channel in the multiple channels of the first kernel information in the multiple channel directions, input it to multiple multiplication modules 122, and perform depthwise convolution operations.

[0089] Based on the above references Figures 1 to 4B In the described embodiments, the electronic device 100 can efficiently perform convolution operations by using, for example, parallel hardware structures in a neural network accelerator.

[0090] Specifically, when performing depthwise convolution operations, the electronic device 100 can obtain the output value for each cycle corresponding to the third cycle from time T2 by simultaneously multiplying an input value with each of a plurality of weight values ​​through a plurality of multiplication modules 122, using kernels arranged in the horizontal direction in the channel direction of the 3D convolution operation. Therefore, regardless of when each output value is obtained, an improvement in computational efficiency (e.g., acceleration of convolution operation computation) can be achieved, which is three times higher than related techniques requiring a three-cycle computation process.

[0091] like Figures 3-4B The example shown pertains to a one-dimensional convolution operation during a depthwise convolution operation. This disclosure can also be applied when it performs a two-dimensional convolution operation during a depthwise convolution operation. (See also...) Figures 5 to 8 A detailed description of an embodiment of performing a two-dimensional convolution operation.

[0092] The electronic device 100 according to this disclosure can perform depthwise convolution operations and 3D convolution operations, and will refer to Figures 9 to 11 Detailed description in such Figure 4A and Figure 4B The detailed process of performing 3D convolution operations under the architecture of this disclosure is shown.

[0093] Figure 5 This is a block diagram illustrating multiple modules that perform two-dimensional convolution operations according to this disclosure; Figures 6 to 8 This is a diagram illustrating a portion of a two-dimensional convolution operation according to various embodiments of the present disclosure.

[0094] like Figure 5 As shown, the multiple modules according to this disclosure may include an input selection module 121, multiple multiplication modules 122, and an intermediate value accumulation module 123. Specifically, the intermediate value accumulation module 123 may include a buffer 123-3. "Buffer 123-3" refers to the configuration of intermediate values ​​corresponding to the rows of the two-dimensional kernel among the intermediate values ​​obtained during the execution of the two-dimensional convolution operation.

[0095] Specifically, when referring to Figures 1 to 4B When performing a one-dimensional convolution operation, intermediate values ​​can be stored in a register included in the intermediate value accumulation module 123. When performing a two-dimensional convolution operation, a buffer 123-3 that stores the intermediate values ​​corresponding to the rows of the kernel may be required, separate from the registers.

[0096] The processor 120 according to an embodiment of this disclosure may further include a buffer 123-3, which may be used to store some intermediate values ​​obtained for each row of the core. Although Figure 5The buffer 123-3 is shown as being included in the intermediate value accumulation module 123, but the buffer 123-3 can be implemented as a component separate from the intermediate value accumulation module 123.

[0097] Figure 6 This diagram illustrates a method for obtaining the result of a two-dimensional convolution operation by sequentially processing each row's one-dimensional convolution operation. Figure 7 This shows that when executing such... Figure 6 The diagram shown illustrates the operation of a circuit comprising multiple modules according to this disclosure. Figure 8 This is a diagram illustrating a method for obtaining the result of a two-dimensional convolution operation by using multiple multiplication modules 122 corresponding to the number of weights included in the two-dimensional kernel.

[0098] exist Figures 6 to 8 In the example, the kernel has a size of 3x3, and the weights corresponding to the first channel are 9. Because... Figures 6 to 8 The depthwise convolution operation is shown, so the operation processing corresponding to the input value of the first channel among multiple channels will be mainly described.

[0099] Reference Figure 6 For ease of description, a group of input values ​​corresponding to each row of multiple input values ​​is denoted as F0, F1, F2, and F3, and a group of weights corresponding to each row of multiple weights is denoted as K0, K1, and K2.

[0100] like Figure 6 As shown, the processor 120 can obtain a first intermediate value by using multiple multiplication modules 122 to calculate the input value F0 of the first row and the weight value K0 of the first row, and store the obtained first intermediate value in buffer 123-3. Alternatively or additionally, the processor 120 can obtain a second intermediate value by using multiple multiplication modules 122 to calculate the input value F1 of the second row and the weight K1 of the second row, add the second intermediate value to the first intermediate value stored in buffer 123-3 by the first intermediate value accumulation module 123-1, and store the sum in buffer 123-3.

[0101] Furthermore, the processor 120 can obtain a third intermediate value based on the calculation results of the third row input value F2 and the third row weight K2 through multiple multiplication modules 122, and can obtain an output value O0 by adding the third intermediate value to the sum of the first and second intermediate values ​​stored in buffers 123-3. Here, "an output value" refers to one of the output values ​​corresponding to each case where the two-dimensional kernel is located on the input data matrix.

[0102] The processor 120 can obtain another output value O1 by adding the intermediate value calculated based on the input value F1 of the second row and the weight K0 of the first row, the intermediate value calculated based on the input value F2 of the third row and the weight K1 of the second row, and the intermediate value calculated based on the input value F3 of the fourth row and the weight K2 of the third row through the first intermediate value accumulation module 123-1.

[0103] Figure 7 It is shown in Figure 6 The diagram illustrates the process of obtaining the output value (O0) during the operation, and shows the sequential operation of each line in the first (line 0), second (line 1), and third (line 2). The operation performed on each line is as follows: Figure 4A and Figure 4B The same applies as shown, but in this example, the calculations should be accumulated to the operation with the weight of the third row. Therefore, buffers 123-3 are used to store intermediate values ​​obtained by summing the intermediate values ​​obtained from the calculations of each row with the calculation results of another row.

[0104] exist Figure 7 In this context, a set of input values ​​corresponding to each row of multiple input values ​​is represented as F0, F1, and F2, and a set of weights corresponding to each row of multiple weights is represented as W. 0,0 W 0,1 W 0,2 W 1,0 W 1,1 W 1,2 W 2,0 W 2,1 and W 2,2 In W m,n In the symbol, m is the sequence number indicating the row number to which kernel m belongs, and n is the sequence number indicating the column number to which kernel n belongs. Weight W 0,0 W 0,1 and W 0,2 Weight W 1,0 W 1,1 and W 1,2 and weight W 2,0 W 2,1 and W 2,2 It can be built in a multi-channel configuration and stored in memory 110 as a second core.

[0105] Specifically, when the input value F0 of the first row among multiple input values ​​corresponding to the first channel is input to the input selection module 121, the processor 120 can control the input selection module 121 so that F0 is input to the multiplication module 122 corresponding to the first channel, and to the multiplication modules 122 corresponding to each of the second and third channels adjacent to the first channel. The processor 120 can then input multiple weights W corresponding to the first row... 0,0 W 0,1 and W 0,2 The input is sequentially fed into each of the three multiplication modules 122 corresponding to each of the first, second, and third channels.

[0106] When the input value F0 is input to all three multiplication modules 122 corresponding to each of the first, second, and third channels, and the weight W 0,0 W 0,1 and W 0,2 When input one by one into each of the three multiplication modules 122, the processor 120 can execute F0 and W through the multiplication module 122 corresponding to the first channel. 0,0 Multiplication between them, to obtain what is called F0*W 0,0 The intermediate value, and can be obtained through the multiplication module 122 corresponding to the second channel at F0 and W. 0,1 Perform multiplication between them to obtain what is called F0*W 0,1 The intermediate value, and can be obtained through the multiplication module 122 corresponding to the third channel at F0 and W. 0,2 Perform multiplication between them to obtain F0*W 0,2 The median value.

[0107] The processor 120 can obtain the output value corresponding to the input value F0 of the first row by summing the intermediate values ​​through the first intermediate value accumulation module 123-1, and store the intermediate value of the output value O0 corresponding to the input value F0 of the first row in the buffer 123-3.

[0108] When the output value corresponding to the input value F1 of the second row is obtained, the processor 120 can add the output value corresponding to the input value F0 of the first row to the intermediate value stored in the buffer 123-3 in the same way as the process of obtaining the output value corresponding to the input value F0 of the first row, and store the sum in the buffer 123-3.

[0109] Furthermore, when the output value corresponding to the input value F2 of the third row is obtained in the same manner as the process of obtaining the output value corresponding to the input value F0 of the first row and the output value corresponding to the input value F1 of the second row, the processor 120 can obtain an output value O0 by summing the output value corresponding to the input value F2 of the third row with the summation value stored in the buffer 123-3 through the first intermediate value accumulation module 123-1.

[0110] Based on the above references Figure 6 and Figure 7 In the described embodiment, the electronic device 100 can perform one-dimensional convolution operations by using multiple multiplication modules 122 in parallel, and can obtain a result based on a two-dimensional convolution operation by summing the results of each one-dimensional convolution operation. Therefore, the electronic device 100 can perform two-dimensional convolution operations. That is, as described above, when the number of weights included in the two-dimensional kernel is nine, and the output value is obtained by accumulating the one-dimensional convolution operation of three weights, the electronic device 100 can achieve a three-fold improvement in computational efficiency.

[0111] Electronic device 100 can perform two-dimensional convolution operations by simply adding a buffer 123-3 for cumulatively storing each result of a one-dimensional convolution operation, as described below. Figure 8 Compared to the previous embodiment, the electronic device 100 can perform two-dimensional convolution operations by using a hardware region.

[0112] Figure 8 This is a diagram illustrating a method for performing two-dimensional convolution operations by using multiple multiplication modules 122 in parallel, corresponding to the number of weights included in the two-dimensional kernel. Figure 7 In the first row of weights, the operations are performed sequentially from the first row to the third row. Conversely, in the second row... Figure 8 In this approach, intermediate values ​​of different output values ​​O0, O1, and O2 can be obtained simultaneously by performing operations from the first to the third row of weights relative to the inputs in the same row. Figure 7 Similarly, in Figure 8 In the diagram, a set of weights corresponding to each row among multiple weights is determined by W. 0,0 W 0,1 W 0,2 W 1,0 W 1,1 W 1,2 W 2,0 W 2,1 and W 2, 2 represents the weight W. 0,0 W 0,1 W 0,2 W 1,0 W 1,1 W 1,2 W 2,0 W2,1 and W 2,2 It is constructed in a multi-channel form of 1*1*9 (horizontal*vertical*depth) and stored in memory 110 as the second core information.

[0113] like Figure 6 As shown in (1), when multiple first row input values ​​F0 are input, processor 120 can perform a multiplication operation with the weight K0 of the first row, and the intermediate value of the operation result can be stored in buffer 123-3 to accumulate the input values ​​of another row and the operation with the weights. This operation can be performed by... Figure 8 The multiplication module shown in (1) is used to perform the multiplication, and the intermediate value of the result can be stored in buffer 123-3.

[0114] Next, as Figure 6 As shown in (2), when multiple input values ​​F1 in the second row are input, the processor 120 can process the input values ​​F1 in the second row through the input values ​​F1 in the second row. Figure 8 The intermediate value of the output value O0 is obtained by summing the output value obtained by performing the operation on the weight K1 of the second row in the arithmetic unit shown in (2) with the pre-stored buffer value (F0*K0). Simultaneously, the processor 120 can obtain the intermediate value of the output value O1 by calculating the same input value F1 using the weight K0 of the first row, such as... Figure 6 As shown in (4). That is to say, Figure 8 The operational circuits of (1) and (2) can operate simultaneously. The buffer 123-3 that stores the output value corresponding to the weight K1 of the second row can be different from the buffer 123-3 that stores the output value corresponding to the weight K0 of the first row.

[0115] When multiple input values ​​F2 are entered in the third row, the processor 120 can utilize, for example, Figure 6 The output value O0 is obtained by adding the result of the operation performed on the weight K2 in the third row shown in (3) and the pre-stored buffer value. This operation unit corresponds to Figure 8 (3). At the same time, Figure 8 (1) and (2) can store the results of intermediate values ​​obtained by performing operations on the second row weight K1 and the first row weight K0 to calculate different output values ​​O1 and O2 in the computation buffer 123-3.

[0116] The processor 120 obtains the intermediate value of the row corresponding to the core and stores the intermediate value in the buffer 123-3, and accumulates the intermediate value obtained for each row in the column direction to perform a two-dimensional convolution operation. However, according to another embodiment, the processor 120 can perform a two-dimensional convolution operation by obtaining the intermediate value of the column corresponding to the core and storing the intermediate value in the buffer 123-3, and accumulating the intermediate value obtained for each column in the row direction.

[0117] Based on the above references Figure 8 In the described embodiment, the electronic device 100 can efficiently perform a two-dimensional convolution operation by using as many multiplication modules 122 in parallel as the number of weights included in the two-dimensional kernel. Specifically, when the number of weights included in the two-dimensional kernel is nine as in the example above, the electronic device 100 can achieve a nine-fold improvement in computational efficiency.

[0118] Figure 9 This is a block diagram illustrating a plurality of modules performing 3D convolution operations according to embodiments of the present disclosure; Figure 10 and Figure 11 This is a diagram illustrating a portion of a 3D convolution operation according to various embodiments of the present disclosure.

[0119] Various embodiments of this disclosure have been made based on the execution of depthwise convolution operations, but as described above, the processor 120 can perform 3D convolution operations by using multiple modules according to this disclosure.

[0120] like Figure 9 As shown, the multiple modules according to this disclosure may include an input selection module 121, multiple multiplication modules 122, and an intermediate value accumulation module 123. Specifically, the intermediate value accumulation module 123 may include a first intermediate value accumulation module 123-1 and a second intermediate value accumulation module 123-2. Here, "first intermediate value accumulation module 123-1" refers to a module used for accumulating intermediate values ​​by adding the input value to the input value as shown in the above reference. Figures 1 to 8 The module that obtains the output value by summing intermediate values ​​obtained during the depthwise convolution operation. "Second intermediate value accumulation module 123-2" refers to a module used to obtain the output value by summing intermediate values ​​obtained during the 3D convolution operation. According to this disclosure, the second intermediate value accumulation module 123-2 can be referred to as a so-called "adder tree." That is, it is indicated as... Figure 10 The configuration of the adder tree is represented by the second intermediate value accumulation module 123-2 according to this disclosure.

[0121] In 3D convolution operations, operations are performed between input values ​​included in the three-dimensional input data and weights included in the three-dimensional kernel. Multiple multiplication modules 122 corresponding to each of the multiple channels are used. Specifically, in 3D convolution operations, a convolution operation is performed between a set of input values ​​with the same rows and columns and different depths among the input values ​​included in the input data and a set of weights with the same rows and columns and different depths among the weights included in the kernel. Therefore, when inputting weights to the multiple multiplication modules 122 in 3D convolution operations, first kernel information can be used instead of second kernel information converted into a multi-channel form.

[0122] Specifically, Figure 10This is a diagram illustrating the process of obtaining the output value based on the input value and weights during time T0 to T5 when performing a 3D convolution operation. Figure 11 This indicates when executing as Figure 10 The diagram shown illustrates the operation of a circuit comprising multiple modules according to this disclosure.

[0123] Specifically, in Figure 10 and Figure 11 The description provides an example of a 3D kernel with a size of 3*1*64 (horizontal*vertical*depth). Figure 10 In this context, a group of input values ​​with the same row and column but different depths is represented as F0, F1, F2, and F3, respectively. Figure 10 In the kernel, a set of weights with the same rows and columns but different depths are shown as W0, W1, and W2.

[0124] At time T0, processor 120 can control input selection module 121 such that the input values ​​included in F0 are input to multiplication module 122 corresponding to each input value channel. Processor 120 can also control input selection module 121 such that input values ​​from multiple channels are input to the multiplication module 122 corresponding to the corresponding channel in the same manner as the input values ​​from F0 corresponding to the first channel are input to the multiplication module 122 corresponding to the first channel. Input values ​​corresponding to the second channel are input to the multiplication module 122 corresponding to the second channel.

[0125] The processor 120 can input the weights included in W0 to the multiplication module 122 corresponding to each weight. For example, the processor 120 can input the weights of multiple channels to the multiplication module 122 corresponding to the corresponding channel in the same way that the weights included in W0 corresponding to the first channel are input to the multiplication module 122 corresponding to the first channel. The weights corresponding to the second channel are input to the multiplication module 122 corresponding to the corresponding channel.

[0126] When the input value included in F0 and the weight included in W0 are input to multiple multiplication modules 122, the processor 120 can obtain a first intermediate value by multiplying the input value (F0(0)) of the first channel included in F0 and the weight value (W0(0)) of the first channel included in W0 by multiplication module 122 corresponding to the first channel, obtain a second intermediate value by multiplying the input value (F0(1)) of the second channel included in F0 and the weight value (W0(1)) of the second channel included in W0 by multiplication module 122 corresponding to the second channel, and obtain intermediate values ​​corresponding to each of the third to 64 channels in a similar manner.

[0127] When an intermediate value corresponding to each of the first to the 64th channels is obtained, the processor 120 can sum the intermediate values ​​corresponding to each of the first to the 64th channels through the second intermediate value module to obtain a first summation value (O0(1)) based on the summation result.

[0128] At time T1, processor 120 can obtain the second summation value O0(2), and add the first summation value O0(1) and the second summation value O0(2) according to the summation result to obtain the summation value O0(1~2).

[0129] Furthermore, at time T2, the processor 120 can obtain a third summation value O0(3), summing the first summation value O0(1), the second summation value O0(2), and the third summation value O0(3), and obtain an output value O0 based on the summation result in the same manner as obtaining the first summation value O0(1) and the second summation value O0(2). Here, an output value refers to one of the output values ​​corresponding to each case where the three-dimensional kernel is located on the input data matrix.

[0130] Based on the above reference Figures 9-11 According to the described embodiments, the electronic device 100 can not only efficiently perform depthwise convolution operations under a single hardware structure, but also perform 3D convolution operations.

[0131] Figure 12 This is a flowchart illustrating a method for controlling an electronic device according to an embodiment of the present disclosure. As described above, the electronic device 100 can perform convolution operations based on input values ​​included in the input data and weights included in the kernel. Specifically, according to the present disclosure, the electronic device 100 can perform depthwise convolution operations and 3D convolution operations by using an input selection module 121, a plurality of multiplication modules 122, and an intermediate value accumulation module 123.

[0132] Reference Figure 12 In operation S1210, the electronic device 100 can identify whether the convolution operation to be performed is a depthwise convolution operation. When the convolution operation to be performed in operation S1210-Y is a depthwise convolution operation, in operation S1220, the electronic device 100 can control the input selection module 121 so that the multiple input values ​​corresponding to the first channel among the multiple channels used to distinguish the input data are input to all two or more of the multiple multiplication modules 122.

[0133] In other words, the electronic device 100 according to this disclosure can control the input selection module 121 such that whenever each of the plurality of inputs corresponding to the first channel is input to the input selection module 121, the input value is input to all the multiplication modules 122 as many as the number of the plurality of weights included in the kernel, which is different from the depthwise convolution operation in the related art where the input value corresponding to the first channel is input to the multiplication module 122 corresponding to the first channel.

[0134] In operation S1230, the electronic device 100 can input multiple weights corresponding to the first channel one by one into two or more multiplication modules 122. In other words, according to the present disclosure, the electronic device 100 can input multiple weights corresponding to the first channel one by one into two or more multiplication modules 122 based on second kernel information in the form of multiple channels, which is different from the depth convolution operation in the related art where multiple weights corresponding to the first channel are only input into the multiplication module 122 corresponding to the first channel.

[0135] In operation S1240, the electronic device 100 can perform a multiplication operation with each of the multiple input values ​​and each of the multiple weights on each of the multiple input values ​​through two or more multiplication modules 122, so as to obtain multiple intermediate values ​​based on the result of each multiplication operation. Furthermore, in operation S1250, the electronic device 100 can sum the intermediate values ​​corresponding to each position of the kernel among the multiple intermediate values ​​through the first intermediate value accumulation module 123-1, so as to obtain multiple output values ​​based on each summation result.

[0136] The operation of the control method of the electronic device 100 according to the present disclosure has been briefly described above, but this is only to omit redundant descriptions of the same content, and various embodiments related to the control process of the processor 120 can also be applied to the control method of the electronic device 100.

[0137] The control method of the electronic device 100 according to the above embodiments can be implemented as a program and provided to the electronic device 100. Specifically, the program including the control method of the electronic device 100 can be stored in and provided therein on a non-transitory computer-readable medium.

[0138] Specifically, in a non-transitory computer-readable recording medium including a program for executing a control method for electronic device 100, when the convolution operation is a depthwise convolution operation, the method for controlling electronic device 100 may include controlling an input selection module 121 such that each of a plurality of input values ​​corresponding to a first channel among a plurality of channels used to distinguish input data is input to all plurality of multiplication modules 122 corresponding to each of two or more channels among the plurality of channels; each of a plurality of weights corresponding to the first channel is input one by one to the plurality of multiplication modules 122; each of the plurality of input values ​​is multiplied by the plurality of multiplication modules 122 with each of the plurality of weights, and a plurality of intermediate values ​​are obtained based on the result of each multiplication operation; and the intermediate values ​​corresponding to each of the plurality of weights are summed by a first intermediate value accumulation module 123-1, and a plurality of output values ​​are obtained based on the summation result.

[0139] Non-transitory computer-readable media can include media that store data semi-permanently rather than for a very short period of time, such as registers, caches, memories, etc., and can be read by a device (i.e., executed by at least one processor). For example, the various applications or programs described above can be stored in non-transitory computer-readable media, such as optical discs (CDs), digital versatile optical discs (DVDs), hard disks, Blu-ray discs, universal serial buses (USB), memory cards, read-only memory (ROMs), etc., and can be provided.

[0140] A brief description of the control method of electronic device 100 and a non-transitory computer-readable recording medium including a program for executing the control method of electronic device 100 is provided, but this is only to avoid repetition of the description, and various embodiments of electronic device 100 can be applied to the control method of electronic device 100 and the computer-readable recording medium including a program for executing the control method of electronic device 100.

[0141] According to one or more embodiments of the present disclosure as described above, an electronic device can efficiently perform depthwise convolution operations. Functions related to the neural network model and the convolution operation process can be performed via memory 110 and processor 120.

[0142] Processor 120 may include one or more processors 142. At this time, the one or more processors 120 may be general-purpose processors, such as central processing unit (CPU), application processor (AP), graphics-specific processing unit, such as graphics processing unit (GPU), vision processing unit (VPU), or AI-specific processors, such as neural processing unit (NPU) or MPU.

[0143] One or more processors 120 control the processing of input data according to predefined operating rules or artificial intelligence (AI) models stored in non-volatile memory 110A and volatile memory 110B. The predefined operating rules or AI models are provided through training or learning.

[0144] Learning can provide, for example, by applying a learning algorithm to multiple learning datasets, predefined operating rules or AI models with desired characteristics. This learning can be performed within the device itself that performs the AI ​​according to the embodiment, and / or can be implemented via a separate server / system.

[0145] AI models can include multiple neural network layers. Each layer has multiple weight values, and layer operations are performed by computing the previous layer and operating on the multiple weights. Examples of neural networks include, but are not limited to, CNNs, deep neural networks (DNNs), recurrent neural networks (RNNs), restricted Boltzmann machines (RBMs), deep belief networks (DBNs), bidirectional recurrent deep neural networks (BRDNNs), generative adversarial networks (GANs), and deep Q-networks.

[0146] Learning algorithms can include methods for training a predetermined target device (e.g., a robot) using multiple training datasets, so as to enable, allow, or control the target device to make a determination or prediction. Examples of learning algorithms include, but are not limited to, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning.

[0147] Machine-readable storage media may be provided in the form of non-transitory storage media. "Non-transitory" storage media may not include signals and may be tangible, but does not distinguish whether data is stored permanently or temporarily in the storage medium. For example, "non-transitory storage media" may include buffers 123-3 that temporarily store data.

[0148] According to various embodiments, the methods disclosed herein can be provided in a computer program product. The computer program product can be traded as a commodity between a seller and a buyer. The computer program product can be distributed in the form of a machine-readable storage medium (e.g., an optical disc ROM (CD-ROM)) or through an app store (e.g., the Play Store). TM Online distribution, or direct online distribution (e.g., download or upload) between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product (e.g., a downloadable app) may be temporarily or at least temporarily stored in a storage medium, such as storage in a manufacturer's server, an app store server, or a relay server.

[0149] Alternatively or additionally, each component (e.g., module or program) according to one or more embodiments may include a single entity or multiple entities, and some of the sub-components described above may be omitted, or other sub-components may be further included in various embodiments. Alternatively or additionally, some components (e.g., module or program) may be integrated into one entity to perform the same or similar functions performed by the respective components prior to integration.

[0150] According to various embodiments, operations performed by modules, programs or other components may be performed in a sequential, parallel, iterative or heuristic manner, or at least some operations may be performed in a different order or omitted, or other operations may be added.

[0151] As used in this disclosure, the terms "unit" or "module" include a unit comprising hardware, software, or firmware, or any combination thereof, and may be used interchangeably with terms such as logic, logic block, section, or circuit. A "unit" or "module" can be a component or the smallest unit or part of an overall structure that performs one or more functions. For example, the module may be configured as an application-specific integrated circuit (ASIC).

[0152] The embodiments can be implemented as software, which includes instructions stored in a machine-readable storage medium that is machine-readable (e.g., a computer). A device including an electronic device (e.g., electronic device 100) can invoke the instructions from the storage medium and become operable according to the invoked instructions.

[0153] When an instruction is executed by the processor, the processor may use other components to perform the function corresponding to that instruction, either directly or under the processor's control. Instructions may include code generated by a compiler or code executed by an interpreter.

[0154] While this disclosure has been described and illustrated with reference to various exemplary embodiments, it will be understood that these exemplary embodiments are intended to be illustrative and not limiting. Those skilled in the art will understand that various changes in form and detail may be made without departing from the true spirit and full scope of this disclosure, including the appended claims and their equivalents.

Claims

1. An electronic device, comprising: The memory is configured to store three-dimensional input data, the three-dimensional input data including (i) multiple input values ​​based on multiple channels, (ii) a first kernel information about the kernel including multiple weights for each of the multiple channels, and (iii) a second kernel information generated by converting multiple weights configured in the form of a two-dimensional matrix for each of the multiple channels into a three-dimensional matrix. as well as The processor includes multiple multiplication modules, each corresponding to a multiple channel, and is configured to perform convolution operations based on multiple input values ​​and multiple weights through the multiple multiplication modules. The processor is further configured as follows: Based on the fact that the convolution operation is a depthwise convolution operation, the input selection module (a) configures multiple input values ​​to correspond to the first channel among multiple channels, and (b) inputs the multiple input values ​​into two or more multiplication modules among multiple multiplication modules. Based on the information from the second core, the first set of weights corresponding to the first channel are input one by one into two or more multiplication modules. By using two or more multiplication modules to perform multiplication operations on each of a plurality of input values ​​with each of a plurality of weights, multiple intermediate values ​​are obtained based on the result of each multiplication operation. The first intermediate value accumulation module sums the intermediate values ​​corresponding to the core positions among multiple intermediate values, and obtains multiple output values ​​based on the summation result.

2. The electronic device according to claim 1, wherein, Each of the multiple input values ​​corresponding to the first channel is the input to the input selection module in each preset cycle, and The input selection module is configured to send each of a plurality of input values ​​to two or more multiplication modules in each preset period.

3. The electronic device according to claim 1, wherein, The two or more channels include a first channel and at least one channel adjacent to the first channel, and The number of the two or more multiplication modules corresponds to the number of weights included in the first core information.

4. The electronic device according to claim 1, wherein, The core is a two-dimensional core, and The processor further includes a buffer that stores an intermediate value corresponding to a row of the core among a plurality of intermediate values, and The processor obtains multiple output values ​​by summing the intermediate values ​​stored in the buffer that correspond to each position of the core through the first intermediate value accumulation module.

5. The electronic device according to claim 4, wherein, The processor is also configured to obtain multiple output values ​​by performing convolution operations in parallel using two or more multiplication modules corresponding to the number of multiple weights included in the kernel.

6. The electronic device according to claim 2, wherein, The processor is also configured to: Since the convolution operation is a three-dimensional convolution operation, the input selection module is controlled to bypass the input values ​​input to the input selection module to multiple multiplication modules, and Based on the first core information, the second set of weights corresponding to each of the multiple multiplication modules is input into the multiple multiplication modules.

7. The electronic device according to claim 6, wherein, The processor further includes a second intermediate value accumulation module to sum the intermediate values ​​of each of the multiple channels obtained through the multiple multiplication modules.

8. A method for controlling an electronic device, the method comprising: Convolution operations are performed on three-dimensional input data through multiple multiplication modules corresponding to multiple channels, wherein the three-dimensional input data includes (i) multiple input values ​​divided according to multiple channels, (ii) a first kernel information including multiple weights of each of the multiple channels with respect to the kernel, and (iii) a second kernel information generated by converting multiple weights configured in two-dimensional matrix form for each of the multiple channels into three-dimensional matrix form. Based on the fact that the convolution operation is a depthwise convolution operation, the control input selection module (a) configures multiple input values ​​corresponding to the first channel among multiple channels, and (b) inputs the multiple input values ​​into two or more multiplication modules among multiple multiplication modules; Based on the second core information, the first set of weights corresponding to the first channel are input one by one into two or more multiplication modules; Multiple intermediate values ​​are obtained based on the result of each multiplication operation by performing multiplication with each of the multiple weights for each of the multiple input values ​​using two or more multiplication modules. as well as The first intermediate value accumulation module sums the intermediate values ​​corresponding to the core positions among multiple intermediate values ​​to obtain multiple output values ​​based on each summation result.

9. The method according to claim 8, wherein, Each of the multiple input values ​​corresponding to the first channel is the input to the input selection module in each preset cycle, and The input selection module is configured to send each input value to two or more multiplication modules in each preset cycle.

10. The method according to claim 8, wherein, The two or more channels include a first channel and at least one channel adjacent to the first channel, and The number of the two or more multiplication modules corresponds to the number of weights included in the first core.

11. The method according to claim 8, wherein, The method further includes summing the intermediate values ​​stored in the buffer corresponding to each position of the core using a first intermediate value accumulation module to obtain multiple output values. The buffer is configured to store an intermediate value that corresponds to a row of the core among a plurality of intermediate values.

12. The method according to claim 11, wherein, The method also includes obtaining multiple output values ​​by performing convolution operations in parallel using two or more multiplication modules corresponding to the number of multiple weights included in the kernel.

13. The method according to claim 9, wherein, The method also includes: Since the convolution operation is a three-dimensional convolution operation, the input selection module is controlled to bypass multiple input values ​​input to the input selection module to multiple multiplication modules; and Based on the first core information, the second set of weights corresponding to each of the multiple multiplication modules is input into the multiple multiplication modules.

14. The method according to claim 13, wherein, The electronic device further includes a second intermediate value accumulation module to sum the intermediate values ​​of each of the multiple channels obtained by the multiple multiplication modules.

15. A non-transitory computer-readable recording medium comprising a program for performing a control method for an electronic device, wherein, The electronic device performs convolution operations based on three-dimensional input data through multiple multiplication modules corresponding to multiple channels. The three-dimensional input data includes (i) multiple input values ​​divided based on multiple channels, (ii) first kernel information with respect to the kernel, including the weights of each of the multiple channels, and (iii) second kernel information generated by converting multiple weights configured in two-dimensional matrix form for each of the multiple channels into three-dimensional matrix form. The method for controlling the electronic device includes: Based on the fact that the convolution operation is a depthwise convolution operation, the input selection module is controlled so that multiple input values ​​corresponding to the first channel among multiple channels are input into all two or more multiplication modules among multiple multiplication modules. Based on the second core information, the first set of weights corresponding to the first channel are input one by one into two or more multiplication modules; Multiple intermediate values ​​are obtained based on the result of each multiplication operation by performing multiplication with each of the multiple weights for each of the multiple input values ​​using two or more multiplication modules; and The first intermediate value accumulation module sums the intermediate values ​​corresponding to the core positions among multiple intermediate values ​​to obtain multiple output values ​​based on each summation result.

Images