Method for performing convolution operation at a specific layer in a neural network by an electronic device and electronic device thereof

Abstract

A method for performing a convolution operation at a specific layer in a neural network by an electronic device, comprising: obtaining N input channel data; performing a first convolution operation by applying a first input channel data group including K first input channel data among the N input channel data to a first kernel filter group including K first kernel filters; performing a second convolution operation by applying a second input channel data group including K second input channel data among the N input channel data to a second kernel filter group including K second kernel filters; and obtaining output channel data based on the first convolution operation and the second convolution operation, wherein K is a natural number less than N.

Description

Technical Field

[0001] The present disclosure relates to a method for an electronic device to perform a convolution operation at a specific layer in a neural network and an electronic device for performing the method. More particularly, it relates to a method for an electronic device to perform a convolution operation at a specific layer in a neural network and an electronic device for performing the method. Background Art

[0002] With the development of computer technology and the exponential growth of data traffic, artificial intelligence (AI) has become a major trend driving future innovation. Because AI mimics how humans think, it has unlimited applications in virtually all industries. Representative AI technologies include pattern recognition, machine learning, expert systems, neural networks, and natural language processing.

[0003] Neural networks model the characteristics of human biological neurons using mathematical expressions and employ algorithms that mimic the human ability known as learning. Using these algorithms, neural networks can generate a mapping between input data and output data, and the ability to generate this mapping can be expressed as the neural network's learning ability. Furthermore, neural networks possess generalization capabilities, enabling them to generate correct output data for input data not previously used for training, based on training results.

[0004] Deepening has been used to achieve high accuracy in convolutional neural networks (CNNs). However, this approach significantly increases computational complexity and reduces inference speed in low-performance environments. To address this issue, research on deep learning architectures such as MobileNet and ShuffleNet has been attempted. Summary of the Invention

[0005] Technical Solution

[0006] The present disclosure provides a method for performing a convolution operation at a specific layer in a neural network by an electronic device and an electronic device for performing the method, the method comprising: obtaining N input channel data; performing a first convolution operation by applying a first input channel data group including K first input channel data selected from the N input channel data to a first kernel filter group including K first kernel filters; performing a second convolution operation by applying a second input channel data group including K second input channel data selected from the N input channel data to a second kernel filter group including K second kernel filters; and obtaining output channel data based on the first convolution operation and the second convolution operation, thereby presenting good performance by reducing accuracy loss while reducing the amount of calculation. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] The above and other aspects, features and advantages of certain embodiments of the present disclosure will become more apparent from the following description taken in conjunction with the accompanying drawings, in which:

[0008] Figure 1 is a block diagram of an electronic device for performing a convolution operation at a specific layer in a neural network according to an embodiment;

[0009] Figure 2 is a flowchart of a method for performing a convolution operation at a specific layer in a neural network by an electronic device according to an embodiment;

[0010] Figure 3 It is a diagram used to describe the structure of a neural network;

[0011] Figure 4 is a diagram for describing a convolution operation in the prior art;

[0012] Figures 5a to 5c A method for performing a convolution operation at a specific layer in a neural network by an electronic device according to an embodiment is shown;

[0013] Figure 6 A method for performing a convolution operation at a specific layer in a neural network by an electronic device according to another embodiment is shown;

[0014] Figure 7 A method for performing a convolution operation at a specific layer in a neural network by an electronic device according to another embodiment is shown;

[0015] Figure 8 A method for performing a convolution operation at a specific layer in a neural network by an electronic device according to another embodiment is shown; and

[0016] Figure 9 is a graph comparing the performance of the prior art and the performance of a method in which an electronic device performs a convolution operation at a specific layer in a neural network according to an embodiment. DETAILED DESCRIPTION

[0017] The present disclosure provides a method for performing a convolution operation at a specific layer in a neural network by an electronic device and an electronic device for performing the method, the method comprising: obtaining N input channel data; performing a first convolution operation by applying a first input channel data group including K first input channel data selected from the N input channel data to a first kernel filter group including K first kernel filters; performing a second convolution operation by applying a second input channel data group including K second input channel data selected from the N input channel data to a second kernel filter group including K second kernel filters; and obtaining output channel data based on the first convolution operation and the second convolution operation, thereby presenting good performance by reducing accuracy loss while reducing the amount of calculation.

[0018] Additional aspects will be set forth in part in the description which follows and, in part, will be obvious from the description, or may be learned by practice of the presented embodiments.

[0019] According to one aspect of the present disclosure, a method for performing a convolution operation at a specific layer in a neural network by an electronic device includes: obtaining N input channel data; performing a first convolution operation in the neural network by the electronic device applying a first input channel data group including K first input channel data among the N input channel data to a first kernel filter group including K first kernel filters; performing a second convolution operation in the neural network by the electronic device applying a second input channel data group including K second input channel data among the N input channel data to a second kernel filter group including K second kernel filters; and obtaining output channel data based on the first convolution operation and the second convolution operation, wherein N is a natural number and K is a natural number less than N.

[0020] The N input channel data may have indexes from 0 to N-1; the first input channel data group may include the K first input channel data corresponding to indexes from 0 to K-1; the second input channel data group may include K second input channel data corresponding to indexes from S to S+K-1; and S may be determined according to a preset channel step.

[0021] The K pieces of first input channel data may include a plurality of input channel data corresponding to indexes from 0 to K-1 respectively; and the K pieces of second input channel data may include a plurality of input channel data corresponding to indexes from S to S+K-1 respectively.

[0022] The number of output channel data items can be determined based on S and K.

[0023] The N pieces of input channel data may include a preset number of padding data.

[0024] Among the N pieces of input channel data, pieces of input channel data corresponding to indexes from 0 to P-1 and pieces of input channel data corresponding to indexes from NP to N-1 may be pieces of padding data.

[0025] The plurality of pieces of padding data may be copied from the N pieces of input channel data to be identical to a portion of a plurality of pieces of output data calculated at a layer before a specific layer.

[0026] The N pieces of input channel data may be based on image data processed by the electronic device.

[0027] The convolution operation can be performed in a convolutional neural network (CNN).

[0028] According to another aspect of the present disclosure, an electronic device for performing a convolution operation at a specific layer of a neural network includes: a memory storing one or more instructions; and a processor configured to execute the one or more instructions stored in the memory to perform the following operations: obtaining N input channel data; performing a first convolution operation in the neural network by applying a first input channel data group including K first input channel data among the N input channel data to a first kernel filter group including K first kernel filters; performing a second convolution operation in the neural network by applying a second input channel data group including K second input channel data among the N input channel data to a second kernel filter group including K second kernel filters; and obtaining output channel data based on the first convolution operation and the second convolution operation, wherein N is a natural number and K is a natural number less than N.

[0029] The N input channel data may have indexes from 0 to N-1; the first input channel data group may include the K first input channel data corresponding to indexes from 0 to K-1; the second input channel data group may include the K second input channel data corresponding to indexes from S to S+K-1; and S may be determined according to a preset channel step.

[0030] The K pieces of first input channel data may include a plurality of input channel data corresponding to indexes from 0 to K-1 respectively; and the K pieces of second input channel data may include a plurality of input channel data corresponding to indexes from S to S+K-1 respectively.

[0031] The number of output channel data items can be determined based on S and K.

[0032] The N pieces of input channel data may include a preset number of padding data.

[0033] Among the N pieces of input channel data, pieces of input channel data corresponding to indexes from 0 to P-1 and pieces of input channel data corresponding to indexes from NP to N-1 may be pieces of padding data.

[0034] The plurality of pieces of padding data may be copied from the N pieces of input channel data to be identical to a portion of a plurality of pieces of output data calculated at a layer before a specific layer.

[0035] The N pieces of input channel data may be based on image data processed by the electronic device.

[0036] The convolution operation can be performed in a convolutional neural network (CNN).

[0037] According to another aspect of the present disclosure, a non-transitory computer-readable recording medium stores a program for a method for performing a convolution operation on a specific layer in a neural network running on an electronic device, the method including: obtaining N input channel data; performing a first convolution operation in the neural network by applying a first input channel data group including K first input channel data among the N input channel data to a first kernel filter group including K first kernel filters; performing a second convolution operation in the neural network by applying a second input channel data group including K second input channel data among the N input channel data to a second kernel filter group including K second kernel filters; and obtaining output channel data based on the first convolution operation and the second convolution operation, wherein N is a natural number and K is a natural number less than N.

[0038] The N input channel data may have indexes from 0 to N-1; the first input channel data group may include the K first input channel data corresponding to indexes from 0 to K-1; the second input channel data group may include the K second input channel data corresponding to indexes from S to S+K-1; and S may be determined according to a preset channel step.

[0039] Terms used in the specification will be schematically described, and then one or more embodiments will be described in detail.

[0040] The terms used in this disclosure are those currently widely used in the art, but these terms may vary according to the intentions of those skilled in the art, precedents, or new technologies in the art. In addition, the applicant may choose to specify a term, and in this case, the detailed meaning of the specified term will be described in the detailed description or can be understood from the detailed description. Therefore, the terms used in this disclosure should not be understood as simple names, but should be understood based on the meaning and overall description of the term.

[0041] Throughout the specification, it should also be understood that when a component “includes” an element, unless there is a description to the contrary, it should be understood that the component does not exclude another element, but may further include another element. In addition, terms such as “… unit”, “… module”, etc. refer to units that perform at least one function or operation, and these units can be implemented as hardware or software or a combination of hardware and software.

[0042] Throughout the disclosure, expressions such as “at least one of a, b, or c” refer to variations of only a, only b, only c, both a and b, both a and c, both b and c, all of a, b, and c, or only a, only b, only c, both a and b, both a and c, both b and c, or all of a, b, and c.

[0043] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings so that those skilled in the art can easily implement the present disclosure. However, the present disclosure can be embodied in many different forms and should not be construed as being limited to the embodiments set forth herein. In the accompanying drawings, portions not relevant to the description may be omitted to clearly describe the present disclosure, and the same reference numerals represent the same elements throughout the specification.

[0044] In this specification, "input channel data" refers to data included in the input channels used in the convolution processing of the neural network, "padding data" refers to data used as input channel data by copying part of the channel data output at the previous layer, and "output channel data" refers to data included in the output channels as the convolution result of the neural network. These types of data may include image data, speech data, etc.

[0045] Figure 1 is a block diagram of an electronic device 100 for performing a convolution operation at a specific layer in a neural network according to an embodiment.

[0046] refer to Figure 1 , the electronic device 100 for performing a convolution operation at a specific layer in a neural network according to an embodiment may include a processor 120 (eg, at least one processor) and a memory 130 .

[0047] The processor 120 may generally control the electronic device 100. In addition, the processor 120 may execute one or more programs stored in the memory 130.

[0048] The memory 130 may store various data and programs or applications for operating and controlling the electronic device 100. The programs stored in the memory 130 may include one or more instructions. The programs (e.g., one or more instructions) or applications stored in the memory 130 may be executed by the processor 120.

[0049] According to an embodiment, the processor 120 can obtain N input channel data, perform a first convolution operation by applying a first input channel data group including K first input channel data selected from the N input channel data to a first kernel filter group including K first kernel filters, perform a second convolution operation by applying a second input channel data group including K second input channel data selected from the N input channel data to a second kernel filter group including K second kernel filters, and obtain output channel data based on the first convolution operation and the second convolution operation.

[0050] Figure 2 is a flowchart of a method for the electronic device 100 to perform a convolution operation at a specific layer in a neural network according to an embodiment.

[0051] refer to Figure 2 , the processor 120 in the electronic device 100 for performing a convolution operation at a specific layer in a neural network may obtain N pieces of input channel data in operation S201.

[0052] For example, the input channel data may be based on image data processed by the electronic device 100 .

[0053] In operation S203 , a first convolution operation may be performed by applying a first input channel data group including K pieces of first input channel data selected from N pieces of input channel data to a first kernel filter group including K first kernel filters.

[0054] In this specification, it should be understood that "plural first input channel data" does not mean the same multiple input channel data, but is used to distinguish multiple input channel data included in the first input channel data group from other multiple input channel data. Similarly, "plural second input channel data" does not mean the same multiple input channel data, but is used to distinguish multiple input channel data included in the second input channel data group from other multiple input channel data. In this regard, the terms "first" and "second" are used for distinction purposes and do not imply a specific order or hierarchy.

[0055] In operation S205, a second convolution operation may be performed by applying a second input channel data group including K second input channel data selected from N input channel data to a second kernel filter group including K second kernel filters. K is a natural number smaller than N.

[0056] The N input channel data may have indexes from 0 to N-1, the first input channel data group may include multiple first input channel data corresponding to indexes from 0 to K-1, the second input channel data group may include multiple second input channel data corresponding to indexes from S to S+K-1, and S may be determined by a preset channel step value.

[0057] The plurality of first input channel data may include a plurality of input channel data corresponding to indexes 0 to K-1, respectively. In addition, the plurality of second input channel data may include a plurality of input channel data corresponding to indexes S to S+K-1, respectively.

[0058] The N pieces of input channel data may include a preset number of padding data.

[0059] A plurality of padding data may be copied from a plurality of input channel data to be identical to a portion selected from a plurality of output data calculated at a layer preceding a specific layer.

[0060] Among the plurality of pieces of input channel data, the plurality of pieces of input channel data corresponding to indexes 0 to P-1 and the plurality of pieces of input channel data corresponding to indexes NP to N-1 may be the plurality of pieces of padding data.

[0061] In operation S207 , a plurality of output channel data may be obtained based on the first convolution operation and the second convolution operation.

[0062] Convolution operations can be performed in convolutional neural networks (CNNs).

[0063] The number of output channel data items can be determined based on S and K.

[0064] In particular, the number of output channel data can be determined by the following mathematical formula 1:

[0065] [Mathematical formula 1]

[0066] out_ch=(in_ch-kernel_num+2*pad) / ch_stride+1

[0067] Among them, out_ch represents the number of output channel data, in_ch represents the number of input channel data, kernel_num represents the number of input channel data of the input channel data group applied to the kernel filter group, 2*pad represents the number of padding data, and ch_stride represents the index interval of multiple input channel data to which the kernel filter group is applied.

[0068] Figure 3 is a diagram for describing the general structure of the neural network 300.

[0069] The neural network 300 may have an architecture of a deep neural network (DNN) or an n-layer neural network. The DNN or N-layer neural network may correspond to a CNN, a recurrent neural network (RNN), a deep belief network, a restricted Boltzmann machine, etc. For example, the neural network 300 may be implemented by a CNN, but is not limited thereto. In addition to the convolutional layer, Figure 3 The CNN corresponding to the example of the neural network 300 may also include a subsampling layer, a pooling layer, a fully connected layer, etc.

[0070] Neural network 300 can be implemented using an architecture with multiple layers, wherein the layers include an input image, a feature map, and an output. In neural network 300, a convolution operation is performed on the input image using a filter called a weight or kernel, thereby outputting a feature map. These generated output feature maps are then convolved with the kernel filter as the input feature map, thereby outputting a new feature map. As a result of repeatedly performing this convolution operation, a recognition result of the features of the input image passed through neural network 300 can ultimately be output.

[0071] For example, when an image with a pixel size of 24×24 is input to Figure 3 When the neural network 300 in FIG. 1 is used, a convolution operation between the input image and the kernel filter can be performed to output a four-channel feature map of size 20×20. Alternatively, a four-channel feature map of size 10×10 can be output by subsampling using only some pixel values ​​of the four-channel feature map of size 20×20. Subsampling schemes may include a maximum pooling scheme, an average pooling scheme, and the like.

[0072] Thereafter, the size of the 10×10 feature map can be reduced by repeated convolution operations and subsampling operations using weights, thereby ultimately outputting global features. The neural network 300 can filter and output robust features that generally represent the image from the input image by repeatedly performing convolution operations and subsampling (or pooling) operations in several layers. The output global features can be input into a fully connected layer to ultimately derive the recognition result of the input image.

[0073] Figure 3 The input feature map and output feature map in can be referenced respectively Figure 1 、 2 , 4, 5A to 5C and 6 to 8 describe the input channel data and output channel data.

[0074] Figure 4 is a diagram for describing a convolution operation in the related art.

[0075] Reference Figure 4To obtain M output channel data from N input channel data, M×N kernel filters are required. For example, to obtain four, for example, first to fourth output channel data 450, 460, 470, and 480, from four input channel data 410, 420, 430, and 440, 4×4 (i.e., 16) kernel filters are required. Furthermore, when the width multiplied by the height of the kernel filter is W×H, the convolution operation requires M×N×W×H filter parameters. For example, when four input channel data and four output channel data are obtained and the kernel filter size is 3×3, the convolution operation requires 4×4×3×3 (i.e., 144) filter parameters. Specifically, the first output channel data 450 can be obtained by performing a convolution operation on the four input channel data 410, 420, 430, and 440 and the kernel filters 411, 412, 413, and 414, respectively, and summing the convolution results. The second output channel data 460 can be obtained by performing a convolution operation on the four input channel data 410, 420, 430 and 440 and the kernel filters 421, 422, 423 and 424, respectively, and summing the convolution results. The third output channel data 470 can be obtained by performing a convolution operation on the four input channel data 410, 420, 430 and 440 and the kernel filters 431, 432, 433 and 434, respectively, and summing the convolution results. The fourth output channel data 480 can be obtained by performing a convolution operation on the four input channel data 410, 420, 430 and 440 and the kernel filters 441, 442, 443 and 444, respectively, and summing the convolution results.

[0076] pass Figure 4 The number of parameters obtained by the convolution operation is determined by the following mathematical formula 2.

[0077] [Mathematical formula 2]

[0078] param_num=in_ch*kernel_W*kernel_H*out_ch

[0079] Here, param_num represents the number of parameters, in_ch represents the number of input channel data, kernel_W represents the width of the kernel filter, kernel_H represents the height of the kernel filter, and out_ch represents the number of output channel data.

[0080] Figures 5a to 5c A method of performing a convolution operation at a specific layer in a neural network according to an embodiment is shown.

[0081] Reference Figures 5a to 5c, the output channel data can be obtained by applying the input channel data group including K input channel data selected from N input channel data to the kernel filter group including K kernel filters to perform convolution operation multiple times. Figure 4 Compared with the convolution operation of the prior art, a smaller number of kernel filters and a smaller number of kernel filter parameters are used. Therefore, the efficiency of low-performance devices can be improved while reducing the computational load of the neural network. As an example, the electronic device 100 can be a device for performing object recognition (e.g., recognizing objects in images, facial recognition, etc.), a device for performing biometric sensing or recognition, a device for performing voice recognition, etc. By implementing the method of performing convolution operations according to one or more embodiments, the electronic device 100 can provide improved efficiency and / or processing speed while reducing the computational load for performing its corresponding operations (e.g., object recognition, voice recognition, etc.).

[0082] refer to Figures 5a to 5c A process of obtaining four (eg, first to fourth) output channel data 560 , 570 , 580 , and 590 with respect to four input channel data 510 , 520 , 530 , and 540 according to an embodiment is specifically described.

[0083] According to an embodiment, 2P pieces of padding data can be generated based on M pieces of input channel data using a preset padding parameter P. Thus, M+2P pieces of new input channel data can be obtained. The M pieces of input channel data can be output channel data output as a result of a convolution operation in a previous layer. For example, when the preset padding parameter p is 1, 2p (i.e., two pieces) of padding data 500 and 550 can be obtained based on the four pieces of input channel data 510, 520, 530, and 540. Thus, six new pieces of input channel data 500, 510, 520, 530, 540, and 550, including the four pieces of input channel data 510, 520, 530, and 540 and the two pieces of padding data 500 and 550, can be obtained.

[0084] Reference Figure 5a , the first output channel data 560 can be obtained by applying a first input channel data group including three first input channel data 500, 510 and 520 selected from six input channel data 500, 510, 520, 530, 540 and 550 to a first kernel filter group including three first kernel filters 501, 502 and 503 to perform a convolution operation of convolving the three first input channel data 500, 510 and 520 with the three first kernel filters 501, 502 and 503 respectively and summing the convolution results.

[0085] refer to Figure 5b, the second output channel data 570 can be obtained by applying a second input channel data group including three second input channel data 510, 520 and 520 selected from the six input channel data 500, 510, 520, 530, 540 and 550 to a second kernel filter group including three second kernel filters 511, 512 and 513 to perform a convolution operation of convolving the three second input channel data 510, 520 and 520 with the three second kernel filters 511, 512 and 513 respectively and summing the convolution results.

[0086] Reference Figure 5c , the third output channel data 580 can be obtained by applying a third input channel data group including three third input channel data 520, 530 and 540 selected from the six input channel data 500, 510, 520, 530, 540 and 550 to a third kernel filter group including three third kernel filters 521, 522 and 523 to perform a convolution operation of the three third input channel data 520, 530 and 540 and the three third kernel filters 521, 522 and 523. The fourth output channel data 590 can be obtained by applying a fourth input channel data group including three fourth input channel data 530, 540 and 550 selected from the six input channel data 500, 510, 520, 530, 540 and 550 to a fourth kernel filter group including three fourth kernel filters 531, 532 and 533 to perform a convolution operation of the three fourth input channel data 530, 540 and 550 and the three fourth kernel filters 531, 532 and 533.

[0087] Although Figure 4 In the example, 16 kernel filters are required for the same input channel data and the same output channel data, but in Figures 5a to 5c In the embodiment of FIG. 1 , 12 kernel filters are used. Therefore, the processing efficiency and speed in the electronic device 100 are improved.

[0088] The number of kernel filters required or used by the method for performing a convolution operation at a specific layer in a neural network according to an embodiment is the product of the number of K kernel filters included in the plurality of kernel filter groups and the number of output channel data. Based on Mathematical Formula 1, the number of kernel filters required or used is represented by the following Mathematical Formula 3.

[0089] [Mathematical formula 3]

[0090] kernelfilter_num=kernel_num*((in_ch-kernel_num+2*pad) / ch_stride+1)

[0091] Here, kernelfilter_num represents the total number of kernel filters used for convolution operation, in_ch represents the number of input channel data, kernel_num represents the number of input channel data of the input channel data group applied to the kernel filter group, 2*pad represents the number of padding data, and ch_stride represents the index interval of the input channel data to which the kernel filter group is applied.

[0092] The number of parameters obtained by the method of performing a convolution operation at a specific layer in a neural network according to an embodiment is determined by the following mathematical formula 4.

[0093] [Mathematical formula 4]

[0094] param_num=ch_kernel*kernel_W*kernel_H*

[0095] ((in_ch-kernel_num+2*pad) / ch_stride+1)

[0096] Here, param_num represents the number of parameters, ch_kernel represents the number of kernel filters included in the kernel filter group applied to the input channel data group, kernel_W represents the width of the kernel filter, kernel_H represents the height of the kernel filter, in_ch represents the number of input channel data pieces, kernel_num represents the number of input channel data pieces of the input channel data group applied to the kernel filter group, 2*pad represents the number of padding data pieces, and ch_stride represents the index interval of the input channel data to which the kernel filter group is applied.

[0097] When applying Mathematical Formula 4 to Figure 5c In the embodiment, the number of parameters is 108. The number 108 is smaller than that obtained by assuming Figure 4 The size of the kernel filter in is 3x3 and the number of parameters computed is 144.

[0098] Figure 5c The plurality of input channel data 500, 510, 520, 530, 540, and 550 may have indexes. In particular, the plurality of input channel data 500, 510, 520, 530, 540, and 550 may have indexes from 0 to 5. Figure 5c In the embodiment of FIG. 5 , the indexes of the plurality of padding data 500 and 550 among the plurality of input channel data 500 , 510 , 520 , 530 , 540 , and 550 are 0 and 5 respectively.

[0099] According to an embodiment, multiple pieces of padding data may be copied from multiple pieces of input channel data to be identical to a portion selected from output channel data calculated at a previous layer of a specific layer. Specifically, padding data 500 having an index of 0 may be identical to input channel data 540 having an index of 4, and padding data 550 having an index of 5 may be identical to input channel data 510 having an index of 1. Alternatively, padding data 500 having an index of 0 may be identical to input channel data 540 having an index of 1, and padding data 550 having an index of 5 may be identical to input channel data 510 having an index of 4.

[0100] In addition, when (or based on) M channel data (which are multiple output channel data calculated in the previous layer) and 2P padding data are multiple new input channel data, the multiple input channel data can have indexes from 0 to M+2P-1. In this case, when (or based on) N=M+2P-1, the indexes of the multiple padding data among the multiple input channel data can be 0 to P-1 and NP to N-1.

[0101] The plurality of padding data with indexes 0 to P-1 can be respectively identical to the plurality of input channel data with indexes N-2P to NP-1, and the plurality of padding data with indexes NP to N-1 can be respectively identical to the plurality of input channel data with indexes P to 2P-1. Alternatively, the plurality of padding data with indexes 0 to P-1 can be respectively identical to the plurality of input channel data with indexes P to 2P-1, and the plurality of padding data with indexes NP to N-1 can be respectively identical to the plurality of input channel data with indexes N-2P to NP-1. Alternatively, the plurality of padding data with indexes 0 to P-1 can all be identical to the input channel data with index P, and the plurality of padding data with indexes NP to N-1 can be identical to the input channel data with index NP-1. The plurality of padding data can be set to a preset default value.

[0102] Figure 6 A method for performing a convolution operation at a specific layer in a neural network according to another embodiment is shown.

[0103] refer to Figure 6 , describes in detail the process of obtaining four (e.g., first to fourth) output channel data 660, 670, 680, and 690 with respect to six input channel data 600, 610, 620, 630, 640, and 650 according to the embodiment. Specifically, the process of obtaining four (e.g., first to fourth) output channel data 660, 670, 680, and 690 with respect to six input channel data 600, 610, 620, 630, 640, and 650 according to the embodiment is described. Figures 5a to 5c A process of performing a convolution operation differently using input channel data without using padding data obtained by copying a portion of the input channel data.

[0104] according to Figure 4The prior art convolution operation requires or uses 4×6 (ie, 24) kernel filters to obtain four output channel data relative to six input channel data. Figure 6 As described below, 12 kernel filters may be needed or used to reduce the amount of computation.

[0105] In addition to the presence (or absence) of padding data, Figure 6 The convolution process and Figures 5a to 5c Specifically, the convolution operation of performing convolution on the three first input channel data 600, 610 and 620 and summing the convolution results can be performed by applying the first input channel data group including the three first input channel data 600, 610 and 620 selected from the six input channel data 600, 610, 620, 630, 640 and 650 to the first kernel filter group including the three first kernel filters 601, 602 and 603, respectively, to obtain the first output channel data 660. The second output channel data 670 can be obtained by applying a second input channel data group including three second input channel data 610, 620 and 630 selected from the six input channel data 600, 610, 620, 630, 640 and 650 to a second kernel filter group including three second kernel filters 611, 612 and 613 to perform a convolution operation of convolving the three second input channel data 610, 620 and 630 with the three second kernel filters 611, 612 and 613 respectively and summing the convolution results. The third output channel data 680 can be obtained by applying a third input channel data group including three third input channel data 620, 630 and 640 selected from the six input channel data 600, 610, 620, 630, 640 and 650 to a third kernel filter group including three third kernel filters 621, 622 and 623 to perform a convolution operation of convolving the three third input channel data 620, 630 and 640 with the three third kernel filters 621, 622 and 623 respectively and summing the convolution results. The fourth output channel data 690 can be obtained by applying a fourth input channel data group including three fourth input channel data 630, 640 and 650 selected from the six input channel data 600, 610, 620, 630, 640 and 650 to a fourth kernel filter group including three fourth kernel filters 631, 632 and 633 to perform a convolution operation of convolving the three fourth input channel data 630, 640 and 650 with the three fourth kernel filters 631, 632 and 633 respectively and summing the convolution results.

[0106] Figures 5a to 5c and Figure 6 This is an embodiment assuming that the channel stride is 1.

[0107] refer to Figures 5a to 5c and Figure 6 Because the channel stride S (e.g., S=1) is smaller than the number K of kernel filters in the kernel filter bank applied to the input channel data group (e.g., K=3), the input channel data included in the input channel data group can be applied to each kernel filter of different kernel filter banks, thereby affecting the acquisition of different multiple output channel data through the convolution operation. Therefore, because the input channel data affects the determination of different multiple output channel data, and the input channel data group including the multiple input channel data is applied to the kernel filter bank to determine the output channel data, the multiple input channel data and the multiple output channel data affect each other.

[0108] Figure 7 A method for performing a convolution operation at a specific layer in a neural network according to another embodiment is shown.

[0109] Figure 7 An example with a channel stride of 2 is shown.

[0110] refer to Figure 7 , describes in detail the process of obtaining four (e.g., first to fourth) output channel data 745, 750, 755 and 760 relative to nine input channel data 700, 705, 710, 715, 720, 725, 730, 735 and 740.

[0111] Specifically, the first output channel data 745 can be obtained by applying a first input channel data group including three first input channel data 700, 705 and 710 selected from the nine input channel data 700, 705, 710, 715, 720, 725, 730, 735 and 740 to a first kernel filter group including three first kernel filters 701, 702 and 703 to perform a convolution operation of convolving the three first input channel data 700, 705 and 710 with the three first kernel filters 701, 702 and 703 respectively and summing the convolution results. The second input channel data group including three second input channel data 710, 715 and 720 selected from the nine input channel data 700, 705, 710, 715, 720, 725, 730, 735 and 740 can be applied to a second kernel filter group including three second kernel filters 711, 712 and 713 to perform a convolution operation of convolving the three second input channel data 710, 715 and 720 with the three second kernel filters 711, 712 and 713 respectively and summing the convolution results, thereby obtaining the second output channel data 750. Here, the size of the interval between the selected plurality of second input channel data 710, 715 and 720 and the plurality of first input channel data 700, 705 and 710 can be S, which is the size of the channel stride. Similarly, the third output channel data 755 can be obtained by applying a third input channel data group including three third input channel data 720, 725 and 730 selected from the nine input channel data 700, 705, 710, 715, 720, 725, 730, 735 and 740 to a third kernel filter group including three third kernel filters 721, 722 and 723 to perform a convolution operation of convolving the three third input channel data 720, 725 and 730 with the three third kernel filters 721, 722 and 723 respectively and summing the convolution results. The fourth output channel data 760 can be obtained by applying a fourth input channel data group including three fourth input channel data 731, 732 and 733 selected from the nine input channel data 700, 705, 710, 715, 720, 725, 730, 735 and 740 to a fourth kernel filter group including three fourth kernel filters 731, 732 and 733 to perform a convolution operation of convolving the three fourth input channel data 730, 735 and 740 with the three fourth kernel filters 731, 732 and 733 respectively and summing the convolution results.

[0112] exist Figure 7 Although according to Figure 4The prior art convolution operation method requires 36 kernel filters corresponding to the product of 9 as the number of pieces of input channel data and 4 as the number of pieces of output channel data to perform the convolution operation, but only 12 kernel filters are required or used to perform the convolution operation according to Figure 7 The convolution operation of the embodiments of the present disclosure.

[0113] In addition, assuming that the kernel filter size is 3x3, although according to Figure 4 The prior art convolution operation method requires 324 parameters to perform the convolution operation, but only 108 parameters are required or used to perform the convolution operation according to Figure 7 The convolution operation of the embodiment of the present disclosure.

[0114] Figure 8 A method for performing a convolution operation at a specific layer in a neural network according to another embodiment is shown.

[0115] refer to Figure 8 , describes in detail the processing of obtaining eight (e.g., the first to eighth) output channel data 830, 835, 840, 845, 850, 855, 860 and 860 relative to new input channel data including four input channel data 805, 810, 815 and 820 and two padding data 800 and 825 when the channel stride is 1.

[0116] A convolution operation of the three first input channel data 800, 805, and 810 with the three first kernel filters 801, 802, and 803 can be performed by applying a first input channel data group including three first input channel data 800, 805, 810 selected from the six input channel data 800, 805, 810, 815, 820, and 825 to a first kernel filter group including three first kernel filters 801, 802, and 803, thereby obtaining first output channel data 830. A convolution operation of the first input channel data 800, 805, and 810 with the three second kernel filters 811, 812, and 813 can be performed by applying the first input channel data group to a second kernel filter group including three second kernel filters 811, 812, and 813, thereby obtaining second output channel data 835. Similarly, the third output channel data 840 can be obtained by applying a second input channel data group including three second input channel data 805, 810 and 815 selected from the six input channel data 800, 805, 810, 815, 820 and 825 to a third kernel filter group including three third kernel filters 821, 822 and 823 to perform a convolution operation of the three second input channel data 805, 810 and 815 with the three third kernel filters 821, 822 and 823. The fourth output channel data 845 can be obtained by applying a second input channel data group including three second input channel data 805, 810 and 815 selected from the six input channel data 800, 805, 810, 815, 820 and 825 to a fourth kernel filter group including three fourth kernel filters 831, 832 and 833 to perform a convolution operation of the second input channel data 805, 810 and 815 with the three fourth kernel filters 831, 832 and 833. The fifth output channel data 850 can be obtained by applying a third input channel data group including three third input channel data 810, 815 and 820 selected from the six input channel data 800, 805, 810, 815, 820 and 825 to a fifth kernel filter group including three fifth kernel filters 841, 842 and 843 to perform a convolution operation of the three third input channel data 810, 815 and 820 and the three fifth kernel filters 841, 842 and 843. The sixth output channel data 855 can be obtained by applying a third input channel data group including three third input channel data 810, 815 and 820 selected from the six input channel data 800, 805, 810, 815, 820 and 825 to a sixth kernel filter group including three sixth kernel filters 851, 852 and 853 to perform a convolution operation of the third input channel data 810, 815 and 820 and the three sixth kernel filters 851, 852 and 853.The seventh output channel data 860 can be obtained by applying a fourth input channel data group including three fourth input channel data 815, 820 and 825 selected from the six input channel data 800, 805, 810, 815, 820 and 825 to a seventh kernel filter group including three seventh kernel filters 861, 862 and 863 to perform a convolution operation of the three fourth input channel data 815, 820 and 825 and the three seventh kernel filters 861, 862 and 863. The eighth output channel data 865 can be obtained by applying a fourth input channel data group including three fourth input channel data 815, 820 and 825 selected from the six input channel data 800, 805, 810, 815, 820 and 825 to an eighth kernel filter group including three eighth kernel filters 871, 872 and 873 to perform a convolution operation of the fourth input channel data 815, 820 and 825 and the three eighth kernel filters 871, 872 and 873.

[0117] exist Figure 8 Although according to Figure 4 The prior art convolution operation method requires 32 kernel filters corresponding to the product of 4 as the number of input channel data and 8 as the number of output channel data to perform the convolution operation, but only 24 kernel filters are required or used to perform the convolution operation according to Figure 8 The convolution operation of the embodiment of the present disclosure.

[0118] exist Figure 8 In the embodiment of the present invention, when a plurality of kernel filter banks are applied to an input channel data group instead of one kernel filter bank to obtain output channel data, the number of obtained parameters is determined by the following mathematical formula 5.

[0119] [Mathematical formula 5]

[0120] param_num=ch_kernel*kernel_W*kernel_H*L*

[0121] ((in_ch-kernel_num+2*pad) / ch_stride+1)

[0122] Here, param_num represents the number of parameters, ch_kernel represents the number of kernel filters included in the kernel filter group applied to the input channel data group, kernel_W represents the width of the kernel filter, kernel_H represents the height of the kernel filter, in_ch represents the number of input channel data, kernel_num represents the number of input channel data of the input channel data group applied to the kernel filter group, 2*pad represents the number of padding data, ch_stride represents the index interval of the input channel data to which the kernel filter group is applied, and L represents the number of kernel filter groups applied to the input channel data group.

[0123] For example, when applying Mathematical Formula 5 to Figure 8 In the embodiment, the number of parameters is 216. This number is less than 288, where 288 is obtained by Figure 4 The number of parameters obtained by the convolution operation of the prior art.

[0124] Figure 9 is a graph comparing the performance of the prior art and the performance of a method in which an electronic device performs a convolution operation at a specific layer in a neural network according to an embodiment.

[0125] The diamond mark 910 indicates the application result of the prior art, the triangle mark 920 and the x mark 930 indicate the application result of MobileNet, and the star mark 940 and the circle mark 950 indicate the application result of ShuffleNet. The square mark 900 indicates the application result of the method of performing a convolution operation on a specific layer in a neural network by an electronic device according to an embodiment. Figure 9 , and according to Figure 6 Compared with the convolution operation of the prior art shown, the method of performing convolution operations in a specific layer in a neural network may have almost no performance degradation and has about one-third the number of parameters (the computational amount of multiplication and addition), and compared with MobileNet and ShuffleNet, which are the results of research for reducing the amount of computation, the peak signal-to-noise ratio (PSNR) can be improved by 1dB or more with the same amount of computation.

[0126] In addition, a software module can be used to implement convolution operations at specific layers in a neural network. Figure 1The operation of the processor 120 in the electronic device 100 is described. When the processor 120 is implemented by a software module (or a program module including instructions), the software module can be stored in a non-transitory computer-readable recording medium. In this case, at least one software module can be provided by an operating system (OS) or a specific application. Alternatively, a portion of at least one software module can be provided by the OS, and the other portion can be provided by a specific application.

[0127] Each component of the block diagram may be integrated, added, or omitted according to the specifications of the electronic device 100 actually implemented. That is, depending on the situation, two or more components may be integrated into one component, or one component may be divided into two or more components. In addition, the functions performed by each block are for the purpose of describing the embodiments of the present disclosure, and its specific operations or devices do not limit the scope of the present disclosure.

[0128] The method of performing a convolution operation at a specific layer in a neural network by the electronic device 100 can be implemented in the form of program commands that can be executed by various computer devices and recorded on a non-transitory computer-readable recording medium. The non-transitory computer-readable recording medium may include program commands, data files, data structures, etc., alone or in combination. The program commands recorded on the non-transitory computer-readable recording medium may be specially designed and constructed for the present disclosure, or may be known and usable by a person of ordinary skill in the field of computer software. Examples of non-transitory computer-readable recording media include magnetic media (e.g., hard disks, floppy disks, or tapes), optical media (e.g., compact disk read-only memories (CD-ROMs) or digital versatile disks (DVDs)), magneto-optical media (e.g., magneto-optical disks), and hardware devices (e.g., ROM, RAM, or flash memory) specifically configured to store and execute program commands. Examples of program commands include high-level language codes that can be executed by a computer using an interpreter and machine language codes generated by a compiler.

[0129] In addition, the method and electronic device for performing a convolution operation at a specific layer in a neural network according to an embodiment of the present disclosure can be provided by including the method and electronic device in a computer program product. The computer program product can be traded between a seller and a buyer.

[0130] The computer program product may include a software (S / W) program, a non-transitory computer-readable storage medium storing the S / W program. For example, the computer program product may include a product (e.g., a downloadable application) in the form of a S / W program electronically distributed by a manufacturer of an electronic device or an electronic market (e.g., GOOGLE PLAYSTORE or APP STORE). For electronic distribution, at least a portion of the S / W program may be stored in a storage medium or temporarily generated. In this case, the storage medium may be included in a server of the manufacturer, a server of the electronic market, or a relay server configured to temporarily store the S / W program.

[0131] In a system including a server and a client device, a computer program product may include the storage medium of the server or the storage medium of the client device. Alternatively, when there is a third device (e.g., a smartphone) connected to the server or client device via communication, the computer program product may include the storage medium of the third device. In addition, the computer program product may include a software program to be transmitted from the server to the client device or the third device, or from the third device to the client device.

[0132] In this case, one of the server, the client device, and the third device may execute the computer program product and perform the method according to the embodiment of the present disclosure. Alternatively, two or more of the server, the client device, and the third device may execute the computer program product and perform the method according to the embodiment of the present disclosure in a distributed manner.

[0133] The method of performing a convolution operation at a specific layer in a neural network by an electronic device according to an embodiment can allow the electronic device to perform a convolution operation at a specific layer in a neural network by using a convolution operation of applying an input channel data group including K input channel data selected from N input channel data to a kernel filter group including K kernel filters, thereby reducing the number of kernel filter parameters and the number of kernel filters, thereby reducing the amount of calculation and improving the performance of the device.

[0134] For example, a server (e.g., a cloud server or an artificial intelligence server) can execute a computer program product stored in the server to control a client device connected to the server via communication, wherein the client device performs a method according to a disclosed embodiment of the present disclosure.

[0135] Although the embodiments of the present disclosure have been described in detail, the scope of the present disclosure is not limited thereto, and various modifications and improved forms by those skilled in the art using at least the basic concept of the present disclosure defined in the claims are also within the scope of the present disclosure.

Claims

1. A method for performing a convolution operation at a specific layer in a neural network by an electronic device, the method comprising: Get N input channel data; performing, by the electronic device, a first convolution operation in the neural network by applying a first input channel data group including K first input channel data among the N input channel data to a first kernel filter group including K first kernel filters; performing, by the electronic device, a second convolution operation in the neural network by applying a second input channel data group including K second input channel data among the N input channel data to a second kernel filter group including K second kernel filters; and Obtaining output channel data based on the first convolution operation and the second convolution operation, Where N is a natural number, and K is a natural number less than N, in: The N input channel data have indexes from 0 to N-1; The first input channel data group includes the K pieces of first input channel data corresponding to indexes from 0 to K-1; The second input channel data group includes the K pieces of second input channel data corresponding to indexes from S to S+K-1; and S is determined by the preset channel stride, The number of output channel data is determined based on S and K. The N pieces of input channel data include a preset number of padding data. The preset number of padding data are copied from the N input channel data and are the same as a portion of the plurality of output data calculated by the layer before the specific layer. Among the N pieces of input channel data, pieces of input channel data corresponding to indexes from 0 to P-1 and pieces of input channel data corresponding to indexes from NP to N-1 are pieces of padding data. The plurality of input channel data corresponding to the indexes from 0 to P-1 are the same as the plurality of input channel data corresponding to the indexes from N-2P to NP-1, and the plurality of input channel data corresponding to the indexes from NP to N-1 are the same as the plurality of input channel data corresponding to the indexes from P to 2P-1. The N pieces of input channel data are based on image data processed by the electronic device.

2. The method according to claim 1, wherein: The K pieces of first input channel data include a plurality of input channel data corresponding to indexes from 0 to K-1 respectively; and The K pieces of second input channel data include a plurality of input channel data corresponding to indexes from S to S+K-1, respectively.

3. The method according to claim 1, wherein The convolution operation is performed in a convolutional neural network (CNN).

4. An electronic device for performing a convolution operation at a specific layer of a neural network, the electronic device comprising: a memory storing one or more instructions; as well as a processor configured to execute the one or more instructions stored in the memory to perform the following operations: Get N input channel data; performing a first convolution operation in the neural network by applying a first input channel data group including K first input channel data among the N input channel data to a first kernel filter group including K first kernel filters; performing a second convolution operation in the neural network by applying a second input channel data group including K second input channel data among the N input channel data to a second kernel filter group including K second kernel filters; and Obtaining output channel data based on the first convolution operation and the second convolution operation, Where N is a natural number, and K is a natural number less than N, in: The N input channel data have indexes from 0 to N-1; The first input channel data group includes the K pieces of first input channel data corresponding to indexes from 0 to K-1; The second input channel data group includes the K pieces of second input channel data corresponding to indexes from S to S+K-1; and S is determined by the preset channel stride, The number of output channel data is determined based on S and K. The N pieces of input channel data include a preset number of padding data. The preset number of padding data are copied from the N input channel data and are the same as a portion of the plurality of output data calculated by the layer before the specific layer. Among the N pieces of input channel data, pieces of input channel data corresponding to indexes from 0 to P-1 and pieces of input channel data corresponding to indexes from NP to N-1 are pieces of padding data. The plurality of input channel data corresponding to the indexes from 0 to P-1 are the same as the plurality of input channel data corresponding to the indexes from N-2P to NP-1, and the plurality of input channel data corresponding to the indexes from NP to N-1 are the same as the plurality of input channel data corresponding to the indexes from P to 2P-1. The N pieces of input channel data are based on image data processed by the electronic device.

5. The electronic device according to claim 4, wherein: The K pieces of first input channel data include a plurality of input channel data corresponding to indexes from 0 to K-1 respectively; and The K pieces of second input channel data include a plurality of input channel data corresponding to indexes from S to S+K-1, respectively.

6. A non-transitory computer-readable recording medium storing a program for executing, on an electronic device, a method for performing a convolution operation at a specific layer in a neural network, the method comprising: Get N input channel data; performing, by the electronic device, a first convolution operation in the neural network by applying a first input channel data group including K first input channel data among the N input channel data to a first kernel filter group including K first kernel filters; performing, by the electronic device, a second convolution operation in the neural network by applying a second input channel data group including K second input channel data among the N input channel data to a second kernel filter group including K second kernel filters; and Obtaining output channel data based on the first convolution operation and the second convolution operation, Where N is a natural number, and K is a natural number less than N, in: The N input channel data have indexes from 0 to N-1; The first input channel data group includes the K pieces of first input channel data corresponding to indexes from 0 to K-1; The second input channel data group includes the K pieces of second input channel data corresponding to indexes from S to S+K-1; and S is determined by the preset channel stride, The number of output channel data is determined based on S and K. The N pieces of input channel data include a preset number of padding data. The preset number of padding data are copied from the N input channel data and are the same as a portion of the plurality of output data calculated by the layer before the specific layer. Among the N pieces of input channel data, pieces of input channel data corresponding to indexes from 0 to P-1 and pieces of input channel data corresponding to indexes from NP to N-1 are pieces of padding data. The plurality of input channel data corresponding to the indexes from 0 to P-1 are the same as the plurality of input channel data corresponding to the indexes from N-2P to NP-1, and the plurality of input channel data corresponding to the indexes from NP to N-1 are the same as the plurality of input channel data corresponding to the indexes from P to 2P-1. The N pieces of input channel data are based on image data processed by the electronic device.

Images