Neural network device

Abstract

A neural network device is disclosed, comprising: a first memory for storing, for each layer of at least one layer in a CNN, activation values ​​of each IFM of a predetermined number of input channels of the layer; a pipeline structure for selecting activation values ​​from the IFM of each of the predetermined number of input channels of the layer, the pipeline structure converting the selected activation values ​​to a Winograd domain on the fly; a second memory for storing elements of a Winograd filter of each of the predetermined number of input channels of the layer; a multiplier for multiplying the converted activation values ​​in the Winograd domain of the layer with selected elements from the Winograd filter to form corresponding product values ​​in the Winograd domain, each selected element in the Winograd filter corresponding to an activation value in the Winograd domain; an adder tree for summing the respective product values ​​in the Winograd domain of the layer to form elements of a feature map in the Winograd domain; and an inverse Winograd transformer for performing an inverse Winograd transformation on the elements of the feature map in the Winograd domain on the fly to form an OFM of the layer.

Description

[0001] This application claims the benefit of priority to U.S. Provisional Application No. 62 / 992,899, filed on March 20, 2020, the disclosure of which is incorporated herein by reference in its entirety. Technical Field

[0002] The subject matter disclosed herein relates to neural network devices. Background Art

[0003] A neural network device (e.g., a neural network chip) is a device used to execute a neural network. Neural network devices typically perform neural network operations using adders and multipliers built into them, but their performance and speed remain to be improved. Summary of the Invention

[0004] Example embodiments provide a neural network device, the neural network device comprising: a first memory storing, for each layer of at least one layer in a convolutional neural network (CNN), an activation value of an IFM for each of a predetermined number of input channels of the layer, each IFM having a first predetermined size; a pipeline structure selecting an activation value from the IFM in each of the predetermined number of input channels of the layer, the pipeline structure converting the selected activation value into a Winograd domain on the fly; a second memory storing an element of a Winograd filter for each of the predetermined number of input channels of the layer, each Winograd filter corresponding to a kernel filter having the first predetermined size; a multiplication a multiplier for multiplying the converted activation values ​​in the Winograd domain of the layer by the selected elements from the Winograd filter to form corresponding product values ​​in the Winograd domain, each selected element in the Winograd filter corresponding to the activation value in the Winograd domain; an adder tree for summing the respective product values ​​in the Winograd domain of the layer to form an element of a feature map in the Winograd domain; and an inverse Winograd transformer for performing an inverse Winograd transformation on the elements of the feature map in the Winograd domain in real time to form an output feature map (OFM) of the layer, the OFM having a second predetermined size.

[0005] An example embodiment provides a method for image recognition, the method comprising: obtaining an input image for generating an input feature map (IFM) of a convolutional neural network (CNN); and executing the CNN based on the input image to obtain an image recognition result, wherein the step of obtaining the image recognition result comprises: for each layer of at least one layer in the CNN, selecting an activation value in the IFM of each of a predetermined number of input channels of the layer, each IFM having a first predetermined size, converting the selected activation value to a Winograd domain on the fly, selecting an element in a Winograd filter in each of the predetermined number of input channels, each Winograd filter corresponding to a kernel filter having the first predetermined size, each selected element in the Winograd filter corresponding to an activation value in the Winograd domain, and comparing the converted activation value in the Winograd domain with the Winograd domain. The method comprises the steps of: multiplying corresponding elements of the IFM in the Winograd filter to form corresponding product values ​​in the Winograd domain, repeatedly selecting activation values ​​from the IFM in each of a predetermined number of input channels, converting the selected activation values ​​to the Winograd domain on the fly, selecting an element in the Winograd filter corresponding to the current converted activation value in the Winograd domain, and multiplying the current converted activation value with the corresponding element to form corresponding product values ​​in the Winograd domain until all activation values ​​of the IFM have been converted and multiplied with the corresponding elements, summing the respective product values ​​in the Winograd domain to form elements of a feature map in the Winograd domain, and performing an inverse Winograd transform on the elements of the feature map in the Winograd domain to form an OFM of the layer, the OFM having a second predetermined size.

[0006] The first layer of CNN can obtain the input image as the IFM of the first layer, and the OFM of the previous layer of CNN can be the IFM of the next layer of CNN.

[0007] The position of the selected activation value in the IFM can be the same position as the position of the selected activation value in other IFMs from which other selected activation values ​​are selected; and wherein the position of the selected element in the Winograd filter from which the element is selected can be the same position as the position of another selected element in another Winograd filter from which another element is selected.

[0008] The following steps may be performed in parallel for a predetermined number of input channels: repeatedly selecting an activation value from the IFM in each of the predetermined number of input channels, converting the selected activation value to a Winograd domain on the fly, selecting an element in the Winograd filter corresponding to the current converted activation value in the Winograd domain, and multiplying the current converted activation value by the corresponding element to form a corresponding product value in the Winograd domain.

[0009] The first predetermined size of the IFM and the first predetermined size of the Winograd filter may include a 4×4 array, and the second predetermined size of the OFM may include a 2×2 array.

[0010] The predetermined number of input channels may be sixteen.

[0011] Each Winograd filter of the first predetermined size may correspond to one of a 3×3 weight kernel, a 1×3 weight kernel, and a 3×1 weight kernel.

[0012] Each IFM of the first predetermined size may include a third predetermined number of rows and a fourth predetermined number of columns, the third predetermined number and the fourth predetermined number being greater than 1, and wherein the step of converting the selected activation value to the Winograd domain on the fly may include: selecting two first activation values ​​in a row of the IFM; converting the two selected first activation values ​​into a first intermediate value; selecting two second activation values, each second activation value being selected from a column from which the first activation value has been selected; converting the selected two second activation values ​​into a second intermediate value; and the first intermediate values ​​and the second intermediate values ​​may be combined to convert the selected activation value to the Winograd domain on the fly.

[0013] The step of converting the selected activation values ​​to the Winograd domain on the fly can be performed in a pipelined manner.

[0014] An example embodiment provides a method for training a convolutional neural network (CNN) for image recognition, the method comprising: executing the CNN based on an input image as training data, generating output data from the CNN; updating parameters of the CNN based on the generated output data; and training the CNN by repeatedly performing the generating step and the updating step, wherein the generating step comprises: for each of at least one layer in the CNN, selecting an activation value in an input feature map (IFM) in each of a predetermined number of input channels of the layer, each IFM having a first predetermined size, converting the selected activation value to a Winograd domain on the fly, selecting an element in a Winograd filter in each of the predetermined number of input channels, each Winograd filter having a first predetermined size, and each selected element in the Winograd filter corresponding to the converted activation value in the Winograd domain, The converted activation values ​​in the Winograd domain are multiplied by corresponding elements to form corresponding product values ​​in the Winograd domain, activation values ​​are repeatedly selected from the IFM in each of a predetermined number of input channels, the selected activation values ​​are converted to the Winograd domain on the fly, an element in the Winograd filter corresponding to the current converted activation value in the Winograd domain is selected, and the current converted activation value in the Winograd domain is multiplied by the corresponding element to form the corresponding product value in the Winograd domain until all activation values ​​in the IFM have been converted and multiplied by the corresponding elements, the respective product values ​​in the Winograd domain are summed to form elements of a feature map in the Winograd domain, and the feature map in the Winograd domain is inverse Winograd transformed on the fly to form the OFM of the layer, the OFM having a second predetermined size.

[0015] The position of the selected activation value in the IFM can be the same position as the position of another selected activation value in another IFM from which the other selected activation value is selected, and wherein the position of the selected element in the Winograd filter from which the element is selected can be the same position as the position of the other selected element in the other Winograd filter from which the other element is selected.

[0016] The following steps may be performed in parallel for a predetermined number of input channels: repeatedly selecting activation values, transforming the selected activation values, selecting elements in a Winograd filter, and multiplying the transformed activation values ​​by the corresponding elements.

[0017] The IFM of the first predetermined size and the Winograd filter of the first predetermined size may include a 4×4 array, and the OFM of the second predetermined size may include a 2×2 array.

[0018] The predetermined number of input channels may be sixteen.

[0019] Each Winograd filter of the first predetermined size may correspond to one of a 3×3 weight kernel, a 1×3 weight kernel, and a 3×1 weight kernel.

[0020] Each IFM of the first predetermined size may include a third predetermined number of rows and a fourth predetermined number of columns, the third predetermined number and the fourth predetermined number being greater than 1, and wherein the step of converting the selected activation value to the Winograd domain on the fly may include: selecting two first activation values ​​in a row of the IFM; converting the two selected first activation values ​​into a first intermediate value; selecting two second activation values, each second activation value being selected from the column from which the first activation value has been selected; converting the selected two second activation values ​​into a second intermediate value; and combining the first intermediate value and the second intermediate value to convert the selected activation value to the Winograd domain on the fly.

[0021] The step of converting the selected activation values ​​to the Winograd domain on the fly can be performed in a pipelined manner. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In the following sections, aspects of the subject matter disclosed herein will be described with reference to exemplary embodiments shown in the accompanying drawings, in which:

[0023] Figure 1A A block diagram depicting an example embodiment of the architecture of a CNN that may be used to convolve activation values ​​and weight values ​​according to the subject matter disclosed herein;

[0024] Figure 1B depicts the four positions of a 3×3 weight kernel when forming four elements of an example OFM across corresponding elements of an example IFM;

[0025] Figure 1C depicts the positions of the Winograd filters used to form four elements of an example OFM relative to the corresponding elements of an example IFM;

[0026] Figure 1D Describes operations performed on elements (such as activation values) of a 4×4 matrix to convert the elements to a Winograd domain based on equations (1) to (7);

[0027] Figure 2A Description based on Figure 1A A functional block diagram of an example embodiment of a system that can be used to multiply activation element values ​​of a set of IFMs (or lanes) by corresponding weight value elements of the CNN architecture depicted in FIG.

[0028] Figure 2B Description based on Figure 1A Another functional block diagram of an example embodiment of a system operable to multiply activation element values ​​of an IFM with corresponding weight value elements of the CNN architecture depicted in FIG.

[0029] Figure 3A A functional block diagram depicting an example embodiment of a system for converting activation values ​​to a Winograd domain on the fly so that activation elements can be convolved with elements of a Winograd filter in accordance with the subject matter disclosed herein;

[0030] Figure 3B depicting details of a row during a first cycle in accordance with the subject matter disclosed herein;

[0031] Figure 3C Depicts the second cycle according to the subject matter disclosed herein Figure 3B hot row;

[0032] Figure 3D is a table showing additional details of the operation of hot row 301a in accordance with the subject matter disclosed herein;

[0033] Figure 4 A functional block diagram depicting an example embodiment of an inverse Winograd transformer according to the subject matter disclosed herein, which can transform, for example, Figure 2A The output of the adder tree in [1] is converted from the Winograd domain back to the original domain of activation values ​​on the fly; and

[0034] Figure 5 Depicting an electronic device including a CNN that converts activation values ​​of an IFM to a Winograd domain on-the-fly according to the subject matter disclosed herein. DETAILED DESCRIPTION

[0035] In the following detailed description, numerous specific details are set forth in order to provide a thorough understanding of the disclosure. However, those skilled in the art will appreciate that the disclosed aspects may be practiced without these specific details. In other instances, well-known methods, processes, components, and circuits have not been described in detail in order not to obscure the subject matter disclosed herein.

[0036] References throughout this specification to "one embodiment" or "an embodiment" mean that a particular feature, structure, or characteristic described in connection with the embodiment may be included in at least one embodiment disclosed herein. Thus, the appearances of the phrases "in one embodiment" or "in an embodiment" or "according to an embodiment" (or other phrases of similar meaning) in different places throughout this specification may not necessarily all refer to the same embodiment. Furthermore, in one or more embodiments, the particular features, structures, or characteristics may be combined in any suitable manner. In this regard, as used herein, the word "exemplary" means "serving as an example, instance, or illustration." Any embodiment described herein as "exemplary" should not be construed as necessarily preferred or advantageous over other embodiments. Furthermore, depending on the context of the discussion herein, singular terms may include corresponding plural forms, and plural terms may include corresponding singular forms. Similarly, hyphenated terms (e.g., "two-dimensional," "predetermined," "specific-pixel," etc.) may occasionally be used interchangeably with corresponding non-hyphenated versions (e.g., "two-dimensional," "predetermined," "specific pixel," etc.), and capitalized terms (e.g., "Counter Clock," "Row Select," "PIXOUT," etc.) may be used interchangeably with corresponding non-capitalized versions (e.g., "counter clock," "row select," "pixout," etc.). Such occasional interchangeable usage should not be considered inconsistent with each other.

[0037] It should also be noted that the various figures (including component diagrams) shown and discussed herein are for illustrative purposes only and are not drawn to scale. Similarly, the various waveforms and timing diagrams are shown for illustrative purposes only. For example, the dimensions of some elements may be exaggerated relative to other elements for clarity. Furthermore, where deemed appropriate, reference numerals have been repeated in the drawings to indicate corresponding and / or similar elements.

[0038] The terms used herein are for the purpose of describing some example embodiments only and are not intended to limit the subject matter of the claims. Unless the context clearly indicates otherwise, the singular form as used herein is also intended to include the plural form. It will also be understood that the terms "including" and / or "comprising" when used in this specification indicate the presence of the stated features, wholes, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or groups thereof. Unless clearly defined in this way, the terms "first", "second", etc. as used herein are used as labels for the nouns following them and do not indicate any type of order (e.g., spatial, temporal, logical, etc.). In addition, the same reference numerals may be used between two or more figures to represent parts, components, blocks, circuits, units or modules with the same or similar functions. However, such usage is only for simplicity of description and ease of discussion and does not mean that the construction or structural details of such components or units are the same between all embodiments or that such commonly referenced components / modules are the only way to implement some example embodiments of the example embodiments disclosed herein.

[0039] It will be understood that when an element or layer is referred to as being on, "connected to," or "coupled to" another element or layer, it can be directly on, directly connected to, or directly coupled to the other element or layer, or intervening elements or layers may be present. In contrast, when an element or layer is referred to as being "directly on," "directly connected to," or "directly coupled to" another element or layer, there are no intervening elements or layers. Like reference numerals refer to like elements throughout. As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items.

[0040] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the subject matter belongs. It will also be understood that, unless expressly defined otherwise herein, terms (such as those defined in general dictionaries) should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and will not be interpreted in an idealized or overly formal manner.

[0041] The subject matter disclosed herein allows activation values ​​to be converted to the Winograd domain on-the-fly (or real-time), so that the reduced number of multiplication operations performed when convolving activation values ​​in the Winograd domain with weight values ​​can be used to increase the performance speed of CNNs and reduce the power consumption of CNNs.

[0042] Figure 1AA block diagram depicts an example embodiment of an architecture 100 of a CNN that can be used to convolve activation values ​​and weight values ​​according to the subject matter disclosed herein. The architecture 100 includes a plurality of weight kernels (or filters) 102 and a plurality of IFMs 101 that generate a plurality of OFMs 103 when convolved. In one example, a first layer of a CNN can receive an input image as an IFM for the first layer, and the OFM of a previous layer of the CNN can be the IFM of a subsequent layer of the CNN.

[0043] like Figure 1A As depicted in , the plurality of IFMs 101 may include 64 IFMs, wherein each IFM may be an IFM of 112×112 elements. Each IFM 101 may be referred to as a lane or as a channel. The elements of an IFM may also be referred to as activations or as activation values. The plurality of weight kernels 102 may include 128 kernel filters, wherein each kernel filter may include 64 3×3 weight kernels. The plurality of OFMs 103 may include 128 OFMs, wherein each OFM may be an OFM of 112×112 elements. The number of IFMs, weight kernels, and OFMs may be the same as Figure 1A In addition, other IFM and filter sizes can be used to obtain OFMs of different sizes. Figure 1A In FIG, D may indicate the number of input channels, H may indicate the height of the IFM or OFM, and W may indicate the width of the IFM or OFM.

[0044] The process for generating OFM 103 from IFM 101 and weight kernels 102 may be as follows. A first group 102a of weight kernels 102, which may include 64 3x3 weight kernels, may be selected, and each individual weight kernel of the first group is associated with a corresponding 3x3 patch of IFM 101. For example, Figure 1A As depicted in FIG, each individual 3×3 weight kernel in the 64-member group 102a is associated with the top-left 3×3 block of the corresponding IFM 101 of the 64 IFMs. The elements of each 3×3 weight kernel are element-wise multiplied with the elements of the corresponding 3×3 IFM block to form nine product value elements in 64 intermediate feature maps (not shown). The product value elements at the same corresponding position in each of the 64 intermediate feature maps are summed to form a single 1×1 block (or element) 103a1 indicated at the top-left position of the first OFM 103a.

[0045] The first set of weight kernels 102a can then be shifted (or strided) horizontally to the right by one element across the IFM 101, and the first set of weight filters 102a can be associated with another 3×3 block of the corresponding IFM 101. The elements of each 3×3 weight kernel are element-wise multiplied with the elements of the corresponding 3×3 IFM block of the new set of IFM blocks to form nine product value elements in 64 intermediate feature maps (again, not shown). The product value elements in the same corresponding positions in each of the 64 intermediate feature maps are summed to form another single 1×1 block (element) of the OFM 103, which will be located horizontally immediately to the right of block 103a1. The first set of weight filters 102a can be shifted (or strided) horizontally and vertically across the IFM 101 based on the horizontal and vertical strides to form all of the respective elements of the first OFM 103a.

[0046] A second set 102b of weight kernels is selected, and each individual weight kernel of the second set is associated with a corresponding 3×3 block of IFM 101. Figure 1A As will be depicted in FIG, each individual 3×3 weight kernel in the 64-member group 102b is associated with the top-left 3×3 block of the corresponding IFM 101 of the 64 IFMs. The elements of each 3×3 weight kernel are element-wise multiplied with the elements of the corresponding 3×3 IFM block to form nine product value elements in 64 intermediate feature maps (not shown). The product value elements at the same corresponding position in each of the 64 intermediate feature maps are summed to form a single 1×1 block (or element) at the top-left position of the second OFM 103b.

[0047] As previously described, the second set 102b of weight kernels may be strided horizontally and vertically across the IFM 101 to form elements of corresponding blocks in the second OFM 103b.128 sets of weight filters are selected and applied to the IFM 101 to form elements of the 128 OFMs.

[0048] Figure 1B Depicted are four positions of a 3x3 weight kernel 112 as it spans the elements of a corresponding example IFM 110 to form four elements of an example OFM 113. The weight kernel 112, depicted by gray shading, can be moved horizontally and vertically to cover a 4x4 block 111 of the IFM 110. Each individual portion of the block 111 is indicated by cross-hatching in a different direction. The cross-hatching of the elements in the OFM 113 indicates the portion of the 4x4 block in the IFM 110 from which the OFM element is formed. Each element in the OFM 113 is formed by nine multiplication operations and eight addition operations. Figure 1BFor the four OFM elements indicated in FIG, a total of 36 multiplication operations and 32 addition operations are performed. The relative complexity (and power consumption) of each multiplication operation is significantly greater than the complexity of the addition operation, so the overall complexity of forming a single element of OFM 113 can be well approximated by referring only to the number of multiplication operations used to form the element. Therefore, 36 multiplication operations will be referred to herein as needing to be performed to form Figure 1B The four elements of OFM 113 are indicated in FIG.

[0049] Figure 1C The Winograd filter 121 used to form the four elements of the example OFM 122 is depicted relative to the corresponding elements of the example IFM 120. The elements in IFM 120 have been converted to the Winograd domain. The 4×4 elements of Winograd filter 121 have been converted to the Winograd domain from a 3×3 weight filter (not shown). Because both IFM 120 and Winograd filter 121 are in the Winograd domain, the elements of OFM 122′ (not shown, which may indicate the intermediate result of element-by-element multiplication of the elements of the Winograd filter with the elements of the corresponding IFM in the Winograd domain) are also in the Winograd domain.

[0050] Figure 1C The four elements indicated in OFM 122 correspond to Figure 1B 113 in OFM 122. To form the four elements in OFM 122, element-wise multiplications are performed between the elements of IFM 120 and the elements of Winograd filter 121, resulting in a total of 16 multiplication operations, which is a significant reduction in multiplication operations compared to the 36 multiplication operations performed when the activation values ​​and weight values ​​are not in the Winograd domain. The number of multiplication operations performed when the elements are in the Winograd domain compares very favorably to the number of multiplication operations required when the elements are not in the Winograd domain.

[0051] The elements of the Winograd filter 121 can be formed offline and therefore require no additional overhead during inference operations of an architecture that can be used to convolve activation values ​​and weight values, such as architecture 100. The conversion of the elements of the IFM to the Winograd domain and the conversion of the resulting OFM are addition-based operations and can be performed on the fly as described herein.

[0052] The following equation provides a mathematical explanation for converting the activation values ​​in a 4×4 block of IFM to the Winograd domain. In the equation, A is the matrix used to convert the 4×4 block of IFM to the Winograd domain, and A Tis the transpose of matrix A, act is the activation value, w is the weight value, and i is the index.

[0053]

[0054]

[0055]

[0056] A0=[1 0] (7)

[0057] Figure 1D Depicts the operations performed on elements of a 4×4 matrix (such as activation values) to convert the elements to the Winograd domain based on equations (1) to (7). More specifically, an example 4×4 matrix (or block of an IFM) containing elements a to p is Figure 1D , which is shown at 150. The example matrix is ​​partially converted to the Winograd domain at 151 and fully converted to the Winograd domain at 152. In equations (2) to (5), IFMs may indicate image feature maps.

[0058] The conversion of a row of elements to a Winograd field can be performed as follows. The value in the first column of the row is obtained by subtracting the value in the third column of the row from the value in the first column of the row. The value in the second column of the row is obtained by adding the value in the second column of the row to the value in the third column. The value in the third column of the row is obtained by subtracting the value in the second column from the value in the third column of the row. The value in the fourth column of the row is obtained by subtracting the value in the fourth column from the value in the second column of the row. This sequence can be observed with reference to the row of elements highlighted at 150 to the row of elements highlighted at 151.

[0059] The conversion continues as follows by operating on the columns of the partially converted matrix 151. The value in the first row of the column is obtained by subtracting the value in the third row of the column from the value in the first row of the column. The value in the second row of the column is obtained by adding the value in the second row of the column to the value in the third row. The value in the third row of the column is obtained by subtracting the value in the second row from the value in the third row of the column. The value in the fourth row of the column is obtained by subtracting the value in the fourth row from the value in the second row of the column. This sequence can be observed by referring to the first column element of the partially converted matrix 151 through the highlighted column element at 152.

[0060] Because weight values ​​typically do not change when a CNN operates in inference mode, they can be converted to the Winograd domain offline and stored after the CNN has been trained. However, activation values ​​do change in inference mode, and converting activation values ​​to the Winograd domain in inference mode may include additional accesses to the activation values ​​in memory, which is not energy-efficient. Furthermore, converting activation values ​​to the Winograd domain and storing the values ​​used during inference mode operation may adversely affect the performance speed of the CNN.

[0061] Although a 3×3 weight kernel has been used as an example, other weight kernels that can be used with the techniques disclosed herein include a 3×1 weight kernel and a 1×3 weight kernel. For the 1×3 weight kernel and the 3×1 weight kernel, a 1×4 block and a 4×1 block of IFM are used, respectively. Compared to the case of a 3×3 weight kernel where the activation transform is applied to a row-by-row transform followed by a column-by-column transform, the transform for the 1×4 IFM block is simply a row-by-row transform, and the transform for the 4×1 IFM block is simply a column-by-column transform.

[0062] Figure 2A Description based on Figure 1A 1 is a functional block diagram of an example embodiment of a system (or neural network device) 200 for multiplying activation element values ​​of a group of IFMs (or lanes) by corresponding weight value elements of the CNN architecture 100 depicted in FIG. The system 200 can be used when both the activation values ​​and the weight values ​​are in the same domain. The system 200 can be arranged to process a group of, for example, 16 IFMs and can include an activation value buffer 201, a weight value buffer 202, a multiplier 203, an adder tree 204, and a return unit 205. The activation value buffer (or cache) 201 and the weight value buffer (or cache) 202 can be organized so that the activation elements and corresponding weight values ​​of a block of IFMs can be accessed in parallel. Figure 2A The block depicted in can be, for example, a 4×4 block. Therefore, the weight filter can also be a 4×4 weight filter. Other block and filter sizes can be used. Figure 2A The terminology used in

[15] is: activation values ​​are indicated by "Act", where the subscript corresponds to the channel or number of channels and the superscript corresponds to the position in the block. Weight values ​​are indicated by "W" with the same subscript and superscript convention.

[0063] In one example, system 200 may perform the following steps in parallel for multiple input channels: repeatedly selecting activation values ​​from the IFM in each input channel, converting the selected activation values ​​to the Winograd domain on the fly, selecting an element in the Winograd filter corresponding to the current converted activation value in the Winograd domain, and multiplying the current converted activation value with the corresponding element to form a corresponding product value in the Winograd domain until all activation values ​​of the IFM have been converted and multiplied with the corresponding elements. Multiplier 203 multiplies the activation values ​​(e.g., the converted activation values ​​in the Winograd domain) with the corresponding weight values ​​in the Winograd filter. Adder tree 204 adds the resulting products output by multiplier 203. The output of adder tree 204 is the input to return unit 205, which can convert the output of adder tree 204 from the Winograd domain to the original domain of the activation values. The return unit 205 may also buffer the output of the adder tree 204 so that the output can be added to the output of other adders (not shown) to form an element of the OFM before being returned to a memory (not shown) for subsequent processing at another layer of the CNN. In this manner, the number of multiplication operations required to execute the CNN is reduced, thereby improving the performance speed of the system 200 in executing the CNN and reducing the power consumption of the system 200. In one example, the system 200 may include a controller (not shown) that receives an input image and, based on the output of the final layer of the CNN, outputs an image recognition result or updates the parameters of the CNN. In one embodiment, the system 200 may execute the CNN based on the input image as training data and generate output data from the CNN. The system 200 may calculate the output data based on the input image using a Winograd filter. The system 200 may update the parameters of the CNN based on the generated output data and train the CNN by repeatedly performing the generating and updating steps. In one example, the training of the CNN is terminated when the number of repetitions reaches a predetermined number or when the difference between the CNN output data and the target data reaches a predetermined accuracy.

[0064] Figure 2B Description based on Figure 1A Another functional block diagram of an example embodiment of a system 200 operable to multiply an activation element value of an IFM with a corresponding weight value element of the CNN architecture 100 depicted in FIG. Figure 2B As depicted in FIG. 2 , the system 200 includes n groups 2100 to 210 n-1 , where each group processes 16 channels of the IFM of the architecture 100 to produce the OFM 103. For example, if a group includes 16 channels, and the architecture 100 includes a total of 64 channels, then Figure 2B The n in will be equal to 4. Figure 2BRectangle 211 in represents a functional component that generates one element of OFM 103 .

[0065] As previously mentioned, converting weight values ​​to the Winograd domain can be convenient because the conversion process can be performed offline after the CNN has been trained. However, on the other hand, when the CNN is operating in inference mode, converting activation values ​​to the Winograd domain can have an adverse effect on the performance speed of the CNN. The subject matter disclosed herein allows for the on-the-fly conversion of activation values ​​to the Winograd domain, so that the reduced number of multiplication operations performed when convolving activation values ​​in the Winograd domain with weight values ​​can be used to improve both the performance speed of the CNN and the reduced power consumption of the CNN.

[0066] Figure 3A A functional block diagram depicting an example embodiment of a system 300 for converting activation values ​​to a Winograd domain on the fly (e.g., in a pipelined manner) so that activation elements can be convolved with elements of a Winograd filter in accordance with the subject matter disclosed herein. System 300 is similar to Figure 2B The system 200 in FIG. 1 is shown, but includes a hot row (or pipeline structure) indicated at 301. Figure 1D As shown in , the immediate Winograd transformation of the activation values ​​of a 3×3 weight kernel consists of two stages of one-dimensional Winograd transformations—one stage for each row and one stage for each column. The hot rows serve as intermediate registers to store intermediate results during the two-stage transformation. Figure 3A In the example, the system 300 may include n groups 3100 to 310 n-1 .

[0067] Figure 3B Depicted is detail of row 301a of thermal row 301 during a first cycle according to the subject matter disclosed herein. Figure 3C Hot row 301a is depicted during the second cycle. Hot row 301a may include a first processing element (PE) 302 and a second PE 303. PE 302 may include an adder 304, a first multiplexer 305, and a second multiplexer 306 connected as shown. There may be two registers 307 and 308 connected between PE 302 and PE 303 as shown to hold intermediate results output from PE 302. PE 303 may include an adder 309 connected as shown.

[0068] Reference Figure 1D The matrix 152 and Figure 3D As shown in Table 3, in the first cycle ( Figure 3B), elements a and c are selected and input to adder 304. Adder 304 outputs ac, which is stored in register 307 and input to adder 309. In one example, in the second cycle ( Figure 3C ), an element is selected from the column from which elements a and c have been selected, for example, elements i and k are selected and input to adder 304. Adder 304 outputs ik, which is stored in register 308 and input to adder 309. Adder 309 outputs (ac)-(ik) as the upper leftmost element in matrix 152.

[0069] Table 3 shows additional details of the operation of hot row 301a. The rows labeled "time" in Table 3 indicate the time in cycles. The rows labeled "cache control" in Table 3 indicate which activation values ​​are selected in the activation value cache 201, and the rows labeled "hot row control" indicate how the hot row is controlled to output the desired activation value in the Winograd domain. Rows H1 and H2 indicate two hot rows. The rows labeled "Wino activation" are the activation values ​​output in the Winograd domain. In Table 3, T may indicate an intermediate variable, ADD may indicate an adder, and STR may indicate a step size.

[0070] The entries indicated by diagonal hatching in Table 3 are associated with the operation of hot row 301a for generating the activation value in the leftmost column of the matrix at 152, while the entries indicated by dots are associated with the operation of hot row 301a for generating the activation value in the second column from the left of the matrix at 152.

[0071] Figure 4 Depicts a functional block diagram of an example embodiment of an inverse Winograd transformer that can transform an adder tree 204 ( Figure 2A ) is converted from the Winograd domain back to the original domain of the activation value. That is, the inverse Winograd converter 400 can convert the OFM 122 ( Figure 1C ) are instantly converted from the Winograd domain back to their original domain. The inverse Winograd transformer 400 includes four adders 401 to 404 and four accumulators 405 to 408 connected as shown. The result from the adder tree 204 is as follows: Figure 4 The input X, with the sign changed as shown in , is input to each individual adder 401 to 404. The function of the inverse Winograd transformer 400 corresponds to the matrix A of equation (1) and the transpose A as shown in equation (4) T .

[0072] Figure 5An electronic device including a CNN that converts activation values ​​of an IFM into a Winograd domain on the fly, according to the subject matter disclosed herein, is depicted. Electronic device 500 may be used in, but is not limited to, a computing device, a personal digital assistant (PDA), a laptop computer, a mobile computer, a web tablet, a wireless phone, a mobile phone, a smartphone, a digital music player, or a wired or wireless electronic device. Electronic device 500 may also be, but is not limited to, part of an ADAS, a mobile device imaging system, an industrial imaging system, a robot, or the like. Electronic device 500 may include a controller 510, an input / output device 520 (such as, but not limited to, a keyboard, a keypad, a display, a touchscreen display, a camera, and / or an image sensor), a memory 530, an interface 540, a graphics processing unit (GPU) 550, and a neural network device 560, all connected to each other via a bus 570. Controller 510 may include, for example, at least one microprocessor, at least one digital signal processor, at least one microcontroller, and the like. Memory 530 may be configured to store command codes or user data to be used by controller 530.

[0073] Electronic device 500 and its various system components may include a neural network device 560 that includes a CNN that converts activation values ​​of an IFM into a Winograd domain on the fly according to the subject matter disclosed herein. Interface 540 may be configured to include a wireless interface configured to transmit or receive data to or from a wireless communication network using RF signals. Wireless interface 540 may include, for example, an antenna, a wireless transceiver, and the like. The electronic device 500 can also be used in the communication interface protocols of the communication system (such as, but not limited to, Code Division Multiple Access (CDMA), Global System for Mobile Communications (GSM), North American Digital Communications (NADC), Extended Time Division Multiple Access (E-TDMA), Wideband CDMA (WCDMA), CDMA2000, Wi-Fi, Municipal Wi-Fi (MuniWi-Fi), Bluetooth, Digital Enhanced Cordless Telecommunications (DECT), Wireless Universal Serial Bus (Wireless USB), Fast Low Latency Access Orthogonal Frequency Division Multiplexing with Seamless Handover (Fast OFDM), IEEE 802.20, General Packet Radio Service (GPRS), iBurst, Wireless Broadband (WiBro), WiMAX, WiMAX Advanced, Universal Mobile Telecommunications Service-Time Division Duplex (UMTS-TDD), High Speed ​​Packet Access (HSPA), Evolution Data Optimized (EVDO), Long Term Evolution Advanced (LTE-Advanced), Multi-channel Multipoint Distribution Service (MMDS), etc.).

[0074] The embodiments of the subject matter and operations described in this specification can be implemented in digital electronic circuits, or in computer software, firmware or hardware (including the structures disclosed in this specification and their structural equivalents), or in a combination of one or more thereof. The embodiments of the subject matter described in this specification can be implemented as one or more computer programs (i.e., one or more modules of computer program instructions) encoded on a computer storage medium to be executed by a data processing device or to control the operation of the data processing device. Alternatively or in addition, the program instructions can be encoded on an artificially generated propagation signal (e.g., a machine-generated electrical signal, optical signal, or electromagnetic signal) that is generated to encode information for transmission to a suitable receiver device for execution by the data processing device. The computer storage medium can be or be included in a computer-readable storage device, a computer-readable storage substrate, a random or serial access storage array or device, or a combination thereof. In addition, when the computer storage medium is not a propagation signal, the computer storage medium can be the source or destination of computer program instructions encoded in an artificially generated propagation signal. The computer storage medium can also be or be included in one or more separate physical components or media (e.g., multiple CDs, disks, or other storage devices). Furthermore, the operations described in this specification can be implemented as operations performed by a data processing apparatus on data stored on one or more computer-readable storage devices or received from other sources.

[0075] Although this specification may contain many specific implementation details, the implementation details should not be interpreted as limiting the scope of any claimed subject matter, but rather should be interpreted as descriptions of specific features of particular embodiments. Specific features described in this specification in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments. Furthermore, although features may be described above as functioning in a particular combination, and even initially claimed as such, in some cases, one or more features from the claimed combination may be deleted from that combination, and the claimed combination may involve sub-combinations or variations of sub-combinations.

[0076] Similarly, although operations are described in a particular order in the accompanying drawings, this should not be understood as requiring that such operations be performed in the particular order shown or in a sequential order, or that all illustrated operations be performed to obtain the desired results. In certain circumstances, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product, or packaged in multiple software products.

[0077] Thus, specific embodiments of the subject matter have been described herein. Other embodiments are within the scope of the claims. In some cases, the actions recited in the claims can be performed in a different order and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the particular order shown or sequential order to achieve the desired results. In some implementations, multitasking and parallel processing may be advantageous.

[0078] As those skilled in the art will appreciate, the innovative concepts described herein can be modified and varied in a wide range of applications. Therefore, the scope of the claimed subject matter should not be limited to any specific exemplary teachings discussed above, but rather is defined by the claims.

Claims

1. A neural network device, comprising: a first memory storing, for each layer of at least one layer in the convolutional neural network, activation values ​​of input feature maps of each of a predetermined number of input channels of the layer, each input feature map having a first predetermined size; a pipeline structure that selects an activation value from an input feature map in each of a predetermined number of input channels of the layer, and the pipeline structure converts the selected activation value into a Winograd domain on the fly; a second memory storing elements of a Winograd filter for each of a predetermined number of input channels of the layer, each Winograd filter corresponding to a kernel filter having a first predetermined size; a multiplier that multiplies the transformed activation value in the Winograd domain of the layer by a selected element from the Winograd filter to form a corresponding product value in the Winograd domain, where each selected element in the Winograd filter corresponds to an activation value in the Winograd domain; An adder tree that sums the product values ​​in the Winograd field of the layer to form an element of a feature map in the Winograd field; and An inverse Winograd transformer is configured to perform an inverse Winograd transformation on elements of the feature map in the Winograd domain to form an output feature map of the layer, wherein the output feature map has a second predetermined size. The selected activation value is converted in a first-stage conversion of the pipeline structure and a second-stage conversion of the pipeline structure, and the neural network device includes an intermediate register for storing intermediate results of the first-stage conversion and the second-stage conversion.

2. The neural network device of claim 1 , further comprising: The controller is configured as: receiving an input image; and The image recognition result is output based on the output feature map of the last layer of the convolutional neural network or the parameters of the convolutional neural network are updated.

3. The neural network device according to claim 2, wherein: The first layer of the convolutional neural network obtains the input image as the input feature map of the first layer, and the output feature map of the previous layer of the convolutional neural network is the input feature map of the next layer of the convolutional neural network.

4. The neural network device according to claim 1, wherein: The position of the selected activation value in the input feature map is the same position as the position of the selected activation values ​​in other input feature maps from which other selected activation values ​​are selected; and Here, the position of the selected element in the Winograd filter from which the element is selected is the same position as the position of another selected element in another Winograd filter from which another element is selected.

5. The neural network device according to claim 1, wherein: The input feature map of the first predetermined size and the Winograd filter of the first predetermined size include a 4×4 array, and the output feature map of the second predetermined size includes a 2×2 array.

6. The neural network device according to claim 1, wherein: The predetermined number of input channels is sixteen.

7. The neural network device according to claim 1, wherein: Each Winograd filter of the first predetermined size corresponds to one of a 3×3 weight kernel, a 1×3 weight kernel, and a 3×1 weight kernel.

8. The neural network device according to any one of claims 1 to 7, wherein: Each input feature map of the first predetermined size includes a third predetermined number of rows and a fourth predetermined number of columns, the third predetermined number and the fourth predetermined number being greater than 1, and Among them, the pipeline structure also: Select the first two activation values ​​in a row of the input feature map; converting the two selected first activation values ​​into first intermediate values; selecting two second activation values, each second activation value being selected from the column from which the first activation value was selected; converting the selected two second activation values ​​into second intermediate values; and The first intermediate value and the second intermediate value are combined to convert the selected activation value to the Winograd domain on the fly.

Images