Method and apparatus for identifying an image

Abstract

A method and apparatus for recognizing an image are provided. The method of recognizing an image includes obtaining an input image, calculating input activations within a neural network based on the input image, generating a bit vector for each of at least one layer within the neural network based on whether each of a plurality of input activations of the layer within the neural network is 0, merging the bit vector into the plurality of input activations such that a bit value included in the bit vector is a most significant bit of a multi-bit representation of the plurality of input activations, merging the bit vector into a plurality of weights such that the bit value included in the bit vector is a most significant bit of a multi-bit representation of the plurality of weights, sorting the plurality of input activations and the plurality of weights based on bits corresponding to the most significant bit, performing an operation between the sorted plurality of input activations and the sorted plurality of weights, and outputting an image recognition result based on a result of the operation.

Description

[0001] This application claims the benefit of Korean Patent Application No. 10-2019-0128715, filed on October 16, 2019, with the Korean Intellectual Property Office, the entire disclosure of which is incorporated herein by reference. Technical Field

[0002] This disclosure relates to pattern recognition, and more specifically, to methods and apparatus for recognizing images. Background Technology

[0003] Image recognition can be achieved using neural networks. Neural networks can be computational systems implemented with reference to a computational architecture.

[0004] Typical neural network devices for image recognition can require a large amount of computation for complex input images. Furthermore, typical neural network devices for image recognition are not efficient at processing computation, and therefore cannot efficiently analyze input images and extract information in real time. Thus, because typical neural network devices for image recognition can require a large amount of computation and / or process computation inefficiently, their image recognition performance may be inefficient when implemented in low-power, high-performance embedded systems (such as smartphones with limited resources). Summary of the Invention

[0005] A method and apparatus for performing image recognition in a neural network are provided. A computer-readable recording medium is provided containing a program for performing the method in a computer.

[0006] The technical problems to be solved are not limited to those mentioned above, and other technical problems may exist.

[0007] Other aspects will be set forth in part in the description which follows, and in part will be apparent from the disclosed embodiments, which are clearly described or may be presented by practice.

[0008] According to one aspect of an embodiment, a method for recognizing an image includes: obtaining an input image; calculating input activations within a neural network based on the input image; generating a bit vector for each of at least one layer within the neural network based on whether each of a plurality of input activations of the layer within the neural network is 0; merging the bit vectors into the plurality of input activations such that the bit values ​​included in the bit vectors are the most significant bits of the multi-bit representations of the plurality of input activations; merging the bit vectors into a plurality of weights such that the bit values ​​included in the bit vectors are the most significant bits of the multi-bit representations of the plurality of weights; sorting the plurality of input activations and the plurality of weights based on the bits corresponding to the most significant bits; performing an operation between the sorted plurality of input activations and the sorted plurality of weights; and outputting an image recognition result based on the result of the operation.

[0009] The step of calculating input activations within a neural network based on an input image may include: the i-th layer of the neural network outputs the output activation of the (i-1)-th layer, which serves as the input activation of the i-th layer, where i is an integer greater than 1. The first layer of the neural network receives the input image and outputs its output activation as the input activation of the second layer based on the input image. The step of generating a bit vector may include: generating a first bit value in response to an input activation among the plurality of input activations being non-zero; and generating a second bit value in response to another input activation among the plurality of input activations being zero.

[0010] The first value can be 1, and the second value can be 0.

[0011] The step of merging bit vectors into the plurality of input activations may include: merging the first bit value into the input activation as the most significant bit of the merged input activation, and merging the second bit value into the additional input activation as the most significant bit of the merged input activation.

[0012] The sorting steps may include: sorting the plurality of input activations in the order of a first group and a second group, the first group including input activations whose first value is merged into the most significant bit, and the second group including input activations whose second value is merged into the most significant bit; and sorting the plurality of weights in the order of a third group and a fourth group, the third group including weights whose first value is merged into the most significant bit, and the fourth group including weights whose second value is merged into the most significant bit.

[0013] In response to the total number of input activations belonging to the second group being less than a preset value, the step of performing the operation may include: performing an operation between the nth input activation among the sorted plurality of input activations and the nth weight among the sorted plurality of weights, where n is a positive integer.

[0014] In response to the total number of input activations in the second group being less than a preset value, the step of performing the operation may include: performing an operation between each of the sorted plurality of input activations and the corresponding weight among the sorted plurality of weights.

[0015] In response to the total number of input activations belonging to the second group being greater than or equal to a preset value, the steps of performing the operation may include: performing an operation between input activations belonging to the first group and weights belonging to the third group; and skipping one or more operations between input activations belonging to the second group and weights belonging to the fourth group.

[0016] The preset value may be the total number of input activations configured to be processed simultaneously by the processor, and the step of skipping one or more operations may include: skipping the operation between the number of input activations corresponding to the preset value from the input activations belonging to the second group and the number of weights corresponding to the preset value from the weights belonging to the fourth group.

[0017] The steps for generating a bit vector may include: generating a first vector by the following steps: generating a first bit value of the first vector in response to an input activation among the plurality of input activations being non-zero, and generating a second bit value of the first vector in response to another input activation among the plurality of input activations being zero; generating a second vector by the following steps: generating a first bit value of the second vector in response to a weight among the plurality of weights being non-zero, and generating a second bit value of the second vector in response to another weight among the plurality of weights being zero; and generating the bit vector by performing an AND logical operation between the first vector and the second vector.

[0018] The step of merging bit vectors into the plurality of weights may include merging the bit values ​​of the most significant bits of the multi-bit representation of the input activations that are merged into the plurality of input activations into the most significant bits of the multi-bit representation of the weights that will be operated on using the input activations.

[0019] The sorting can be performed using any of the following algorithms: bubble sort, shell sort, bitonic sort, comb sort, cyclic sort, and odd-even merge sort.

[0020] The step of performing the operation between the sorted plurality of input activations and the sorted plurality of weights may include: generating the output activation of the current layer of the neural network, wherein the last layer of the neural network outputs an image recognition result based on the result of the operation between the output activation of the previous layer and the corresponding weights.

[0021] According to one aspect of another embodiment, an apparatus for recognizing an image includes: a processor configured to: acquire an input image; compute input activations within a neural network based on the input image; generate a bit vector for each of at least one layer within the neural network based on whether each of a plurality of input activations of the layer within the neural network is 0; merge the bit vectors into the plurality of input activations such that the bit values ​​included in the bit vectors are the most significant bits of the multi-bit representation of the plurality of input activations; merge the bit vectors into a plurality of weights such that the bit values ​​included in the bit vectors are the most significant bits of the multi-bit representation of the plurality of weights; sort the plurality of input activations and the plurality of weights based on the bits corresponding to the most significant bits; perform an operation between the sorted plurality of input activations and the sorted plurality of weights; and output an image recognition result based on the result of the operation.

[0022] To compute input activations within a neural network based on an input image, the processor can be configured to: output the output activation of the i-th layer of the neural network based on the output activation of the (i-1)-th layer, which serves as the input activation of the i-th layer, where i is an integer greater than 1. The first layer of the neural network receives the input image and outputs its output activation as the input activation of the second layer. To generate a bit vector, the processor can be configured to: generate a first bit value in response to one of the input activations being non-zero; and generate a second bit value in response to another input activation being zero.

[0023] For sorting, the processor can be configured to: sort the plurality of input activations in the order of a first group and a second group, the first group including input activations whose first value is merged into the most significant bit, the second group including input activations whose second value is merged into the most significant bit; and sort the plurality of weights in the order of a third group and a fourth group, the third group including weights whose first value is merged into the most significant bit, the fourth group including weights whose second value is merged into the most significant bit.

[0024] In response to the total number of input activations belonging to the second group being less than a preset value, the processor may be configured to perform an operation between the nth input activation among the sorted plurality of input activations and the nth weight among the sorted plurality of weights, where n is a positive integer.

[0025] In order to perform the operation, in response to the total number of input activations belonging to the second group being greater than or equal to a preset value, the processor can be configured to: perform the operation between the input activations belonging to the first group and the weights belonging to the third group; and skip one or more operations between the input activations belonging to the second group and the weights belonging to the fourth group.

[0026] The preset value can be the total number of input activations configured to be processed by the processor simultaneously, and in order to skip the one or more operations, the processor can be configured to skip the operation between the number of input activations corresponding to the preset value from the input activations belonging to the second group and the number of weights corresponding to the preset value from the weights belonging to the fourth group.

[0027] To generate a bit vector, the processor can be configured to: generate a first vector by the following steps: generating a first bit value of the first vector in response to an input activation among the plurality of input activations being non-zero, and generating a second bit value of the first vector in response to another input activation among the plurality of input activations being zero; generate a second vector by the following steps: generating a first bit value of the second vector in response to a weight among the plurality of weights being non-zero, and generating a second bit value of the second vector in response to another weight among the plurality of weights being zero; and generate the bit vector by performing an AND logical operation between the first vector and the second vector.

[0028] In order to merge bit vectors into the plurality of weights, the processor may be configured to merge "the most significant bit of the multi-bit representation of the input activation that is merged into the plurality of input activations" into "the most significant bit of the multi-bit representation of the weight that will be operated on with the input activation among the plurality of weights".

[0029] For sorting purposes, the processor can be configured to sort the plurality of input activations and the plurality of weights according to any one of the following algorithms: bubble sort, shell sort, bitonic sort, comb sort, cyclic sort, and odd-even merge sort.

[0030] The device may further include: a memory storing instructions that, when executed by a processor, configure the processor to perform bit vector generation, bit vector to merging of the plurality of input activations, bit vector to merging of the plurality of weights, sorting of the plurality of input activations and the plurality of weights, and performing the operations between the sorted plurality of input activations and the sorted plurality of weights.

[0031] According to one aspect of another embodiment, a method for recognizing an image includes: obtaining an input image; calculating input activations within a neural network based on the input image; storing, for each layer in the neural network, a plurality of input activations and corresponding weights of said layer in a memory; obtaining the plurality of input activations and the plurality of weights from the memory; sorting the plurality of input activations in an order of input activations having non-zero values ​​and input activations having zero values; sorting the plurality of weights in an order of weights having non-zero values ​​and weights having zero values; performing operations between the sorted plurality of input activations and the sorted plurality of weights by skipping one or more operations between input activations having zero values ​​and weights having zero values; and outputting an image recognition result based on the result of the operations.

[0032] According to one aspect of another embodiment, a method for performing an operation between a plurality of input activations and a plurality of weights in a neural network includes: generating a bit vector based on whether each of the plurality of input activations is 0; merging the bit vector into the plurality of input activations to allow bit values ​​included in the bit vector to be the most significant bit (MSB) of a multi-bit representation corresponding to each of the plurality of input activations; merging the bit vector into the plurality of weights to allow bit values ​​included in the bit vector to be the MSB of a multi-bit representation corresponding to each of the plurality of weights; sorting the plurality of input activations and the plurality of weights based on the bits corresponding to the MSBs; and performing an operation between the sorted plurality of input activations and the sorted plurality of weights.

[0033] The steps of generating a bit vector may include: generating a first bit value for the input activations among the plurality of input activations whose values ​​are not 0; and generating a second bit value for the input activations among the plurality of input activations whose values ​​are 0.

[0034] The sorting steps may include: sorting the plurality of input activations in the order of a first group and a second group, the first group including input activations whose first value is merged into the MSB, and the second group including input activations whose second value is merged into the MSB; and sorting the plurality of weights in the order of a third group and a fourth group, the third group including weights whose first value is merged into the MSB, and the fourth group including weights whose second value is merged into the MSB.

[0035] When the number of input activations belonging to the second group is less than a preset value, the step of performing the operation may include: performing an operation between the nth input activation among the sorted plurality of input activations and the nth weight among the sorted plurality of weights, where n is a positive integer.

[0036] When the number of input activations belonging to the second group is a preset value or greater, the steps of performing the operation may include: performing an operation between the input activations belonging to the first group and the weights belonging to the third group; and skipping at least some of the operations between the input activations belonging to the second group and the weights belonging to the fourth group.

[0037] The preset value can correspond to the total number of input activations that can be processed simultaneously, and the operation between the number of input activations corresponding to the preset value from the input activations belonging to the second group and the number of weights corresponding to the preset value from the weights belonging to the fourth group can be skipped.

[0038] The steps of generating a bit vector may include: generating a first vector by generating a first value for input activations with a non-zero value among the plurality of input activations and generating a second bit value for input activations with a zero value among the plurality of input activations; generating a second vector by generating a first value for weights with a non-zero value among the plurality of weights and generating a second bit value for weights with a zero value among the plurality of weights; and generating a bit vector by performing an AND logical operation between the first vector and the second vector.

[0039] The step of sorting the weights may include: merging the bit values ​​in the MSB of the multi-bit representation corresponding to the first input activation among the plurality of input activations into the MSB of the multi-bit representation corresponding to the first weight among the plurality of weights that will be operated on with the first input activation.

[0040] The sorting steps can be performed by choosing one of the following algorithms: bubble sort, shell sort, bitonic sort, comb sort, cyclic sort, or odd-even merge sort.

[0041] According to one aspect of another embodiment, an apparatus for implementing a neural network includes: a memory, and a processor configured to perform a convolution operation between a plurality of input activations and a plurality of weights, wherein the processor generates a bit vector based on whether each of the plurality of input activations is 0; merges the bit vector into the plurality of input activations to allow bit values ​​included in the bit vector to be the most significant bit (MSB) of a multi-bit representation corresponding to each of the plurality of input activations; merges the bit vector into the plurality of weights to allow bit values ​​included in the bit vector to be the MSB of a multi-bit representation corresponding to each of the plurality of weights; sorts the plurality of input activations and the plurality of weights based on bits corresponding to MSBs; and performs an operation between the sorted plurality of input activations and the sorted plurality of weights.

[0042] The processor can generate a first value for input activations that are not zero among the plurality of input activations, and generate a second value for input activations that are zero among the plurality of input activations.

[0043] The processor can sort the plurality of input activations in the order of a first group and a second group, the first group including input activations whose first value is merged into the MSB, the second group including input activations whose second value is merged into the MSB; and sort the plurality of weights in the order of a third group and a fourth group, the third group including weights whose first value is merged into the MSB, and the fourth group including weights whose second value is merged into the MSB.

[0044] When the number of input activations belonging to the second group is less than a preset value, the processor can perform the operation between the nth input activation among the sorted input activations and the nth weight among the sorted weights, where n is a positive integer.

[0045] When the number of input activations belonging to the second group is a preset value or greater, the processor may perform the operation between the input activations belonging to the first group and the weights belonging to the third group, and skip at least some of the operations between the input activations belonging to the second group and the weights belonging to the fourth group.

[0046] The preset value may correspond to the total number of input activations that can be processed simultaneously. The processor may skip the operation between the number of input activations corresponding to the preset value from the input activations belonging to the second group and the number of weights corresponding to the preset value from the weights belonging to the fourth group.

[0047] The processor generates a first vector by generating a first value for input activations with non-zero values ​​among the plurality of input activations and generating a second bit value for input activations with zero values ​​among the plurality of input activations; generates a second vector by generating a first value for weights with non-zero values ​​among the plurality of weights and generating a second bit value for weights with zero values ​​among the plurality of weights; and generates a bit vector by performing an AND logical operation between the first vector and the second vector.

[0048] The processor may merge the bit values ​​in the MSB of the multi-bit representation corresponding to the first input activation among the plurality of input activations into the MSB of the multi-bit representation corresponding to the first weight among the plurality of weights that will be operated on with the first input activation.

[0049] The processor can sort the plurality of input activations and the plurality of weights according to one of the following algorithms: bubble sort, shell sort, bitonic sort, comb sort, cyclic sort, or odd-even merge sort.

[0050] According to one aspect of another embodiment, a computer-readable recording medium is provided having a program for performing the method in a computer. Attached Figure Description

[0051] The above and other aspects, features, and advantages of the disclosed specific embodiments will become more apparent from the following description taken in conjunction with the accompanying drawings, in which:

[0052] Figure 1 This illustrates a neural network architecture according to one or more embodiments;

[0053] Figure 2 This illustrates operations performed by a neural network according to one or more embodiments;

[0054] Figure 3 The configuration of a neural network device according to one or more embodiments is shown;

[0055] Figure 4 A method is illustrated for performing a convolution operation between input activations and weights in a neural network according to one or more embodiments;

[0056] Figure 5 This illustrates the generation of bit vectors according to one or more embodiments;

[0057] Figure 6 This illustrates adding a bit vector to input activation according to one or more embodiments;

[0058] Figure 7 This illustrates adding a bit vector to a weight according to one or more embodiments;

[0059] Figure 8 This illustrates the sorting of input activations and weights according to one or more embodiments;

[0060] Figure 9 This illustrates the operation between the input activations and the weights of the sorting process according to one or more embodiments;

[0061] Figure 10 This illustrates the generation of bit vectors according to one or more embodiments;

[0062] Figure 11 This illustrates the sorting of input activations and weights according to one or more embodiments;

[0063] Figure 12 An electronic system according to one or more embodiments is shown; and

[0064] Figure 13 The configuration of a neural network device according to one or more embodiments is shown. Detailed Implementation

[0065] The following detailed embodiments are provided to aid the reader in gaining a comprehensive understanding of the methods, apparatus, and / or systems described herein. However, various changes, modifications, and equivalents of the methods, apparatus, and / or systems described herein will become apparent upon understanding this disclosure. For example, the order of operations described herein is merely illustrative and is not limited to those orders set forth herein, but may be changed as will become clear upon understanding this disclosure, except for operations that must occur in a specific order. Furthermore, for clarity and conciseness, descriptions of features known in the art may be omitted.

[0066] Reference will now be made in detail to the embodiments, examples of which are shown in the accompanying drawings, wherein the same reference numerals always denote the same elements. In this respect, one or more embodiments may have different forms and should not be construed as limited to the description set forth herein. Therefore, the embodiments are described below only by reference to the accompanying drawings to explain aspects. As used herein, the term “and / or” includes any and all combinations of one or more of the associated listed items. Expressions such as “at least one of…” modify the entire column of elements, rather than a single element within the column, when following a list of elements.

[0067] Unless otherwise defined, all terms used herein (including technical and scientific terms) shall have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains, upon understanding the disclosure of this application. Unless expressly defined herein, terms (such as those defined in general dictionaries) shall be interpreted as having a meaning consistent with their meaning in the context of the relevant field and in the disclosure of this application, and shall not be interpreted in an idealized or overly formalistic manner.

[0068] Throughout this specification, when an element is referred to as being "connected to" another element, the element may be directly connected to the other element or electrically connected to the other element, with an intermediate element provided in between. Furthermore, unless otherwise stated, the term "comprising" an element does not exclude other elements, but rather includes the element itself.

[0069] The terminology used herein is for the purpose of describing various examples only and is not intended to limit disclosure. Unless the context clearly indicates otherwise, the singular form is intended to include the plural form as well. The terms “comprising,” “including,” and “having” indicate the presence of the stated features, quantities, operations, components, elements, and / or combinations thereof, but do not preclude the presence or addition of one or more other features, quantities, operations, components, elements, and / or combinations thereof.

[0070] In the following description, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. However, exemplary embodiments of the present disclosure may be implemented in many alternative forms and should not be construed as being limited to the exemplary embodiments of the present disclosure set forth herein.

[0071] Figure 1 A neural network architecture according to one or more embodiments is shown.

[0072] Figure 1 The neural network 1 shown can be a deep neural network (hereinafter referred to as "DNN") architecture. The DNN can be a convolutional neural network (CNN), a recurrent neural network (RNN), a deep belief network, a restricted Boltzmann machine, but is not limited to these. The neural network 1 can be a DNN comprising an input layer (layer 1), four hidden layers (layers 2, 3, 4, and 5), and an output layer (layer 6). For example, when the neural network 1 is a convolutional neural network (CNN), the multiple layers (layers 1 through 6) can correspond to some layers of the CNN (such as convolutional layers, pooling layers, and fully connected layers).

[0073] Each layer in neural network 1 may include multiple artificial nodes, also referred to as "neurons," "processing elements (PEs)," or similar terms. While nodes may be called "artificial nodes" or "neurons," such designations are not intended to assign any relation to how the neural network architecture computationally maps or thereby intuitively identifies information, or how human neurons operate. That is, the terms "artificial node" or "neuron" are merely technical terms referring to the nodes in the hardware implementation of the neural network. Figure 1 As shown, for example, the input layer (layer 1) may include five nodes, and the hidden layer (layer 2) may include seven nodes. However, this is just an example, and each layer included in neural network 1 may include a variety of numbers of nodes.

[0074] The nodes included in each layer of the neural network 1 can connect to each other to exchange data. For example, a node can receive data from other nodes, perform operations on the data, and output the results to other nodes.

[0075] A neural network 1 implemented using a DNN architecture can include multiple layers that process valid information. Therefore, neural network 1 can process more complex datasets than a neural network consisting of a single layer. Although Figure 1 The diagram shows neural network 1 consisting of six layers, but this is merely an example, and neural network 1 may include fewer or more layers. That is, neural network 1 may include... Figure 1 The diagram shows the structure of various layers with different structures.

[0076] Figure 2The diagram illustrates operations performed by a neural network for image recognition according to one or more embodiments.

[0077] Therefore, as Figure 2 As shown, neural network 2 may have a structure including an input layer, a hidden layer, and an output layer. It can perform operations based on received input data (e.g., I1 and I2) and generate output data (e.g., O1 and O2) based on the results of the operations. Specifically, neural network 2 can receive the input image to be recognized as input data via the input layer, then perform operations on the input image at each layer, and finally output the recognition result as output data via the output layer based on the results of the operations of the previous layer. For example, but not limited to, the category / label, location, and region of the object identified from the image.

[0078] Neural network 2 can be an n-layer neural network or a DNN with two or more hidden layers, as described above. For example, such as Figure 2 As shown, neural network 2 can be a DNN comprising an input layer (layer 1), two hidden layers (layers 2 and 3), and an output layer (layer 4). Because neural network 2, implemented with a DNN architecture, includes more layers capable of processing effective information, neural network 2 can process more complex datasets than a neural network with a single layer. Neural network 2 may include fewer or more layers or fewer or more channels. That is, neural network 2 may include... Figure 2 The diagram shows the structure of various layers with different structures.

[0079] Neural network 2 can be configured to perform image recognition, as a non-limiting example, by mapping input and output data to each other in a non-linear relationship based on deep learning. As a non-limiting example, such deep learning refers to a machine learning scheme implemented by a processor for solving problems (such as problems related to automated image recognition) on large datasets. Deep learning can be implemented by mapping input and output data via supervised or unsupervised learning or training, such that when trained, the resulting machine learning model, engine, or example NN can intuitively map further input data to output data with the desired accuracy or reliability. Here, it is noted that the term "may" (e.g., what an example or embodiment may include or implement) is used with respect to examples or embodiments to indicate that there exists at least one example or embodiment that includes or implements such a feature, while all examples and embodiments are not limited thereto.

[0080] Each layer included in neural network 2 may include multiple channels. A channel may correspond to multiple artificial nodes, also known as neurons, PEs, units, or similar terms. For example, as... Figure 2As shown, the input layer (layer 1) may include two channels (nodes), and each of the hidden layers (layers 2 and 3) may include three channels. However, this is just an example, and each layer included in neural network 2 may include a variety of numbers of channels (nodes).

[0081] The channels included in each layer of the neural network 2 can be connected to each other to process data. For example, one channel can receive data from other channels, perform operations on the data, and output the results to other channels.

[0082] The output value of a channel can be referred to as the activation, or the value produced by such a predetermined activation function of the corresponding channel. The input and output of each channel can be referred to as the input activation and output activation, respectively. That is, activation can be a parameter corresponding to the output of a channel and the input of another channel included in the next layer, due to one or more connections to the next layer. Each channel can determine its own activation based on the activation received as a result from the channels included in the previous layer, along with weights and biases. Weights can be parameters used to compute the output activation in each channel, and can be values ​​assigned to the connections between channels. For example, the output of a channel from a previous layer can be provided as the input to a channel in the next or subsequent layer through a weighted connection between the channel in the previous layer and the channel in the next layer, the weights of which are adjusted in various ways during the training of the neural network until the neural network is trained for the desired target. Additional connections to the channels in the next layer may exist, such as providing bias connection values ​​through weighted or unweighted connections and / or providing weighted recurrent connections as in the examples above. During training and implementation, such connections and connection weights can be selectively implemented, removed, and altered to generate or obtain a resulting neural network, which is then trained and can be implemented accordingly for the training objective (such as identifying any of the objectives in the examples above).

[0083] Therefore, return to Figure 1 Each channel can be processed by a computational unit or processing element that receives input (e.g., via such a weighted connection) and outputs an activated input, and the input-output of each channel can be mapped. The computational unit can correspond to the activation function of the channel. As a non-limiting example, when σ is the activation function, It represents the weights from the k-th channel included in layer (i-1) to the j-th channel included in layer i. It is the bias of the j-th channel included in the i-th layer, and When the j-th channel of the i-th layer is activated, activation The following equation 1 can be used to calculate it.

[0084] Equation 1:

[0085]

[0086] like Figure 2 As shown, the activation of the first channel CH1 in the second layer (layer 2) can be achieved by... This indicates that... Furthermore, according to Equation 1, It can have The value of . However, Equation 1 above is merely an example used to describe the activations, weights, and biases used to process data in the neural network 2, and the embodiments are not limited thereto. Activation can be a value obtained by processing a weighted sum of activations received from the previous layer using an activation function (such as a sigmoid function or a rectified linear unit (ReLU) function).

[0087] In a typical neural network that implements the structure of neural network 2 as described above, many datasets can be exchanged between multiple interconnected channels and undergo numerous computational processes while passing through layers. Therefore, neural networks of one or more embodiments can reduce the loss of accuracy while advantageously reducing the amount of computation required to process complex input data, thereby increasing processing speed and / or reducing the processing power of the processor-implemented device that can implement neural networks of one or more embodiments.

[0088] Figure 3 A neural network apparatus according to one or more embodiments is shown.

[0089] The neural network device 300 can be a neural network (e.g., Figure 1 Neural network 1 or Figure 2 The neural network device 300 is implemented as or using various types of devices (such as personal computers (PCs), server devices, mobile devices, and embedded devices). For example, the neural network device 300 may be implemented as or using various devices (such as smartphones, tablet devices, augmented reality (AR) devices, Internet of Things (IoT) devices, autonomous vehicles, robotic devices, and medical devices, but not limited thereto), which use one or more layers of convolutional interactions of nodes discussed herein to form a neural network and / or any or any combination of neural network layers to perform image recognition, etc. Furthermore, the neural network device 300 may be a hardware (HW) accelerator dedicated to implementing or driving the aforementioned devices or a hardware accelerator dedicated to implementing or driving neural networks (such as neural processing units (NPUs), tensor processing units (TPUs), and neural engines, but not limited thereto).

[0090] Reference Figure 3 The neural network device 300 may include a memory 310 and a processor 320. Those skilled in the art will understand, upon understanding this disclosure, that, in addition to... Figure 3 In addition to the components shown, the neural network device 300 may also include general-purpose components.

[0091] Memory 310 may store data processed and to be processed by neural network device 300. For example, memory 310 may store input activation data, weight data, etc., processed in the layers of the neural network. Memory 310 may be, but is not limited to, random access memory (RAM) (such as dynamic random access memory (DRAM) and static random access memory (SRAM)), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), CD-ROM, Blu-ray disc or other optical disc storage device, hard disk drive (HDD), solid-state drive (SSD), or flash memory.

[0092] Reference Figure 3 The memory 310 may include an input activation memory 311 and a weight memory 312. The input activation memory 311 and the weight memory 312 may store input activation data and weight data, respectively. The activation memory 311 may be physically separated from the weight memory 312, but the embodiments are not limited thereto. For example, Figure 3 An exemplary memory 310 is shown storing input activation data and weight data, but the activation memory 311 and the weight memory 312 may share a physically connected memory 310.

[0093] Processor 320 can read / write data (e.g., input activation data or weight data) processed in neural network device 300 from memory 310, and run neural network device 300 using the read / written data. For example, processor 320 can perform operations between input activation data and weight data.

[0094] Additionally, processor 320 may include multiplier 321 and adder 322.

[0095] Multiplier 321 may include an array of multiple multipliers. For example, multiplier 321 may include k multipliers. Multiplication operations between input activations and weights can be performed in parallel in the k multipliers. For example, in multiplier 321, point multiplication operations between k input activations and k weights corresponding to the input activations can be performed. Therefore, k can be the basic unit of operation performed by processor 320.

[0096] Adder 322 can receive from multiplier 321 the results of multiplication operations between k input activations and k weights corresponding to the input activations. Adder 322 can sum all the results of the k multiplication operations to output the result.

[0097] The processor 320 may be a central processing unit (CPU), graphics processing unit (GPU), application processor (AP), etc., included in the neural network device 300, but is not limited to these.

[0098] When at least one of the input activations and weights is 0, the result of their multiplication is 0, so this does not affect the outcome. Therefore, when at least one of the input activations and weights is 0, the operation between the input activations and weights can be skipped (e.g., it is determined that it will not be performed and / or not performed).

[0099] As described above, the processor 320 can perform operations between k input activations and k weights corresponding to each input activation as a basic unit. Therefore, when the processor 320 is able to group the k input activations with a value of 0 from the input activation data received from the memory 310, the operations between the k input activations with a value of 0 and the k weights corresponding to each of the k input activations with a value of 0 can be skipped. Thus, by skipping such operations, the processor 320 of one or more embodiments can reduce the amount of computation required to process complex input data, thereby advantageously increasing processing speed and / or reducing the processing power of the processor 320 capable of performing the operations.

[0100] Therefore, processor 320 can group input activations with a value of 0 before processing the input activation data received from memory 310 in multiplier 321, and determine whether to skip the operation of the grouped input activations. In the following, reference will be made to... (The sentence is incomplete and requires further context to be fully translated.) Figures 4 to 13 The embodiments are described in further detail.

[0101] Figure 4 This paper illustrates a method for performing a convolution operation between input activations and weights in a neural network according to one or more embodiments.

[0102] Reference Figure 4 In operation 410, the neural network device 300 can generate a bit vector based on whether each input activation is 0. As a non-limiting example, reference will be made to... Figure 5 Further description of generating bit vectors.

[0103] Figure 5 This illustrates the generation of bit vectors according to one or more embodiments.

[0104] First, the processor 320 can obtain k+n input activations 500 and k+n weights 510 corresponding to the k+n input activations 500 from the memory 310.

[0105] k+n inputs activate 500, which can include input activation a1 to input activation a2. k+nEach input activation can be represented by multiple bits. For example, each input activation can be represented by 4 bits, but is not limited to this, and can be represented by various numbers of bits. Similarly, k+n weights can contain weights w1 to w... k+n Each weight can be represented by multiple bits. For example, each weight can be represented by 4 bits, but it is not limited to this.

[0106] The processor 320 can generate a first bit value for each of the k+n input activations 500 that is not zero, and a second bit value for each of the k+n input activations 500 that is zero. The first bit value can be 1, and the second bit value can be 0, and is not limited thereto. For example, when input activation a1 is not zero, a first bit value of 1 can be generated to correspond to input activation a1. When input activation a2 is zero, a second bit value of 0 can be generated to correspond to input activation a2. In this way, the bit vector 520 can be generated based on whether each of the k+n input activations 500 is zero. Therefore, the bit vector 520 can include k+n bit values ​​corresponding to the k+n input activations 500 respectively.

[0107] Return to reference Figure 4 In operation 420, the neural network device 300 may incorporate bit vector 520 into input activation 500 such that each bit value included in bit vector 520 will be the most significant bit (MSB) of the multi-bit expression corresponding to each input activation. As a non-limiting example, reference will be made to... Figure 6 Further description of incorporating bit vector 520 into the input activation.

[0108] Figure 6 This illustrates merging bit vectors into input activation according to one or more embodiments.

[0109] Reference Figure 6 The processor 320 can merge each bit value of the bit vector 520 into each input activation corresponding to each bit value among the k+n input activations 500. For example, the first bit value 1 generated for input activation a1 can be merged into input activation a1, and the second bit value 0 generated for input activation a2 can be merged into input activation a2. In this respect, the first bit value 1 added to input activation a1 can be the MSB of the multi-bit representation corresponding to input activation a1, and the second bit value 0 added to input activation a2 can also be the MSB of the multi-bit representation corresponding to input activation a2. Therefore, each input activation among the k+n input activations 600 generated by adding bit vector 520 can be represented by 5 bits.

[0110] Return to reference Figure 4In operation 430, the neural network device 300 may incorporate the bit vector 520 into the weights, such that each bit value included in the bit vector 520 will be the MSB of a multi-bit representation corresponding to each weight. As a non-limiting example, reference will be made to... Figure 7 Further description of merging bit vector 520 into the weights.

[0111] Figure 7 This illustrates merging bit vectors into weights according to one or more embodiments.

[0112] Reference Figure 7 The processor 320 can merge the bit vector 520 generated based on k+n input activations 500 into k+n weights 510. The bit values ​​"merged into the MSB of the multi-bit representation corresponding to the first input activation among the multiple input activations" can be merged into the MSB of the multi-bit representation corresponding to the first weight among the multiple weights that will be operated on using the first input activation.

[0113] For example, the first bit value 1 generated for input activation a1 can be merged into the weight w1 to which the operation is performed using input activation a1, and the second bit value 0 generated for input activation a2 can be merged into the weight w2 to which the operation is performed using input activation a2. Here, the first bit value 1 merged into input activation a1 can be the MSB of a multi-bit representation corresponding to weight w1, and the second bit value 0 merged into input activation a2 can also be the MSB of a multi-bit representation corresponding to weight w2. Therefore, each weight in the k+n weights 610 generated by merging bit vector 520 into k+n weights 510 can be represented by 5 bits.

[0114] Return to reference Figure 4 In operation 440, the neural network device 300 can sort the input activations and weights based on the bits corresponding to the MSB. As a non-limiting example, reference will be made to... Figure 8 The process of sorting input activations and weights is further described.

[0115] Figure 8 This illustrates the sorting of input activations and weights according to one or more embodiments.

[0116] Reference Figure 8 The processor 320 can apply the sorting algorithm to the k+n input activations 600 generated by merging bit vectors 520 into k+n input activations 500 and the k+n weights 610 generated by merging bit vectors 520 into k+n weights 510. In this case, the processor 320 can sort the k+n input activations 600 and the k+n weights 610 based on the bits corresponding to the MSB.

[0117] For example, the processor 320 can sort the k+n input activations 600 generated by merging bit vector 520 into k+n input activations 500 in the order of a first group and a second group, the first group including input activations whose first bit value 1 is merged into the MSB, and the second group including input activations whose second bit value 0 is merged into the MSB.

[0118] In a similar manner, the processor 320 can sort the k+n weights 610 generated by merging bit vectors 520 into k+n weights 510 in the order of the third group and the fourth group, with the third group including weights whose first bit value is 1 merged into the MSB and the fourth group including weights whose second bit value is 0 merged into the MSB.

[0119] For example, input activations that include the first value 1 merged into the MSB may include input activation a1, input activation a3, and input activation a4, which constitute the first group. k+n .

[0120] Additionally, the input activations that include the second bit value 0 merged into the MSB may include input activation a2 and input activation a, which constitute the second group. k+n-1 .

[0121] In a similar manner, the weights, including the first value 1 merged into the MSB, may include the weights w1, w3, and w that constitute the third group. k+n Additionally, the weights including the second bit value 0 merged into the MSB may include the weights w2 and w that constitute the fourth group. k+n-1 It should be understood that within each group, the order of input activations in that group is the same as the order of their corresponding weights in that group. Therefore, in one embodiment, the original order relationship between input activations or weights remains unchanged within each group. For example, if the original order relationship between input activation a1 and input activation a3 is that input activation a1 comes first and input activation a3 comes last, then in the first group, input activation a1 will still come first and input activation a3 will still come last, but the invention is not limited to this.

[0122] As described above, the same bit vector can be merged into each input activation and weight, and the same sorting algorithm can be applied here. Therefore, the nth input activation in the sorted input activations 800 and the nth weight in the sorted weights 810 can be a pair for the operation to be performed by the dot product (where n is a positive integer).

[0123] Furthermore, the sorting algorithm can be, but is not limited to, bubble sort, shell sort, bitonic sort, comb sort, cyclic sort, or odd-even merge sort. The sorting algorithm can be odd-even merge sort. Merge sort is a sorting method that involves dividing the array into individual numbers and merging these numbers. The odd-even merge sort algorithm sorts items by comparing odd and even positions.

[0124] Return to reference Figure 4 In operation 450, the neural network device 300 can perform operations between sorted input activations and sorted weights. As a non-limiting example, reference will be made to... Figure 9 The operation between the input activation and the weights of the sorting is further described.

[0125] Figure 9 This illustrates the operation between the input activation and the sorting weights for performing sorting according to one or more embodiments.

[0126] When the number of input activations belonging to the second group among the sorted input activations 800 is less than a preset value, the processor 320 may perform operations between the nth input activation among the sorted input activations 800 and the nth weight among the sorted weights 810 (where n is a positive integer). The preset value may correspond to the total number of input activations that can be operated on simultaneously (e.g., in parallel). For example, the preset value may be k when the dot product operation between k input activations and k weights corresponding to the input activations can be performed simultaneously in the multiplier 321.

[0127] Reference Figure 9 The result of the operation between the first input activation a1 in the sorted input activations 800 and the first weight w1 in the sorted weights 810 is S1. Similarly, the result of the operation between the second input activation a3 in the sorted input activations 800 and the second weight w3 in the sorted weights 810 is S3. Then, the processor 320 can sum all the results obtained from the operation between the sorted input activations 800 and the sorted weights 810 to obtain the result S.

[0128] exist Figure 9 In the process, when the result of each operation between the input activations belonging to the second group in the sorted input activations 800 and the weights belonging to the fourth group in the sorted weights 810 is 0, the result does not affect the result S. However, when the number of input activations belonging to the second group is less than w (e.g., row size), the operation between the input activations belonging to the second group and the weights belonging to the fourth group in the sorted weights 810 is performed and not skipped.

[0129] When the number of input activations belonging to the second group among the sorted input activations 800 is greater than or equal to a preset value, the processor 320 may execute the operation between the input activations belonging to the first group and the weights belonging to the third group, and skip (e.g., determine that it will not be executed and / or will not be executed) one or more operations between the input activations belonging to the second group and the weights belonging to the fourth group.

[0130] For example, processor 320 may skip operations between a number of input activations belonging to the second group that correspond to (or are equal to) a preset value and a number of weights belonging to the fourth group that correspond to (or are equal to) a preset value. For example, neural network device 300 may skip operations corresponding to a preset value between operations between input activations belonging to the second group and weights belonging to the fourth group.

[0131] Reference Figure 9 When the number of input activations belonging to the second group among the sorted input activations 800 is k, the operation between the input activations belonging to the second group and the weights belonging to the fourth group can be skipped. Therefore, the results of all operations between the input activations belonging to the first group and the weights belonging to the third group are summed to obtain the result S.

[0132] Also refer to Figure 9 When the number of input activations belonging to the second group among the sorted input activations 800 is greater than k, the operation between the k input activations belonging to the second group and the k weights belonging to the fourth group can be skipped.

[0133] In another embodiment, when the total number of sorted input activations 800 is less than or equal to a preset value, the processor 320 may perform operations between the nth input activation among the sorted input activations 800 and the nth weight among the sorted weights 810 (where n is a positive integer). The preset value may correspond to the total number of input activations that can be operated on simultaneously (e.g., in parallel).

[0134] When the total number of sorted input activations 800 is greater than a preset value, the processor 320 may execute operations between input activations belonging to the first group and weights belonging to the third group, and skip (e.g., determine that it will not execute and / or will not execute) one or more operations between input activations belonging to the second group and weights belonging to the fourth group (e.g., operations between input activations belonging to the second group and weights belonging to the fourth group that are equal to the difference between the total number of sorted input activations 800 and the preset value may be skipped). For example, the number of input activations belonging to the first group is less than or equal to the preset value.

[0135] Figure 10 This illustrates the generation of bit vectors according to one or more embodiments.

[0136] As referenced above Figures 4 to 9 The neural network device 300 can generate a bit vector based on whether each input activation is 0, and perform operations between sorted input activations and sorted weights accordingly. In this regard, when the number of input activations with a value of 0 among the sorted input activations is less than a preset value, one or more operations between the input activations belonging to the second group and the weights belonging to the fourth group are not skipped.

[0137] When either the weight or the input activation is 0, the result of the multiplication operation between the input activation and the weight is 0, which does not affect the final result. Therefore, the bit vector can be generated based on whether at least one of the input activation and the weight is 0.

[0138] For example, as referenced above Figure 5 The processor 320 can generate a first value for each of the k+n input activations 500 whose value is not 0, and generate a second value for each of the k+n input activations 500 whose value is 0. The first value can be 1, and the second value can be 0, and is not limited thereto. For example, when input activation a1 is not 0, the first value 1 can be generated to correspond to input activation a1. When input activation a2 is 0, the second value can be generated to correspond to input activation a2. In this way, the first vector 1000 can be generated based on whether each of the k+n input activations 500 is 0.

[0139] In a similar manner, such as Figure 10 As shown, the processor 320 can generate a first value for the weights among the k+n weights 510 whose values ​​are not 0, and a second value for the weights among the k+n weights 510 whose values ​​are 0. For example, when weight w1 is 0, the second value 0 can be generated to correspond to weight w1. When weight w2 is not 0, the first value 1 can be generated to correspond to weight w2. In this way, the second vector 1010 can be generated based on whether each of the k+n weights 510 is 0.

[0140] Then, the processor 320 can generate the bit vector 1020 by performing an AND logical operation between the first vector 1000 and the second vector 1010. Thus, in one example, the bit value of the bit vector 1020 (corresponding to the input activation and weight) can be generated as 0 when either the input activation or the weight has a value of zero, and the bit value of the bit vector 1020 can be generated as 1 when both the input activation and the weight have a value of one.

[0141] The bit vector 1020 generated as described above may include more bit values ​​corresponding to 0 than the bit vector generated based on whether each input activation is 0.

[0142] Figure 11 This illustrates the sorting of input activations and weights according to one or more embodiments.

[0143] Reference Figure 11 The nth input activation in input activation 500 and the nth weight in weight 510 can form a pair (where n is a positive integer).

[0144] In this respect, the processor 320 can merge each bit value of the bit vector 1020 into a corresponding pair in one of the k+n pairs of input activations 500 and weights 510. For example, the generated second bit value 0 corresponding to input activation a1 and weight w1 can be merged into a pair of input activation a1 and weight w1. The second bit value 0 merged into the pair of input activation a1 and weight w1 can be the MSB of the multi-bit representation of that pair. Therefore, each of the k+n pairs 1100 generated by merging the bit vector 1020 into the k+n pairs of input activations 500 and weights 510 can be represented by 9 bits.

[0145] Then, the processor 320 can apply a sorting algorithm to each of the k+n pairs 1100 generated by merging the bit vector 1020 into k+n pairs of input activations 500 and k+n weights 510. The neural network device 300 can sort the k+n pairs 1100 generated by merging the bit vector 1020 into k+n pairs of input activations 500 and k+n weights 510 based on the bits corresponding to the MSB. For example, the neural network device 300 can sort the k+n pairs 1100 in the order of a first group and a second group, the first group including pairs where the first bit value is merged into the MSB, and the second group including pairs where the second bit value is merged into the MSB.

[0146] When the number of pairs belonging to the second group in the sorted pairs 1110 is less than a preset value, the neural network device 300 can perform operations between the input activations and weights in the sorted k+n pairs 1110.

[0147] Optionally, when the number of pairs belonging to the second group among the sorted pairs 1110 is greater than or equal to a preset value, the neural network device 300 may skip at least some of the operations included in the operations between the input activations and weights of the pairs belonging to the second group. For example, the neural network device 300 may skip the operations between the input activations and weights of each pair from the pairs belonging to the second group that are included in the number of pairs corresponding to the preset value.

[0148] In another embodiment, when the total number of pairs in the sorted pairs 1110 is less than or equal to a preset value, the neural network device 300 can perform operations between input activations and weights in the sorted k+n pairs 1110.

[0149] Optionally, when the total number of pairs in the sorted pairs 1110 is greater than a preset value, the neural network device 300 may perform operations between the input activations and weights of the pairs belonging to the first group, and skip at least some of the operations included in the operations between the input activations and weights of the pairs belonging to the second group. For example, the neural network device 300 may skip the operations between the input activations and weights of each pair from the pairs belonging to the second group that are equal to the difference between k+n and the preset value. For example, the number of pairs belonging to the first group is less than or equal to the preset value.

[0150] The preset value can be the total number of inputs that can be activated simultaneously for computation.

[0151] In one embodiment, the method described above with reference to the accompanying drawings can be applied to operations of all layers of the neural network or only some layers.

[0152] Figure 12 An electronic system according to one or more embodiments is shown.

[0153] Reference Figure 12 The electronic system 1200 can extract useful information by analyzing input data in real time based on a neural network, and can determine the situation or control the components of the electronic device on which the electronic system 1200 is installed based on the extracted information. For example, the electronic system 1200 can be applied to or installed in robotic devices (such as unmanned aerial vehicles and advanced driver assistance systems (ADAS)), smart TVs, smartphones, medical devices, mobile devices, image display devices, measuring devices, IoT devices, and any other various types of electronic devices. In another example, the electronic system 1200 includes the electronic device, or the electronic system 1200 is external to the electronic device.

[0154] Electronic system 1200 may include CPU 1210, RAM 1220, neural network device 1230, memory 1240, sensor module 1250, and communication module 1260. Electronic system 1200 may also include input / output module, security module, power control device, etc. Some hardware components of electronic system 1200 may be mounted on at least one semiconductor chip. Neural network device 1230 may be the aforementioned hardware accelerator specifically designed to implement or drive a neural network, or a device including the aforementioned hardware accelerator.

[0155] CPU 1210 controls the overall operation of electronic system 1200. CPU 1210 may include one processor core (single-core) or multiple processor cores (multi-core). CPU 1210 can process or execute programs and / or data stored in memory 1240. In one embodiment, CPU 1210 can control the function of neural network device 1230 by executing programs stored in memory 1240. CPU 1210 may be implemented by CPU, GPU, AP, etc.

[0156] RAM 1220 may temporarily store programs, data, or instructions. For example, programs and / or data stored in memory 1240 may be temporarily stored in RAM 1220 under the control of booting code or CPU 1210. RAM 1220 may be implemented by memory such as dynamic RAM (DRAM) or static RAM (SRAM).

[0157] The neural network device 1230 can perform neural network operations based on received input data and generate information signals based on the results of the operations. The neural network may include, but is not limited to, convolutional neural networks (CNNs), recurrent neural networks (RNNs), deep belief networks, restricted Boltzmann machines, etc. The neural network device 1230 may correspond to a hardware accelerator specifically designed to implement or drive the aforementioned neural network.

[0158] The information signal may include one of various types of recognition signals (such as image recognition signals). For example, the neural network device 1230 may receive frame data included in a video stream as input data and generate a recognition signal from the frame data for an object included in an image indicated by the frame data. However, this disclosure is not limited thereto, and depending on the type or function of the electronic device on which the electronic system 1200 is installed, the neural network device 1230 may receive various types of input data and generate recognition signals based on the input data.

[0159] Memory 1240 is a storage device for storing data (such as operating system (OS), various programs, and individual data entries). In one embodiment, memory 1240 may store intermediate results (such as output feature maps) generated during the computational execution processing of neural network device 1230 as an output feature list or output feature matrix. In one embodiment, memory 1240 may store compressed output feature maps. Additionally, memory 1240 may store quantized neural network data (such as parameters, weight maps, or weight lists used by neural network device 1230).

[0160] The memory 1240 may be DRAM, but is not limited thereto. The memory 1240 may include at least one of volatile memory and non-volatile memory. Non-volatile memory may include ROM, PROM, EPROM, EEPROM, flash memory, PRAM, MRAM, RRAM, FRAM, etc. Volatile memory may include DRAM, SRAM, SDRAM, etc. In one or more embodiments, the memory 1240 may include at least one of HDD, SSD, CF, SD, Micro-SD, Mini-SD, xD, and Memory Stick.

[0161] Sensor module 1250 can collect information about the periphery of the electronic device on which electronic system 1200 is installed. Sensor module 1250 can sense or receive signals from outside the electronic device (such as image signals, voice signals, magnetic signals, biometric signals, and touch signals) and convert the sensed or received signals into data. For this purpose, sensor module 1250 may include at least one of various types of sensing devices (such as microphones, imaging devices, image sensors, light detection and ranging (LIDAR) sensors, ultrasonic sensors, infrared sensors, biosensors, or touch sensors).

[0162] Sensor module 1250 may provide transformed data as input data to neural network device 1230. For example, sensor module 1250 may include an image sensor and may generate a video stream by capturing images of the external environment of the electronic device and provide successive data frames of the video stream to neural network device 1230 in the order of input data. However, this disclosure is not limited thereto, and sensor module 1250 may provide various types of data to neural network device 1230.

[0163] The communication module 1260 may include various wired or wireless interfaces capable of communicating with external devices. For example, the communication module 1260 may include: a local area network (LAN), a wireless local area network (WLAN) (such as Wi-Fi), a wireless personal area network (WPAN) (such as Bluetooth), a wireless universal serial bus (USB), Zigbee, near field communication (NFC), radio frequency identification (RFID), power line communication (PLC), or a communication interface capable of connecting to mobile cellular networks (such as third-generation (3G), fourth-generation (4G), long-term evolution (LTE), and fifth-generation (5G)).

[0164] In the following text, reference will be made as a non-limiting example. Figure 13 The neural network device 1230 is described in further detail.

[0165] Figure 13 A neural network apparatus according to one or more embodiments is shown.

[0166] Reference Figure 13 The neural network device 1300 according to the embodiment includes a memory 1310 (including an input activation memory 1311 and a weight memory 1312) and a processor 1320.

[0167] Figure 13 The neural network device 1300 shown includes components for performing the above-mentioned references. Figures 4 to 12 The elements of the method for convolution operation between input activations and weights in a neural network are described. Therefore, the above refers to... Figures 4 to 12 The given description can also be applied to Figure 13 The neural network device 1300 shown in the figure.

[0168] because Figure 13 The neural network device 1300 can be used with Figure 3 The neural network devices 300 are similar or identical, so for Figure 3 The description of the neural network device 300 can also be applied to Figure 13 The neural network device 1300.

[0169] In addition, because Figure 13 The memory 1310 can be connected with Figure 3 The memory 310 is similar to or the same as the memory 310, so for Figure 3 The description of memory 310 can also be applied to Figure 13 The memory 1310.

[0170] Figure 13 The processor 1320 may include a bit vector generator 1321, a bit vector merger 1322, a sorting unit 1323, a buffer 1324, and a processing unit 1325. Additionally, the processing unit 1325 may include a multiplier 1326 and an adder 1327.

[0171] Bit vector generator 1321 can generate bit vectors based on whether each input activation is 0. For example, bit vector generator 1321 can generate a first bit value for input activations whose values ​​are not 0 among a plurality of input activations, and generate a second bit value for input activations whose values ​​are 0 among a plurality of input activations.

[0172] Bit vector merger 1322 can merge bit vectors into input activations such that each bit value included in the bit vector is the MSB of a multi-bit representation corresponding to each input activation. Additionally, bit vector merger 1322 can merge bit vectors into weights such that each bit value included in the bit vector is the MSB of a multi-bit representation corresponding to each weight. For example, bit values ​​merged into the MSB of the multi-bit representation corresponding to a first input activation among multiple input activations can be merged into the MSB of the multi-bit representation corresponding to a first weight among multiple weights that will be computed with the first input activation.

[0173] The sorting unit 1323 can sort the input activations and weights based on the bits corresponding to the MSB. For example, the sorting unit 1323 can sort the input activations in the order of a first group and a second group, where the first group includes input activations whose first value is merged into the MSB, and the second group includes input activations whose second value is merged into the MSB. Additionally, the sorting unit 1323 can sort the weights in the order of a third group and a fourth group, where the third group includes weights whose first value is merged into the MSB, and the fourth group includes weights whose second value is merged into the MSB.

[0174] Buffer 1324 can store k+n input activations and k+n weights obtained from memory 1310. Buffer 1324 can be included in processor 1320 for pipelined applications.

[0175] The processing unit 1325 can perform operations between sorted input activations and sorted weights. For example, when the number of input activations belonging to the second group is less than a preset value, the processing unit 1325 can perform operations between the nth input activation in the sorted input activations and the nth weight in the sorted weights (n is a positive integer).

[0176] Furthermore, when the number of input activations belonging to the second group is a preset value or greater, the processing unit 1325 can perform operations between input activations belonging to the first group and weights belonging to the third group, and skip at least some of the operations between input activations belonging to the second group and weights belonging to the fourth group. In this regard, the preset value corresponds to the total number of input activations that can be operated on simultaneously, and operations between the number of input activations from the second group corresponding to the preset value and the number of weights from the fourth group corresponding to the preset value can be skipped.

[0177] because Figure 13 The multiplier 1326 and adder 1327 can be respectively connected to Figure 3 The multiplier 321 and the adder 322 are similar or identical, so Figure 3 The descriptions of multiplier 321 and adder 322 can also be applied to Figure 13The multiplier 1326 and the adder 1327.

[0178] Regarding Figures 1 to 13The described neural network device includes a memory, input activation memory, weight memory, processor, multiplier, adder, electronic system, CPU, RAM, sensor module, communication module, bit vector generator, bit vector merger, sorting unit, buffer, processing unit, neural network device 300, memory 310, input activation memory 311, weight memory 312, processor 320, multiplier 321, adder 322, electronic system 1200, CPU 1210, and RAM. 1220, 1230, 1240, 1250, 1260, 1300, 1310, 1311, 1312, 1320, 1321, 1322, 1323, 1324, 1325, 1326, 1327, and other devices, units, modules, apparatuses, and other components are implemented by hardware components or are representatives of hardware components. Examples of hardware components that can be used to perform the operations described in this application include, where appropriate, controllers, sensors, generators, drivers, memories, comparators, arithmetic logic modules, adders, subtractors, multipliers, dividers, integrators, and any other electronic components configured to perform the operations described in this application. In other examples, one or more of the hardware components performing the operations described in this application are implemented by computing hardware (e.g., by one or more processors or computers). A processor or computer may be implemented by one or more processing elements, such as logic gate arrays, controllers and arithmetic logic modules, digital signal processors, microcomputers, programmable logic controllers, field-programmable gate arrays, programmable logic arrays, microprocessors, or any other means or combination of means configured to respond to and execute instructions in a defined manner to achieve a desired result. In one example, the processor or computer includes or is connected to one or more memories storing instructions or software executed by the processor or computer. Hardware components implemented by the processor or computer may execute instructions or software (such as an operating system (OS) and one or more software applications running on the OS) for performing the operations described herein. The hardware components may also access, manipulate, process, create, and store data in response to the execution of instructions or software. For simplicity, the singular terms “processor” or “computer” may be used in the description of the examples described herein, but in other examples, multiple processors or computers may be used, or a processor or computer may include multiple processing elements or multiple types of processing elements or both. For example, a single hardware component or two or more hardware components may be implemented by a single processor, or two or more processors, or a processor and a controller.One or more hardware components may be implemented by one or more processors, or processors and controllers, and one or more other hardware components may be implemented by one or more other processors, or additional processors and additional controllers. One or more processors, or processors and controllers, may implement a single hardware component or two or more hardware components. Hardware components may have any one or more different processing configurations, examples of which include: a single processor, a discrete processor, a parallel processor, Single Instruction Single Data (SISD) multiple processing, Single Instruction Multiple Data (SIMD) multiple processing, Multiple Instruction Single Data (MISD) multiple processing, and Multiple Instruction Multiple Data (MIMD) multiple processing.

[0179] Figures 1 to 13 The methods for performing the operations described in this application, as shown, are executed by computing hardware (e.g., by one or more processors or a computer), which is implemented to execute instructions or software as described above to perform the operations performed by the methods described in this application. For example, a single operation or two or more operations may be executed by a single processor, or two or more processors, or a processor and a controller. One or more operations may be executed by one or more processors, or a processor and a controller, and one or more other operations may be executed by one or more other processors, or additional processors and additional controllers. One or more processors, or a processor and a controller, may execute a single operation or two or more operations.

[0180] Instructions or software for controlling computing hardware (e.g., one or more processors or computers) to implement hardware components and perform the methods described above can be written as computer programs, code segments, instructions, or any combination thereof to individually or collectively instruct or configure one or more processors or computers, such as machines or special-purpose computers, to perform the operations performed by the hardware components and methods described above. In one example, the instructions or software include machine code (such as machine code generated by a compiler) that is directly executed by one or more processors or computers. In another example, the instructions or software include high-level code that is executed by one or more processors or computers using an interpreter. The instructions or software can be written in any programming language based on the block diagrams and flowcharts shown in the accompanying drawings and the corresponding descriptions used herein, wherein the block diagrams and flowcharts shown in the accompanying drawings and the corresponding descriptions used herein disclose algorithms for performing the operations performed by the hardware components and methods described above.

[0181] Instructions or software for controlling computing hardware (e.g., one or more processors or computers) to implement hardware components and perform the methods described above, along with any associated data, data files, and data structures, may be recorded, stored, or fixed on one or more non-transitory computer-readable storage media. Examples of non-transitory computer-readable storage media include: read-only memory (ROM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-RLTH, BD-RE, Blu-ray Disc Retrieval System (BRS), etc. Optical or optical disc storage devices, hard disk drives (HDDs), solid-state drives (SSDs), card storage devices (such as multimedia cards or microcards (e.g., Secure Digital (SD) or Extreme Digital (XD))), magnetic tape, floppy disks, magneto-optical data storage devices, optical data storage devices, hard disks, solid-state drives, and any other devices configured to store instructions or software and any associated data, data files, and data structures in a non-transitory manner, and to provide the instructions or software and any associated data, data files, and data structures to one or more processors or computers, such that one or more processors or computers can execute the instructions. In one example, the instructions or software and any associated data, data files, and data structures are distributed across a networked computer system, such that the instructions and software and any associated data, data files, and data structures are stored, accessed, and executed in a distributed manner by one or more processors or computers.

[0182] While this disclosure includes specific examples, it will be clear upon understanding this disclosure that various changes in form and detail may be made in these examples without departing from the spirit and scope of the claims and their equivalents. The examples described herein are to be considered descriptive only and not for limiting purposes. The description of features or aspects in each example should be considered applicable to similar features or aspects in other examples. Suitable results may be achieved if the described techniques are performed in a different order, and / or if components in the described system, architecture, apparatus, or circuit are combined in a different manner, and / or replaced or supplemented by other components or their equivalents. Therefore, the scope of the disclosure is not limited by the specific embodiments but by the claims and their equivalents, and all variations within the scope of the claims and their equivalents should be construed as included in the disclosure.

Claims

1. A method for recognizing an image, the method comprising: Obtain the input image; Calculate input activation within the neural network based on the input image; For each of at least one layer in a neural network, a bit vector is generated based on whether each of the plurality of input activations of the layer in the neural network is 0; The bit values ​​of the bit vector are merged into the plurality of input activations, such that the bit values ​​included in the bit vector are the most significant bits of the multi-bit representation of the plurality of input activations; The bit values ​​of the bit vector are merged into the multiple weights corresponding to the multiple input activations, such that the bit values ​​included in the bit vector are the most significant bits of the multi-bit representation of the corresponding multiple weights; The plurality of input activations and the plurality of weights are sorted based on the bits corresponding to the most significant bit, such that the nth input activation and the nth weight among the sorted plurality of input activations and the sorted plurality of weights correspond to a pair for performing an operation. The operation between the sorting of the plurality of input activations and the sorting of the plurality of weights is performed; and Based on the calculation results, the image recognition result is output. The step of generating the bit vector includes: generating a first bit value for the input activations among the plurality of input activations whose values ​​are not 0, and generating a second bit value for the input activations among the plurality of input activations whose values ​​are 0. The step of merging bit values ​​of the bit vector into the plurality of input activations includes: merging each bit value of the bit vector into each input activation corresponding to each bit value in the plurality of input activations, such that each bit value included in the bit vector is the most significant bit of the multi-bit representation of each input activation corresponding to each bit value in the plurality of input activations. The sorting step includes: sorting the plurality of input activations in the order of a first group and a second group, wherein the first group includes input activations whose first value is merged into the most significant bit, and the second group includes input activations whose second value is merged into the most significant bit; and sorting the plurality of weights in the order of a third group and a fourth group, wherein the third group includes weights whose first value is merged into the most significant bit, and the fourth group includes weights whose second value is merged into the most significant bit. Among them, whether to skip the operation between the input activations belonging to the second group and the weights belonging to the fourth group is determined.

2. The method according to claim 1, wherein, The steps for calculating input activation within a neural network based on an input image include: The i-th layer of the neural network activates the output of the (i-1)-th layer, which serves as the input to the i-th layer; and the output activation of the i-th layer is output. Here, i is an integer greater than 1. In this system, the first layer of the neural network receives the input image and outputs the activation value of the first layer as the input activation value of the second layer based on the input image.

3. The method according to claim 2, wherein, The first value is 1, and the second value is 0.

4. The method according to claim 2, wherein, The step of merging bit values ​​of a bit vector into the plurality of input activations includes: merging the first bit value into the input activations with a non-zero value as the most significant bit of the merged input activation, and merging the second bit value into the input activations with a zero value as the most significant bit of the merged input activation.

5. The method according to claim 1, wherein, In response to the total number of input activations belonging to the second group being less than a preset value, the step of performing the operation includes: performing an operation between the nth input activation among the sorted plurality of input activations and the nth weight among the sorted plurality of weights, and n is a positive integer.

6. The method according to claim 1, wherein, In response to the total number of input activations in the second group being less than a preset value, the step of performing the operation includes: performing an operation between each of the sorted plurality of input activations and the corresponding weight among the sorted plurality of weights.

7. The method according to claim 1, wherein, In response to the total number of input activations belonging to the second group being greater than or equal to a preset value, the step of performing the operation further includes: Perform the operation between the input activations belonging to the first group and the weights belonging to the third group.

8. The method according to claim 7, wherein, The default value is the total number of inputs that are configured to be activated simultaneously by the processor, and The steps for skipping calculations include: in response to the total number of input activations belonging to the second group being greater than or equal to a preset value, skipping the calculation between the number of input activations from the second group corresponding to the preset value and the number of weights from the fourth group corresponding to the preset value; or in response to the total number of the plurality of input activations being greater than the preset value, skipping the calculation between the number of input activations belonging to the second group and the weights belonging to the fourth group that is equal to the difference between the total number of the plurality of input activations and the preset value.

9. The method according to claim 1, wherein, The steps to generate a bit vector include: The first vector is generated through the following steps: The first value of the first vector is generated in response to an input activation that is not zero among the plurality of input activations. A second bit value of the first vector is generated in response to another input activation being 0 among the plurality of input activations; The second vector is generated through the following steps: The first value of the second vector is generated in response to a non-zero weight among the plurality of weights. A second value of the second vector is generated in response to another weight among the plurality of weights being 0; and A bit vector is generated by performing a bitwise AND operation between the first vector and the second vector.

10. The method according to claim 1, wherein, The step of merging bit values ​​of a bit vector into the plurality of weights includes merging "the most significant bit of the multi-bit representation of the input activation that is merged into the plurality of input activations" into "the most significant bit of the multi-bit representation of the weight that will be operated on using the input activations among the plurality of weights".

11. The method according to claim 1, wherein, The sorting is performed using any one of the following algorithms: bubble sort, shell sort, bitonic sort, comb sort, cyclic sort, and odd-even merge sort.

12. The method according to claim 1, wherein, The step of performing the operation between the sorted plurality of input activations and the sorted plurality of weights includes: generating the output activation of the current layer of the neural network. In this process, the last layer of the neural network outputs the image recognition result based on the operation between the output activation of the previous layer and the corresponding weights.

13. A non-transitory computer-readable storage medium storing instructions, which, when executed by one or more processors, configure the processors to perform the method of any one of claims 1 to 12.

14. An apparatus for recognizing images, the apparatus comprising: The processor is configured as follows: Obtain the input image; Calculate input activation within the neural network based on the input image; For each of at least one layer in a neural network, a bit vector is generated based on whether each of the plurality of input activations of the layer in the neural network is 0; The bit values ​​of the bit vector are merged into the plurality of input activations, such that the bit values ​​included in the bit vector are the most significant bits of the multi-bit representation of the plurality of input activations; The bit values ​​of the bit vector are merged into the multiple weights corresponding to the multiple input activations, such that the bit values ​​included in the bit vector are the most significant bits of the multi-bit representation of the corresponding multiple weights; The plurality of input activations and the plurality of weights are sorted based on the bits corresponding to the most significant bit, such that the nth input activation and the nth weight among the sorted plurality of input activations and the sorted plurality of weights correspond to a pair for performing an operation. The operation between the sorting of the plurality of input activations and the sorting of the plurality of weights is performed; and Based on the calculation results, the image recognition result is output. In order to generate the bit vector, the processor is configured to: generate a first bit value for the input activations among the plurality of input activations whose values ​​are not 0, and generate a second bit value for the input activations among the plurality of input activations whose values ​​are 0. Specifically, to incorporate bit values ​​from the bit vector into the plurality of input activations, the processor is configured to: incorporate each bit value from the bit vector into each input activation corresponding to each bit value in the plurality of input activations, such that each bit value included in the bit vector is the most significant bit of the multi-bit representation of each input activation corresponding to each bit value in the plurality of input activations. Specifically, for sorting, the processor is configured to: sort the plurality of input activations in the order of a first group and a second group, wherein the first group includes input activations whose first value is merged into the most significant bit, and the second group includes input activations whose second value is merged into the most significant bit; and sort the plurality of weights in the order of a third group and a fourth group, wherein the third group includes weights whose first value is merged into the most significant bit, and the fourth group includes weights whose second value is merged into the most significant bit. Among them, whether to skip the operation between the input activations belonging to the second group and the weights belonging to the fourth group is determined.

15. The device according to claim 14, wherein, To compute input activations within a neural network based on an input image, the processor is configured to: output the output activation of the i-th layer of the neural network based on the output activation of the (i-1)-th layer, which is the input activation of the i-th layer, where i is an integer greater than 1. In this system, the first layer of the neural network receives the input image and outputs the activation value of the first layer as the input activation value of the second layer based on the input image.

16. The device according to claim 14, wherein, In response to the total number of input activations belonging to the second group being less than a preset value, the processor is configured to: perform an operation between the nth input activation among the sorted plurality of input activations and the nth weight among the sorted plurality of weights, and n is a positive integer.

17. The device according to claim 14, wherein, In order to perform the operation, in response to the total number of input activations belonging to the second group being greater than or equal to a preset value, the processor is also configured to: Perform the operation between the input activations belonging to the first group and the weights belonging to the third group.

18. The device according to claim 17, wherein, The default value is the total number of inputs that are configured to be activated simultaneously by the processor, and In order to skip operations, the processor is configured to: in response to the total number of input activations belonging to the second group being greater than or equal to a preset value, skip the operation between the number of input activations belonging to the second group corresponding to the preset value and the number of weights belonging to the fourth group corresponding to the preset value; or in response to the total number of the plurality of input activations being greater than the preset value, skip the operation between the number of input activations belonging to the second group and the weights belonging to the fourth group equal to the difference between the total number of the plurality of input activations and the preset value.

19. The device according to claim 14, wherein, In order to generate bit vectors, the processor is configured as follows: The first vector is generated through the following steps: The first value of the first vector is generated in response to an input activation that is not zero among the plurality of input activations. A second bit value of the first vector is generated in response to another input activation being 0 among the plurality of input activations; The second vector is generated through the following steps: The first value of the second vector is generated in response to a non-zero weight among the plurality of weights. A second value of the second vector is generated in response to another weight among the plurality of weights being 0; as well as A bit vector is generated by performing a bitwise AND operation between the first vector and the second vector.

20. The device according to claim 14, wherein, In order to merge bit values ​​of a bit vector into the plurality of weights, the processor is configured to merge "the most significant bit of the multi-bit representation of the input activation that is merged into the plurality of input activations" into "the most significant bit of the multi-bit representation of the weight that will be operated on with the input activation among the plurality of weights".

21. The device according to claim 14, wherein, For sorting purposes, the processor is configured to sort the plurality of input activations and the plurality of weights according to any one of the following algorithms: bubble sort, shell sort, bitonic sort, comb sort, cyclic sort, and odd-even merge sort.

22. The device according to claim 14, further comprising: The memory stores instructions that, when executed by the processor, configure the processor to perform the generation of a bit vector, the merging of bit values ​​of the bit vector into the plurality of input activations, the merging of bit values ​​of the bit vector into the plurality of weights, the sorting of the plurality of input activations and the plurality of weights, and the execution of the operations between the sorted plurality of input activations and the sorted plurality of weights.

Images