Binary neural network device and training method thereof

Abstract

A binary neural network apparatus including a binarization last layer is provided. In accordance with an embodiment of the present disclosure, a binary neural network apparatus includes: a binary neural network including a first layer, one or more second layers disposed between the first layer and the third layer, and a third layer having a binary input and a binary weight and configured to output a binarized bit sequence; and a non-binary converter configured to convert the binarized bit sequence into non-binary data and output the non-binary data.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority from Korean Patent Application No. 10-2023-0073080 filed on June 7, 2023, in the Korean Intellectual Property Office, the entire contents of which are incorporated herein by reference for all purposes. Technical Field

[0003] The following description relates to a binary neural network device including a binarized last layer and a method of training the binary neural network. Background Art

[0004] With the development of artificial neural network technology, artificial neural networks are used in various electronic systems to analyze input data and extract effective information. Specifically, convolutional neural networks (CNNs) have excellent performance in extracting spatial features included in images. However, the convolution operation used to run convolutional neural networks requires high input / output bandwidth. By binarizing weights and activation functions, convolution multiplication operations can be replaced by binary operations. Ordinary binary neural networks can increase speed and significantly reduce the required storage capacity, but due to information loss, accuracy may be reduced. In order to maintain accuracy, ordinary binary neural networks have been developed in the following manner: multiplication operations are provided in the input layer and the last layer of the neural network while maintaining full precision. Summary of the invention

[0005] According to one aspect of the present disclosure, a binary neural network device may include: a binary neural network including a first layer, one or more second layers, and a third layer, wherein the one or more second layers are arranged between the first layer and the third layer, and the third layer has a binary input and a binary weight, and is configured to output a binarized bit sequence; and a non-binary converter, configured to convert the binarized bit sequence into non-binary data and output the non-binary data.

[0006] The third layer may be the last layer of the binary neural network, and the non-binary converter is configured to convert the binarized bit sequence into non-binary data by using a predefined linear or non-linear function.

[0007] The third layer may be the last layer of the binary neural network, and the non-binary converter is configured to convert the binarized bit sequence into non-binary data by using a predefined lookup table.

[0008] The third layer may be the last layer of the binary neural network, and the non-binary converter is configured to convert the binarized bit sequence into non-binary data by using the position of the last activated bit in the binarized bit sequence.

[0009] The third layer may be the last layer of the binary neural network, and the non-binary converter is configured to convert the binarized bit sequence into non-binary data by using the number of activation bits in the binarized bit sequence.

[0010] The third layer may be the last layer of the binary neural network, and the non-binary converter is configured to convert the binarized bit sequence into non-binary data by using a weighted sum of each position in the binarized bit sequence.

[0011] The first layer of a binary neural network can have binary inputs.

[0012] The binary neural network device may further include a first binary converter configured to convert non-binary data into a bit sequence to be used as a binary input of the first layer.

[0013] In a training operation, the binary neural network may be configured to convert a non-binary reference true value into a reference true value bit sequence by using a second binary converter, and train the binary neural network based on a loss between the converted reference true value bit sequence and a binarized bit sequence output by the third layer.

[0014] In a training operation, in response to the non-binary transformer being defined as differentiable, a binary neural network may be trained based on a loss between a non-binary ground truth value and non-binary data that is an output of the non-binary transformer.

[0015] According to another aspect of the present disclosure, a training method for a binary neural network includes a first layer, one or more second layers, and a third layer. The method may include: obtaining first output data for training data from the first layer; obtaining second output data for the first output data from the one or more second layers; obtaining a bit sequence as third output data for the second output data from a third layer having binary inputs and binary weights; and training the binary neural network by calculating the loss of a loss function based on the bit sequence and a non-binary reference true value.

[0016] The training method may further include converting the bit sequence into non-binary data using a non-binary converter, wherein calculating the loss may include: in response to the non-binary converter being defined as differentiable, calculating the loss based on the converted non-binary data and the non-binary reference true value.

[0017] Converting the bit sequence into non-binary data may include converting the bit sequence into non-binary data by using a position of a last activated bit in the bit sequence.

[0018] Converting the bit sequence into non-binary data may include converting the bit sequence into non-binary data by using a number of activated bits in the bit sequence.

[0019] Converting the bit sequence into non-binary data may include converting the bit sequence into non-binary data by using a weighted sum of each position in the bit sequence.

[0020] The first layer may have a binary input, wherein the method may further include: converting non-binary training data into binary training data by using a first binary converter, and providing the binary training data to the first layer.

[0021] The training method may further include converting the non-binary reference true value into a reference true value bit sequence by using a second binary converter, wherein calculating the loss includes calculating the loss based on the bit sequence and the converted reference true value bit sequence.

[0022] According to another aspect of the present disclosure, an electronic device may include: a memory configured to store an image, and weights and activation functions of a binary neural network; and at least one processor configured to load the image from the memory and process the image through the binary neural network, wherein the binary neural network includes a first layer, one or more second layers, and a third layer, the one or more second layers are arranged between the first layer and the third layer, and the third layer has a binary input and a binary weight and is configured to output a binarized bit sequence; and a non-binary converter configured to convert the binarized bit sequence into non-binary data and output the non-binary data.

[0023] The binary neural network may be trained based on at least one of a reference true value, a binarized bit sequence, and non-binary data.

[0024] The electronic device may also include a video codec device or an image signal processing device that processes images using a binary neural network. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 is a block diagram showing a binary neural network device according to an embodiment of the present disclosure.

[0026] Figure 2 A diagram illustrating an example of a conversion method of a non-binary converter.

[0027] Figure 3 is a block diagram showing a binary neural network device according to another embodiment of the present disclosure.

[0028] Figure 4 is a flowchart illustrating an inference method according to an embodiment of the present disclosure.

[0029] Figure 5 is a flowchart illustrating an inference method according to another embodiment of the present disclosure.

[0030] Figure 6is a block diagram illustrating a training device according to an embodiment of the present disclosure.

[0031] Figure 7 is a block diagram showing a training device according to another embodiment of the present disclosure.

[0032] Figure 8 is a flow chart illustrating a training method according to an embodiment of the present disclosure.

[0033] Fig. 9 is a flow chart showing a training method according to another embodiment of the present disclosure.

[0034] Fig.10 is a block diagram of an electronic device including a binary neural network device. DETAILED DESCRIPTION

[0035] Details of other embodiments are included in the following detailed description and drawings. Advantages and features of the present invention and methods of implementing the same will be more clearly understood according to the embodiments described in detail below with reference to the drawings. Throughout the drawings and detailed description, unless otherwise described, the same reference numerals will be understood to refer to the same elements, features and structures.

[0036] It should be understood that although the terms "first", "second", etc. may be used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish elements from each other. Unless otherwise explicitly stated, any reference to the singular may include the plural. In addition, unless explicitly described to the contrary, expressions such as "including" or "comprising" will be understood to mean including the elements described, but not excluding any other elements. And, terms such as "unit" or "module" should be understood as units that perform at least one function or operation and can be embodied as hardware, software or a combination thereof.

[0037] Figure 1 is a block diagram showing a binary neural network device according to an embodiment of the present disclosure. Figure 2 A diagram illustrating an example of a conversion method of a non-binary converter.

[0038] The binary neural network device includes a neural network for supporting the performance of an intellectual property (IP) core, such as an image signal processing (ISP) device or a video codec implemented in chip form, and can be included in the IP core or installed as a separate hardware or software module in an electronic device.

[0039] refer to Figure 1, the binary neural network device 100 may include a binary neural network 110 and a non-binary converter 120. Some of the weights and activations of the binary neural network 110 may be represented as binary values ​​(e.g., +1 and -1, or 1 and 0) instead of floating point numbers to reduce memory storage and computation requirements.

[0040] The binary neural network 110 may include a first layer 111, one or more second layers 112, and a third layer 113, wherein the one or more second layers 112 are disposed between the first layer 111 and the third layer 113. The first layer 111, the one or more second layers 112, and the third layer 113 may also be referred to as an input layer, a hidden layer, and an output layer, respectively.

[0041] The first layer 111 uses non-binary data ID as input. Non-binary data can be real data or real numbers representing images or having a floating point format, such as single-precision floating point (FP32, also known as "32-bit floating point") using 32 bits to represent real data or real numbers, half-precision floating point (FP16, also known as "16-bit floating point") using 16 bits to represent real data or real numbers, brain floating point (BF16, also known as "16-bit floating point") using 16 bits to represent real data or real numbers, 8-bit integers (int8) using 8 bits to represent real data or real numbers, etc. The first layer 111 can be a convolutional layer or a linear layer having floating point weights and performing full-precision floating point operations. The first layer 111 can binarize the result of the floating point operation and provide the result to the second layer 112. In this case, binarization is the process of quantizing floating point data into binary data (-1 or +1), and various binarization methods can be used. For example, a hyperbolic tangent function (Tanh(x)), a sine function (Sign(x)), or HTann(x) etc. may be used, but the method is not limited thereto.

[0042] In an embodiment of the present disclosure, the first layer 111 may transform a real number into a binary bit sequence by a thermometer method, in which the real number is encoded into a binary bit sequence within a predetermined digital range. Specifically, the thermometer method may include the following operations: selecting a digital range (e.g., a range from 0 to 1), determining the number of bits used to represent the real number (e.g., 8 bits), dividing the range of the real number into segments each represented by one bit (e.g., for a thermometer resolution of 32, segment 1 for 0 to 31, segment 2 for 32 to 63, segment 3 for 64 to 95, segment 4 for 96 to 127, segment 5 for 128 to 159, segment 6 for 160 to 191, segment 7 for 192 to 223, segment 8 for 234 to 255), and converting the real number into an 8-bit thermometer code (e.g., when the real number is 109, it is 0000111).

[0043] The second layer 112 has a binary input. The second layer 112 can be a network configured to perform bitwise operations, such as XNOR, which compares two binary values ​​(bits) and returns true (1) if the two input bits are equal (both are 0 or both are 1); and, for example, POP-COUNT, which counts the number of set bits (bits with a value of 1) in a binary sequence by binarizing the weights and activation functions. In this case, the weight can be quantized to 1 bit, and the activation function consists of 1 bit or multiple bits. The second layer 112 can perform a binary operation on the binary input provided from the first layer 111, and can provide a binary output to the third layer 113.

[0044] The third layer 113 may include a binary weight and / or a binary activation function to perform a binary operation by using the binary input provided from the second layer 112. The third layer 113 is the last layer of the binary neural network, and may perform a binary operation to output a bit sequence (BS) consisting of a single bit or a plurality of bits.

[0045] In an embodiment of the present disclosure, the third layer 113 may be implemented as a convolutional layer or a linear (fully connected) layer. When the third layer 113 is implemented as a convolutional layer, the output bit sequence (BS1) of the third layer 113 may be a feature map, which is a three-dimensional (3D) tensor with dimensions of width, height, and channel. When the third layer 113 is implemented as a linear layer, the output bit sequence (BS2) of the third layer 113 may be a one-dimensional (1D) vector, which is called a logarithmic ratio (logit) or a score.

[0046] The non-binary converter 120 can convert the binarized bit sequence BS output by the third layer 113 into non-binary data OD, and can output non-binary data. The non-binary converter 120 can be implemented in the form of a hardware chip or a software module. The non-binary converter 120 can be defined as differentiable or non-differentiable in various forms without restriction. The non-binary converter 120 can be configured to convert the bit sequence BS into non-binary data OD by using a predefined linear or non-linear function. Alternatively, the non-binary converter 120 can be configured to convert the bit sequence BS into non-binary data OD by using a predefined lookup table.

[0047] Figure 2 A diagram illustrating an example of a conversion method of a non-binary converter.

[0048] The non-binary converter 120 may be implemented as follows: Figure 2 One of the various conversion methods a, b and c is shown, but is not specifically limited to the exemplary method.

[0049] refer to Figure 2In the conversion method (a), the non-binary converter 120 can convert the bit sequence output by the third layer 113 into real data or real numbers based on the position of the last activated bit. For example, the data can be pre-quantized by dividing the real value to be represented by the number of channels. The term "channel" represents the dimension of the output bit sequence. In the conversion method (a), an example of defining a real value ranging from 0 to 256 bits as 32 (256 / 8) channels with a resolution of 8 is shown, where the position of the last activated bit in the first bit sequence is 3. The value 3 corresponds to the following situation: the real value is quantized by dividing 95 by the number of channels 32, so that the real value corresponding to the position of the last activated bit is 95. The position of the last activated bit in the second bit sequence is 5, and the real value corresponding to the number 5 is 159. The non-binary converter 120 can output a real value by using a linear or nonlinear function or by referring to a lookup table that maps a position to a real value, wherein the linear or nonlinear function is defined to output a real value corresponding to the position of the last activated bit.

[0050] refer to Figure 2 In the conversion method (b), the non-binary converter 120 can convert the bit sequence into non-binary data by using the number of activation bits in the bit sequence. There may be a predefined lookup table NT or a linear / nonlinear function, wherein the predefined lookup table NT defines a mapping between the number of activation bits and the corresponding real value, and the linear / nonlinear function is defined to output a real value corresponding to the number of activation bits. When the number of activation bits in the first bit sequence is 3, its corresponding real value 95 is output, and when the number of activation bits in the second bit sequence and the third bit sequence is 4, its corresponding real value 127 is output.

[0051] refer to Figure 2 In the conversion method (c), the non-binary converter 120 can convert the bit sequence into non-binary data by using the weighted sum of each position in the bit sequence. A weight can be assigned to each position in the bit sequence, in which case the weight applied to each position in the bit sequence increases from the least significant bit LSB toward the most significant bit (MSB). In the conversion method (c), an example of using a binary to decimal conversion method is shown, but the conversion method is not limited to this.

[0052] In a normal binary neural network, multiplication operations are performed in the last layer, and a small number of multiplication operations may not cause problems for the software, but the IP core requires a separate multiplier to support multiplication, which is inefficient in terms of area and usefulness. In addition, if the IP core performs multiplication through a central processing unit (CPU) or the like, without directly implementing the multiplier, input and output occur in a dynamic random access memory (DRAM), which is inefficient in terms of power efficiency and latency. In this embodiment of the present disclosure, the last layer of the binary neural network is binarized, thereby minimizing the required storage capacity and allowing efficient memory usage.

[0053] Figure 3 is a block diagram showing a binary neural network device according to another embodiment of the present disclosure.

[0054] refer to Figure 3 The binary neural network device 300 includes a binary converter 310 , a binary neural network 320 and a non-binary converter 330 .

[0055] The binary converter 310 can convert the input non-binary data ID into a binary bit sequence, and the binary bit sequence can be provided to the binary neural network 320. The binary converter 310 can adopt various binarization methods defined as quantizing the floating point data in a single channel into binary data (-1 or +1) in multiple channels. For example, by dividing the input value range (e.g., 256) by the number of channels (e.g., 8), the bit sequence can be converted so that each channel represents a binary bit. However, the binarization method is not limited to this, and a hyperbolic tangent function (Tanh(x)), a sine function (Sign(x)) or HTann(x) etc. can be used. The binary converter 310 can be implemented in the form of a hardware chip or a software module.

[0056] In an embodiment of the present disclosure, the binary converter 310 can transform the real number into a binary bit sequence by a thermometer method, in which the real number is encoded as a binary bit sequence within a predetermined digital range. Specifically, the thermometer method may include the following operations: selecting a digital range (e.g., a range from 0 to 1), determining the number of bits used to represent the real number (e.g., 8 bits), dividing the range of the real number into segments each represented by one bit (e.g., for a thermometer resolution of 32, segment 1 for 0 to 31, segment 2 for 32 to 63, segment 3 for 64 to 95, segment 4 for 96 to 127, segment 5 for 128 to 159, segment 6 for 160 to 191, segment 7 for 192 to 223, segment 8 for 234 to 255), and converting the real number into an 8-bit thermometer code (e.g., when the real number is 109, it is 0000111).

[0057] The binary neural network 320 may include a first layer 321 , one or more second layers 322 , and a third layer 323 , wherein the one or more second layers 322 may be disposed between the first layer 321 and the third layer 323 .

[0058] The first layer 321 has a binary input and a binary weight and / or a binary activation function. The first layer 321 can perform a binary operation by using the binary data output by the binary converter 310 as an input, and can output binary data. By binarizing the first layer 321 as the input layer of the binary neural network 320, the storage capacity and efficiency can be further improved compared to the case where the multiplication operation is performed while maintaining full precision in the first layer 321.

[0059] The second layer 322 may be a network having a binary input and configured to perform bitwise operations (e.g., XNOR and POP-COUNT) by binarizing the weights and activation functions. In this case, the weights may be quantized to 1 bit, and the activation functions may consist of 1 bit or more bits. The second layer 322 may perform a binary operation on the binary input provided from the first layer 321, and may provide a binary output to the third layer 323.

[0060] The third layer 323 may include a binary weight and / or a binary activation function to perform a binary operation by using the binary input provided from the second layer 322. The third layer 323 is the last layer of the binary neural network, and may perform a binary operation to output a bit sequence (BS) consisting of a single bit or a plurality of bits.

[0061] In an embodiment of the present disclosure, the third layer 323 may be implemented as a convolutional layer or a linear (fully connected) layer. When the third layer 323 is implemented as a convolutional layer, the output bit sequence (BS1) of the third layer 323 may be a feature map, which is a three-dimensional (3D) tensor with dimensions of width, height, and channel. When the third layer 323 is implemented as a linear layer, the output bit sequence (BS2) of the third layer 323 may be a one-dimensional (1D) vector, which is called a logarithmic ratio (logit) or a score.

[0062] By using Figure 2The conversion method defined in various ways shown, the non-binary converter 330 can convert the binarized bit sequence BS output by the third layer 323 into non-binary data OD, and can output non-binary data. The non-binary converter 330 can be implemented in the form of a hardware chip or a software module. The non-binary converter 330 can be defined as differentiable or non-differentiable in various forms without restriction. When the non-binary converter 330 uses a smooth and continuously changing function and therefore has a derivative, the non-binary converter 330 is differentiable, so that the non-binary converter 330 can calculate the rate of change of the function at any point. In this case, the non-binary converter 330 can predict its output based on the change of the input data value of the non-binary converter 330. The non-binary converter 330 can be configured to convert the bit sequence BS into non-binary data OD by using a predefined linear or nonlinear function. Alternatively, the non-binary converter 330 can be configured to convert the bit sequence BS into non-binary data OD by using a predefined lookup table.

[0063] Figure 4 is a flowchart illustrating an inference method according to an embodiment of the present disclosure.

[0064] Figure 4 The method can be based on Figure 1 The binary neural network device 100 of the embodiment is implemented. The binary neural network device 100 is described in detail above, so it will be briefly described below.

[0065] In operation 410, the binary neural network device 100 may input non-binary input data to the first layer of the binary neural network, and may obtain first output data of the first layer. In this case, the first layer may perform a full-precision multiplication operation, and may binarize the operation result to output binary data.

[0066] Then, in operation 420, the first output data as the output of the first layer may be provided to the second layer of the binary neural network, and the second layer may perform a binary operation on the binary input to obtain second output data.

[0067] Subsequently, in operation 430, the second output data as the output of the second layer can be provided to the third layer as the last layer of the binary neural network, and the third layer configured to perform a binary operation on the binary input can obtain a binary bit sequence as the third output data.

[0068] Next, in operation 440, the binary neural network device 100 may convert the binary bit sequence as the third output data into non-binary data by using the non-binary converter, and may output the non-binary data. The non-binary data output by using the binary converter may be provided to the next processing step (e.g., encoding and decoding, image signal processing).

[0069] Figure 5 is a flowchart illustrating an inference method according to another embodiment of the present disclosure.

[0070] Figure 5 The method can be based on Figure 3 The binary neural network device 300 of the embodiment is performed. The binary neural network device 300 is described in detail above, so it will be briefly described below.

[0071] In operation 510 , the binary neural network device 300 may convert non-binary input data into a binary bit sequence by using a binary converter, and may provide the converted bit sequence to a first layer of the binary neural network.

[0072] Then, in operation 520, first output data may be obtained through the first layer of the binary neural network 300. In this case, the first layer may be a layer that is binarized to perform a binary operation on a binary input.

[0073] Subsequently, in operation 530, first output data as an output of the first layer may be provided to the second layer, and the second layer may perform a binary operation on the binary input to obtain second output data.

[0074] Next, in operation 540, the second output data may be provided to a third layer which is a last layer of the binary neural network, and the third layer may perform a binary operation on the binary input to obtain a binary bit sequence as third output data.

[0075] Then, in operation 550, the binary neural network device 300 may convert the binary bit sequence as the third output data into non-binary data by using the non-binary converter, and may output the non-binary data. The non-binary data output by using the binary converter may be provided to the next processing step (e.g., encoding and decoding, image signal processing).

[0076] Figure 6 is a block diagram illustrating a training device according to an embodiment of the present disclosure.

[0077] refer to Figure 6The training device 700a includes a binary neural network device 600, a second binary converter 710 for training a binary neural network 620 of the binary neural network device 600, and a loss calculator 720.

[0078] As described above, the binary neural network device 600 includes: a binary neural network 620, in which the first layer and / or the last layer is binarized; and a non-binary converter 630 configured to convert a bit sequence as an output of the binary neural network 620 into non-binary data. In addition, as described above, the binary neural network device 600 may further include a first binary converter 610.

[0079] The training data TD may be non-binary data (e.g., image, real value) and may be directly input to the binary neural network 620 as described above, or may be converted into a binary bit sequence by the first binary converter 610 to be input to the binary neural network 620. The binary neural network 620 may output a training bit sequence BS_T binarized for the training data TD.

[0080] The second binary converter 710 can convert the non-binary reference real value into the reference real value bit sequence BS_G by binarizing the non-binary reference real value. In this case, the second binary converter 710 can adopt the same conversion method as the first binary converter 610, but is not necessarily limited thereto, and can adopt different methods.

[0081] The loss calculator 720 can calculate the loss of the loss function by using the training bit sequence BS_T output by the binary neural network 620 and the reference true value bit sequence BS_G converted by the second binary converter 710, and can train the binary neural network 620 by using the back propagation method so that the loss can be minimized. The reference true value and the actual training result are in the same domain as the bit sequence, so that various loss functions can be used by considering the bit sequence. For example, the loss function can be intersection over union (IoU), binary cross entropy (BCE), peak signal-to-noise ratio (PSNR), mean square error (MSE), cross entropy loss, log likelihood loss, frequency domain loss, etc., but not limited to this. In addition, the binary neural network 620 can be trained by directly calculating the loss of the loss function in binary form, so that the process of converting to floating point values ​​can be removed, and the training speed can be increased.

[0082] Figure 7 is a block diagram showing a training device according to another embodiment of the present disclosure.

[0083] refer to Figure 7 , the training device 700b includes a binary neural network device 600 and a loss calculator 720.

[0084] As described above, the binary neural network device 600 includes: a binary neural network 620, in which the first layer and / or the last layer is binarized; and a non-binary converter 630 configured to convert a bit sequence as an output of the binary neural network 620 into non-binary data. In addition, as described above, the binary neural network device 600 may further include a first binary converter 610.

[0085] The training data TD may be non-binary data (e.g., image, real value) and may be directly input to the binary neural network 620 as described above, or may be converted into a binary bit sequence by the first binary converter 610 to be input to the binary neural network 620. The binary neural network 620 may output a training bit sequence BS_T binarized for the training data TD.

[0086] The non-binary converter 630 may convert the bit sequence BS_T, which is the output of the binary neural network 620, into non-binary data. In this case, the non-binary converter 630 may be defined as a differentiable, for example, linear function.

[0087] The loss calculator 720 can calculate the loss of the loss function by using the non-binary training data TD and the non-binary data converted by the non-binary converter 630 of the binary neural network 620, and can train the binary neural network 620 by using the back propagation method so that the loss can be minimized. If the non-binary converter 630 is defined as differentiable, the back propagation flows so that the loss function can be designed to allow end-to-end training by direct comparison of the non-binary data as the final output of the binary neural network 600 with the reference true value.

[0088] Figure 8 is a flow chart illustrating a training method according to an embodiment of the present disclosure.

[0089] The following will briefly describe Figure 8 The method of avoiding redundancy is an example of a training method performed by the training device 700a.

[0090] In operation 810, the training data TD may be non-binary data (e.g., image, real value) and may be directly input into the binary neural network as described above in the form of non-binary data, or may be converted into a binary bit sequence by a first binary converter to be input into the binary neural network.

[0091] Then, in operations 820, 830 and 840, first output data, second output data and third output data can be obtained through the first layer, second layer and third layer of the binary neural network, respectively. In this case, the last layer of the binary neural network can be binarized to output a binary bit sequence.

[0092] Subsequently, in operation 850 , the second binary converter may convert the non-binary reference real value into a reference real value bit sequence.

[0093] Next, in operation 860, a loss of a loss function may be calculated by using the training bit sequence obtained in operation 840 and the reference true value bit sequence converted in operation 850, and a binary neural network may be trained to minimize the loss by performing back propagation in operation 870.

[0094] Fig. 9 is a flow chart showing a training method according to another embodiment of the present disclosure.

[0095] The following will briefly describe Fig. 9 The method is an example of a training method performed by the training device 700b to avoid redundancy.

[0096] In operation 910, the training data TD may be non-binary data (e.g., image, real value) and may be directly input into the binary neural network as described above in the form of non-binary data, or may be converted into a binary bit sequence by a first binary converter to be input into the binary neural network.

[0097] Then, in operations 920, 930, and 940, first output data, second output data, and third output data may be obtained through the first layer, the second layer, and the third layer of the binary neural network, respectively.

[0098] Subsequently, in operation 950, the non-binary converter may convert the training bit sequence output in operation 940 into non-binary data.

[0099] Next, in operation 960 , a loss of a loss function may be calculated by using the training bit sequence and the non-binary reference true value obtained in operation 940 , and a binary neural network may be trained to minimize the loss by performing back propagation in operation 970 .

[0100] Fig.10 is a block diagram of an electronic device including a binary neural network device.

[0101] The electronic device can be various types of wearable devices (e.g., smart watches, smart bracelets, smart glasses, smart headphones, smart rings, smart patches, smart necklaces) and mobile devices (e.g., smart phones, tablet PCs, etc.), or household appliances or various Internet of Things (IoT) devices based on Internet of Things (IoT) technology (e.g., home IoT devices, etc.), but the electronic device is not limited to this and can include various information processing devices.

[0102] refer to Fig.10 , the electronic device 1000 includes a binary neural network device 1010, an image processing device 1020, and a memory. The binary neural network device 1010 and the image processing device 1020 can be implemented as separate hardware or software modules. Alternatively, the binary neural network device 1010 can be integrated into the image processing device 1020 to be implemented as one module.

[0103] The memory may include, but is not limited to, a dynamic random access memory (DRAM) 1031 and a cache memory 1032. The DRAM 1031 may store input data IDs such as images, videos, and data such as weights and activation functions of a neural network. In addition, the memory may store various processing results of the electronic device, including, for example, pre-processed data of the binary neural network device 1010, processing results OD of the image processing device 1020, and the like.

[0104] The binary neural network device 1010 may be the binary neural network devices 100, 300, and 600 described in the above embodiments. Data IDs such as images or video frames (current frame (t) and reference frame (t-1)) may be loaded from the memory 1031 to be input to the binary neural network device 1010. As described above, all layers of the binary neural network device 1010 may be binarized, or the intermediate layers and the last layer thereof except for the input layer may be binarized. By binarizing the binary neural network, binary weights and binary activation functions may be used, so that the required capacity may be reduced, allowing efficient memory use, and thus increasing the speed of accessing weights and activation function values ​​at the cache memory 1032 level.

[0105] The binary neural network device 1010 may preprocess the input image or video frame by reading the weights and activation functions from the cache memory 1032, and may output preprocessed data PD used by the image processing device 1020 to optimize image processing. The preprocessed data PD as the output of the binary neural network device 1010 may include a saliency map (SM) or global motion information (GM), etc., but is not limited thereto, and may include various information required for using a deep learning neural network in image processing.

[0106] The image processing device 1020 can be a video codec device or a video stabilizer for performing video encoding, or an image signal processing (SIP) device for addressing physical degradation including noise reduction (NR), high dynamic range (HDR), deblurring, frame rate up conversion (FRUC), etc.

[0107] By using the saliency map (SM) output by the binary neural network device 1010, the image processing device 1020 can identify important areas in the image or extract areas of interest from the image. In addition, the image processing device 1020 can perform motion estimation in the video by using global motion information (GM). For example, the global motion information (GM) may include initial search position information within a search range during motion estimation between a current frame (t) and a reference frame (t-1). The image processing device 1020 may perform motion estimation after moving the center of the search range to the initial search position by using the global motion information (GM).

[0108] The electronic device 1000 may also include one or more processors, communication modules, camera modules, output devices, or sensor devices configured to detect various data (for example, an accelerometer, a gyroscope sensor, a geomagnetic sensor, a proximity sensor, an illumination sensor, a fingerprint sensor, etc.), input devices configured to receive commands and / or data to be used from a user (for example, a microphone, a mouse, a keyboard and / or a digital pen (for example, a stylus, etc.)), etc.

[0109] One or more processors may control components of the electronic device 1000 and may perform various data processing or calculations. The processor may load image or video data stored in the memory 1031 to input to the binary neural network device 1010. The processor may include a main processor (e.g., a central processing unit (CPU) or an application processor (AP), etc.), and an auxiliary processor (e.g., a graphics processing unit (GPU), an image signal processor (ISP), a sensor hub processor, or a communication processor (CP), etc.) that may operate independently of the main processor or may operate in conjunction with the main processor.

[0110] The camera module can capture still images or moving images, etc., and can store the captured images in the memory 1031. The camera module may include a lens assembly having one or more lenses, an image sensor, an image signal processor, and / or a flash. The lens assembly included in the camera module can collect light emitted from an object to be imaged.

[0111] The output device may visually / non-visually output an image captured by the camera module and / or data processed by the image processing apparatus 1020. The output device may include a sound output device, a display device, an audio module, and / or a haptic module.

[0112] The communication module can support the establishment of a direct (e.g., wired) communication channel and / or a wireless communication channel between the electronic device and other electronic devices, servers, or sensor devices within the network environment and perform communication via the established communication channel. The communication module can transmit images captured by the camera module and / or data processed by the image processing device 1020 (e.g., video encoded using a codec, etc.).

[0113] The present disclosure may be implemented as computer-readable codes written on a computer-readable recording medium. The computer-readable recording medium may be any type of recording device that stores data in a computer-readable manner.

[0114] Examples of computer-readable recording media include ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage, and carrier wave (e.g., data transmission via the Internet). Computer-readable recording media can be distributed on multiple computer systems connected to a network so that computer-readable codes are written therein and executed therefrom in a decentralized manner. Those skilled in the art to which the present invention belongs can easily infer the functional programs, codes, and code segments required to implement the present invention.

[0115] The present disclosure has been described herein with reference to preferred embodiments. However, it is apparent to those skilled in the art that various changes and modifications may be made without changing the technical concept and basic features of the present disclosure. Therefore, it is clear that the above embodiments are illustrative in all aspects and are not intended to limit the present disclosure.

Claims

1. A binary neural network device, include: A binary neural network, comprising a first layer, one or more second layers, and a third layer, wherein the one or more second layers are arranged between the first layer and the third layer, and the third layer has a binary input and a binary weight, and is configured to output a binarized bit sequence; and The non-binary converter is configured to convert the binarized bit sequence into non-binary data and output the non-binary data.

2. The binary neural network device according to claim 1, in, The third layer is the last layer of the binary neural network, and the non-binary converter is configured to convert the binarized bit sequence into the non-binary data by using a predefined linear or non-linear function.

3. The binary neural network device according to claim 1, in, The third layer is the last layer of the binary neural network, and the non-binary converter is configured to convert the binarized bit sequence into the non-binary data by using a predefined lookup table.

4. The binary neural network device according to claim 1, in, The third layer is the last layer of the binary neural network, and the non-binary converter is configured to convert the binarized bit sequence into the non-binary data by using the position of the last activated bit in the binarized bit sequence.

5. The binary neural network device according to claim 1, in, The third layer is the last layer of the binary neural network, and the non-binary converter is configured to convert the binarized bit sequence into the non-binary data by using the number of activated bits in the binarized bit sequence.

6. The binary neural network device according to claim 1, in, The third layer is the last layer of the binary neural network, and the non-binary converter is configured to convert the binarized bit sequence into the non-binary data by using a weighted sum of each position in the binarized bit sequence.

7. The binary neural network device according to claim 1, in, The first layer of the binary neural network has a binary input.

8. The binary neural network device according to claim 7, further comprising: include: The first binary converter is configured to convert non-binary data into a bit sequence to be used as a binary input of the first layer.

9. The binary neural network device according to claim 1, in, In a training operation, the binary neural network is configured to convert a non-binary reference real value into a reference real value bit sequence by using a second binary converter, and train the binary neural network based on a loss between the converted reference real value bit sequence and the binarized bit sequence output by the third layer.

10. The binary neural network device according to claim 1, in, In a training operation, in response to the non-binary converter being defined as differentiable, the binary neural network is trained based on a loss between a non-binary ground truth value and non-binary data, the non-binary data being an output of the non-binary converter.

11. A method for training a binary neural network, the binary neural network comprising a first layer, one or more second layers, and a third layer, the method include: Obtain first output data for training data from the first layer; obtaining second output data for the first output data from the one or more second layers; Obtaining a bit sequence from the third layer having binary input and binary weight as third output data for the second output data; as well as The binary neural network is trained by calculating the loss of a loss function based on the bit sequence and the non-binary reference true value.

12. The training method according to claim 11, further comprising converting the bit sequence into non-binary data by using a non-binary converter, in, Calculating the loss includes: in response to the non-binary converter being defined as differentiable, calculating the loss based on the converted non-binary data and the non-binary reference true value.

13. The training method according to claim 12, in, Converting the bit sequence into the non-binary data comprises: converting the bit sequence into the non-binary data by using the position of the last activated bit in the bit sequence.

14. The training method according to claim 12, in, Converting the bit sequence into the non-binary data comprises converting the bit sequence into the non-binary data by using the number of activated bits in the bit sequence.

15. The training method according to claim 12, in, Converting the bit sequence into the non-binary data includes converting the bit sequence into the non-binary data by using a weighted sum of each position in the bit sequence.

16. The training method according to claim 11, in, The first layer has binary input, The training method further includes: converting non-binary training data into binary training data by using a first binary converter, and providing the binary training data to the first layer.

17. The training method according to claim 11, further comprising converting the non-binary reference true value into a reference true value bit sequence by using a second binary converter, in, Calculating the loss includes calculating the loss based on the bit sequence and a converted reference true value bit sequence.

18. An electronic device, include: A memory configured to store an image, and weights and activation functions of a binary neural network; as well as at least one processor configured to load the image from the memory and process the image through the binary neural network, wherein the binary neural network includes a first layer, one or more second layers, and a third layer, the one or more second layers are arranged between the first layer and the third layer, and the third layer has a binary input and a binary weight, and is configured to output a binarized bit sequence; as well as The non-binary converter is configured to convert the binarized bit sequence into non-binary data and output the non-binary data.

19. The electronic device according to claim 18, in, The binary neural network is trained based on at least one of a reference true value, the binarized bit sequence, and the non-binary data.

20. The electronic device of claim 18, further comprising a video codec device or an image signal processing device that uses the binary neural network to process the image.

Images