Method and apparatus for performing deep neural network learning

Abstract

The encoding device is connected to a learning circuit that processes and learns a deep neural network, and the encoding device is configured to perform encoding for reconfiguring the connection or disconnection of a plurality of edges in a layer of the deep neural network using an edge sequence, where the edge sequence is generated based on a random number sequence and dropout information, and the dropout information indicates a ratio between connected edges and disconnected edges among the plurality of edges included in a layer of the deep neural network.

Description

Technical Field

[0001] The present disclosure relates to methods and apparatuses for performing deep neural network learning to make inferences using a deep neural network. Background Art

[0002] An artificial intelligence (AI) system is a computer system that implements or attempts to imitate human-level intelligence. Different from conventional rule-based intelligent systems, AI systems learn and make judgments. As the use of AI systems increases, the recognition rate of AI systems becomes higher, for example, more accurately understanding user preferences. Therefore, conventional rule-based intelligent systems are gradually being replaced by deep learning-based AI systems.

[0003] AI technology includes machine learning (deep learning) and elemental technologies that utilize machine learning.

[0004] Machine learning is an algorithmic technology for classifying or learning the features of input data. Elemental technologies are technologies that use machine learning algorithms, such as deep learning, and include technical fields such as language understanding, visual understanding, inference / prediction, knowledge representation, and motion control.

[0005] The various fields to which AI technology is applied are as follows: Language understanding is a technology for recognizing and applying / processing human language / characters and includes natural language processing, machine translation, dialogue systems, query response, speech recognition / synthesis, etc.; Visual understanding is a technology for recognizing and processing objects like human vision and includes object recognition, object tracking, image search, human recognition, scene understanding, spatial understanding, image enhancement, etc.; Inference prediction is a technology for judgment, logical inference, and prediction and includes knowledge / probability-based inference, optimization prediction, preference-based planning, recommendation, etc.; Knowledge representation is a technology for automating human experience information into knowledge data and includes knowledge construction (data generation / classification), knowledge management (data utilization), etc.; Motion control is a technology for controlling vehicle autonomous driving and robot motion and includes motion control (navigation, collision, and driving), operation control (behavior control), etc. Summary of the Invention

[0006] Solution to the Problem

[0007] The inference process using a deep neural network can be used to accurately classify or assign input information. For higher inference accuracy of the deep neural network, a relatively large number of operation processes may be required, thus increasing the number of layers or the depth of the formed deep neural network. As the number of layers of the formed deep neural network increases, the amount of operations required to obtain an inference through the deep neural network increases. Therefore, various methods have been used to reduce the amount of computation while improving the inference accuracy of the deep neural network. For example, a method of performing learning by omitting certain edges, nodes, etc. that make up the layers of the formed deep neural network is used to reduce the amount of operations occurring during the learning process, thereby improving the inference accuracy of the deep neural network.

[0008] However, among the methods of reducing the amount of operations, the process of arbitrarily removing certain edges or nodes that make up a layer will be implemented by software, thus requiring another operation and making it difficult to optimize the deep neural network. Specifically, the above software implementation scheme can be executed by an operating system. Specifically, operations such as memory allocation need to be performed or system calls for random number generation to generate signals for removing certain edges or nodes need to be executed. Therefore, when using the software implementation scheme to reduce the amount of operations related to obtaining an inference using artificial intelligence, due to the execution of complex operations and the computational operations performed, the waiting time increases, and the amount of operations also increases.

[0009] Therefore, a method and apparatus for solving the increase in the amount of operations and the increase in waiting time occurring in the above software implementation scheme may be required.

[0010] Other aspects will be partly described subsequently, and partly will be obvious from the specification, or can be learned through the practice of the embodiments presented in this disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The above and other aspects, features, and advantages of certain embodiments of the present disclosure will become apparent from the following description in conjunction with the drawings, in which:

[0012] Figure 1A is a block diagram showing an encoding device and a learning circuit for generating and processing information required for performing deep neural network learning according to an embodiment of the present disclosure;

[0013] Figure 1B is a view depicting deep neural network operations performed on a learning circuit for processing deep neural network learning according to an embodiment of the present disclosure;

[0014] Figure 2 is a flowchart of an encoding method executed by using an encoding device according to an embodiment of the present disclosure;

[0015] Figure 3is a block diagram showing an encoding device, a learning circuit, and a random number generation circuit according to an embodiment of the present disclosure;

[0016] Figure 4 is a view describing a method of outputting an edge sequence used in the learning process of a deep neural network in an encoding device according to an embodiment of the present disclosure;

[0017] Figure 5 is a flowchart of an encoding device executed by using an encoding device according to another embodiment of the present disclosure;

[0018] Figure 6 is a view describing a method of comparing a first edge sequence with a random number sequence to determine a second edge sequence by using an encoding device according to an embodiment of the present disclosure;

[0019] Figure 7 is a flowchart of a method of comparing a first edge sequence with a random number sequence to determine a second edge sequence according to an embodiment of the present disclosure;

[0020] Figure 8 is a view describing a deep neural network connected to an encoding device according to an embodiment of the present disclosure;

[0021] Figure 9 is a view showing the connection or disconnection state of edges included in each layer of a deep neural network according to an embodiment of the present disclosure;

[0022] Figure 10A shows a process of adjusting the weights of each layer during the learning process executed by a learning circuit according to an embodiment of the present disclosure;

[0023] Figure 10B shows a process of adjusting the weights of each layer during the learning process executed by a learning circuit according to an embodiment of the present disclosure;

[0024] Figure 11 is a block diagram of an encoding device that performs encoding by using the weights of edges determined in a previous operation cycle according to an embodiment of the present disclosure;

[0025] Figure 12 is a flowchart of a process of generating a second edge sequence based on the weights of a plurality of edges stored in a register by an encoding device according to an embodiment of the present disclosure;

[0026] Figure 13 is a block diagram showing an encoding device, a learning circuit, and a counter according to an embodiment of the present disclosure;

[0027] Figure 14is a block diagram showing an encoding device, a learning circuit, and a selector according to an embodiment; and

[0028] Figure 15 is a block diagram showing an encoding device, a learning circuit, a register, a counter, a random number generation circuit, and a selector according to an embodiment of the present disclosure. Detailed Embodiments

[0029] Preferred Modes for Carrying Out the Invention

[0030] According to an embodiment of the present disclosure, there is provided an encoding device including a memory and an encoder. The memory stores a random number sequence generated by a random number generator. The encoder is configured to receive dropout information of a deep neural network, where the dropout information indicates a ratio between connected edges and disconnected edges among a plurality of edges included in a layer of the deep neural network; generate an edge sequence indicating connection or disconnection of the plurality of edges based on the dropout information and the random number sequence; and output the edge sequence for reconfiguring connection or disconnection of the plurality of edges.

[0031] The random number sequence may be based on a clock signal of the random number generator.

[0032] The size of the random number sequence may be determined based on the number of a plurality of edges in a layer of the deep neural network.

[0033] The dropout information, the random number sequence, and the edge sequence may each have a bit width formed by binary numbers.

[0034] The encoder may generate the edge sequence based on a first ratio and a second ratio, where the first ratio is a ratio between bits having a bit value of 0 in the random number sequence and bits having a bit value of 1 in the random number sequence, and the second ratio is a ratio between bits having a bit value indicating a connected edge in the dropout information and bits having a bit value indicating a disconnected edge in the dropout information.

[0035] The encoder may generate the edge sequence based on patterns of bits having a bit value of 0 and bits having a bit value of 1 in the random number sequence, and patterns of bits having a bit value indicating a connected edge and bits having a bit value indicating a disconnected edge in the dropout information.

[0036] The size of the random number sequence may be equal to the number of edges in a layer of the deep neural network.

[0037] The edge sequence may serve as a basis for performing a dropout operation in a layer of the deep neural network.

[0038] The encoder can obtain the weights of multiple edges in a layer of a deep neural network, perform a pruning operation based on the result of comparing the weights with a preset threshold weight, and generate an edge sequence based on the pruning operation to connect or disconnect the multiple edges in the layer of the deep neural network.

[0039] The encoding device may further include a selector configured to select one of multiple types of input signals and output the selected signal. Among them, the encoder receives the operation result from the deep neural network to determine whether an overflow has occurred in the operation result, and performs a dynamic fixed-point operation to modify the representable information range used in the deep neural network based on whether an overflow has occurred.

[0040] According to an embodiment of the present disclosure, there is provided an encoding method executed by an encoding device. The encoding method includes: storing a random number sequence generated by a random number generator; receiving discard information of a deep neural network, where the discard information indicates the ratio between connected edges and disconnected edges among multiple edges included in a layer of the deep neural network; generating an edge sequence indicating the connection or disconnection of the multiple edges based on the discard information and the random number sequence; and outputting the edge sequence for reconfiguring the connection or disconnection of the multiple edges.

[0041] The random number sequence may be based on the clock signal of the random number generator.

[0042] The size of the random number sequence may be determined based on the number of multiple edges in a layer of the deep neural network.

[0043] The discard information, the random number sequence, and the edge sequence may have a bit width formed by binary numbers.

[0044] Generating the edge sequence may include generating the edge sequence based on a first ratio and a second ratio. The first ratio is the ratio between bits with a bit value of 0 and bits with a bit value of 1 in the random number sequence, and the second ratio is the ratio between bits with a bit value indicating a connected edge and bits with a bit value indicating a disconnected edge in the discard information.

[0045] Generating the edge sequence may include generating the edge sequence based on the pattern of bits with a bit value of 0 and bits with a bit value of 1 in the random number sequence, and the pattern of bits with a bit value indicating a connected edge and bits with a bit value indicating a disconnected edge in the discard information.

[0046] The size of the random number sequence may be equal to the amount of edges in a layer of the deep neural network.

[0047] The edge sequence may serve as the basis for the discard operation in a layer of the deep neural network.

[0048] The encoding method may further include: obtaining weights of a plurality of edges in a layer of a deep neural network; performing a pruning operation based on a result of comparing the weights of the plurality of edges with a preset threshold weight; and generating an edge sequence based on the pruning operation to indicate connection or disconnection of the plurality of edges in the layer of the deep neural network.

[0049] The encoding method may further include: receiving an operation result from a deep neural network; determining whether an overflow has occurred in the operation result; and performing a dynamic fixed-point operation to modify a representable range of values used in the deep neural network based on whether the overflow has occurred.

[0050] Embodiments of the invention

[0051] This application is based on and claims priority to Korean Patent Application No. 10-2018-0020005, filed with the Korean Intellectual Property Office on February 20, 2018, the entire contents of which are incorporated herein by reference.

[0052] Reference will now be made in detail to embodiments of the disclosure, examples of which are illustrated in the accompanying drawings, wherein like reference numerals always refer to like elements. In this regard, the embodiments of the disclosure may have different forms and should not be construed as limited to the descriptions set forth herein. Accordingly, the embodiments of the disclosure are described below only by referring to the drawings to explain various aspects. Expressions such as "at least one" when following a list of elements modify the entire list of elements and not individual elements of the list. Throughout the disclosure, the expression "at least one of a, b, or c" only means a only, b only, c only, both a and b, both a and c, both b and c, all of a, b, and c, or variations thereof.

[0053] The present disclosure may be described in terms of functional block components and various processing steps. Some or all of these functional blocks may be implemented by any number of hardware and / or software components configured to perform the specified functions. For example, the functional blocks according to the present disclosure may be implemented using one or more microprocessors or circuit components for certain functions. In addition, the functional blocks according to the present disclosure may be implemented using various programming or scripting languages. Algorithms implemented in software executed on one or more processors may be used to implement the functional blocks. In addition, the present disclosure may employ any number of related technologies for electronic configuration, signal processing, and / or data processing, etc.

[0054] In addition, the connecting lines or connectors shown in the various drawings presented are intended to represent example functional relationships and / or physical or logical couplings between the various elements. It should be noted that many alternative or additional functional relationships, physical connections, or logical connections may exist in an actual device.

[0055] In addition, terms such as "unit" and "module" refer to a unit that performs at least one function or operation, and the unit can be implemented as hardware or software or a combination of hardware and software. However, the "unit" or "module" can also be stored in an addressable storage medium and implemented by a program executable by a processor.

[0056] For example, the "unit" or "module" can be implemented by software components, object-oriented software components, class components and task components, as well as processes, functions, attributes, procedures, subroutines, program code segments, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays, and variables.

[0057] In this document, "inference" is performed in the direction of obtaining output data to be output from the output layer from the input data input to the input layer, and "learning" can be performed in the direction of using the output data of the output layer as the input data input to the input layer.

[0058] In the classification or partitioning of input information by a deep neural network, by inputting input data into the input layer and performing operations through multiple layers forming the hidden layer of the deep neural network, output data in which the input data is classified or assigned or output data corresponding to the input data can be output from the output layer.

[0059] To improve the accuracy of the output data, after outputting the output data, the weight values applied to the multiple layers forming the deep neural network can be adjusted through learning. While adjusting the weights to improve the accuracy of the output data, overfitting may occur. Due to overfitting, the accuracy of the training data can increase, but the output accuracy related to new input data may decrease. To solve the decrease in accuracy caused by overfitting, a dropout operation can be used.

[0060] Hereinafter, an encoding device capable of quickly performing a dropout operation will be described in detail with reference to the accompanying drawings.

[0061] Figure 1A It is a block diagram showing an encoding device 100 that generates and processes information required for performing deep neural network learning according to an embodiment of the present disclosure.

[0062] Refer to Figure 1A , the encoding device 100 includes a memory 110 and an encoding circuit 130 or an encoder. In addition, the encoding device 100 can be connected to a learning circuit 120 that performs deep neural network learning. The encoding device 100 can receive information output during the operation of the learning circuit 120, and can also send information generated in the encoding device 100 to the learning circuit 120.

[0063] The learning circuit 120 can perform operations through a deep neural network including an input layer, a hidden layer, and an output layer. The hidden layer can include multiple layers, for example, a first hidden layer, a second hidden layer, and a third hidden layer.

[0064] Reference will be made Figure 1B to describe the operations of the deep neural network performed in the learning circuit 120.

[0065] Reference Figure 1B , the deep neural network 150 includes an input layer 155, a hidden layer 165, and an output layer 170. In Figure 1B it, the deep neural network 150 performs deep neural network operations on the classification information included in the input data and outputs the information as shown by way of example. Specifically, when the input data is image data, the deep neural network 150 outputs result data including the classification type of the image object included in the image data as output data.

[0066] The multiple layers forming the deep neural network 150 can include multiple nodes that receive data, such as node 175 of the input layer 155. In addition, two adjacent layers are interconnected by multiple edges 177 (such as node 176), as Figure 1B shown. A weight value can be assigned to each node, and the deep neural network 150 can obtain output data based on the operations performed on the input signal and the weight value. For example, node 175 can perform a multiplication operation of the input signal and the weight value to generate the product of the input signal and the weight value as the output value. If the node performing the operation exists in the output layer 170, the output value can be output as the result of the deep neural network 150 in the output layer 170, or if the node performing the operation exists in the hidden layer 165, the output value can be output as an intermediate result in one of the hidden layers 165, and the output value can be transmitted through the edge 177 to another node 176 in the adjacent layer 165.

[0067] Reference Figure 1B to the embodiment of the present disclosure shown, the input layer 155 receives input data, for example, image data 180 including a cat as an image object.

[0068] In addition, reference Figure 1B is made to, the deep neural network 150 can include a first layer 181 (layer 1) formed between the input layer 155 and the first hidden layer, a second layer 182 formed between the first hidden layer and the second hidden layer, a third layer 183 formed between the second hidden layer and the third hidden layer, and a fourth layer 184 formed between the third hidden layer and the output layer 170.

[0069] A plurality of nodes included in the input layer 155 of the deep neural network 150 receive signals corresponding to the image data 180. In addition, the output data 185 corresponding to the image data 180 that has been analyzed by the deep neural network 150 can be output from the output layer 170 through operations in a plurality of layers included in the hidden layer 165. In the illustrated example, an operation for classifying the type of image object included in the input image is performed in the deep neural network 150, and thus the result value "Cat probability: 98%" can be output through the output data. To improve the accuracy of the output data output by the deep neural network 150, learning is performed by iteratively transmitting the analysis result from the output layer 170 to the input layer 155, and the weight values can be iteratively evaluated to improve the accuracy of the output data.

[0070] Return reference Figure 1A , according to an embodiment of the present disclosure, the encoding device 100 may include: a memory 110 and an encoding circuit 130. The memory 110 stores a binary sequence or bitstream having a specific size, and the encoding circuit 130 generates a binary sequence to be output to the learning circuit 120 based on the binary sequence stored in the memory 110. The binary sequence refers to a binary number having a bit size or width equal to or greater than two bits. According to an embodiment of the present disclosure, the learning circuit 120 may perform an inference process on the input information and perform a learning process based on the inference result.

[0071] The above binary sequence may be information indicating whether an edge between nodes of a plurality of layers constituting the deep neural network 150 is connected. Specifically, the binary sequence may be information indicating whether each of a plurality of edges formed in a layer included in the deep neural network 150 is connected or disconnected. For example, referring to Figure 1B , the binary sequence may be information related to an edge sequence indicating whether each of a plurality of edges 177 included in a certain layer (for example, the first layer 181) is connected or disconnected.

[0072] For example, the value 0 included in the binary sequence may indicate the disconnection of a certain edge, and 1 in the binary sequence may indicate the connection of a certain edge. Hereinafter, the binary sequence as information indicating whether an edge in a certain layer included in the deep neural network 150 is connected will be referred to as an edge sequence.

[0073] Referring to Figure 1A, the encoding device 100 may include a memory 110 and an encoding circuit 130. The memory 110 may store a first edge sequence that has been output in a first operation period. The encoding circuit 130 may generate a second edge sequence based on a random number sequence obtained in a second operation period different from the first operation period and the first edge sequence, where the second edge sequence indicates whether a plurality of edges included in a certain layer in the deep neural network are connected or disconnected. The learning circuit 120 may configure the neural network 150 to connect or disconnect each of the plurality of edges in a certain layer based on the second edge sequence generated by using the encoding circuit 130.

[0074] Here, an operation period such as the first operation period or the second operation period may refer to an operation period including a time period from the start of a certain operation in a certain layer in the deep neural network until the end of the operation. In addition, the operation period that occurs first in time may be referred to as the first operation period, and the second operation period may occur after the first operation period. The second operation period may occur immediately after the first operation period without any intermediate operation period, or may not occur immediately after the first operation period in time in the case of one or more intermediate operation periods between the first operation period and the second operation period.

[0075] For example, when the encoding circuit 130 performs learning through the deep neural network 150, the first operation period may be the operation period in which learning is performed in the fourth layer 184, and the second operation period may be the operation period in which learning is performed in the third layer 183.

[0076] According to an embodiment of the present disclosure, in each operation cycle, the encoding circuit 130 may operate according to a certain operation cycle and generate an edge sequence to be output to the learning circuit 120. According to an embodiment of the present disclosure, the encoding circuit 130 may generate a second edge sequence based on a first edge sequence stored in the memory 110, and transmit the second edge sequence to the learning circuit 120 to determine the connection state of the edges in the deep neural network. According to an embodiment of the present disclosure, the first edge sequence may have been generated in an operation cycle different from the operation cycle in which the encoding circuit 130 will generate the second edge sequence, used by the learning circuit 120, and stored in the memory 110. According to an embodiment of the present disclosure, the edge sequences stored in the memory 110 may be stored according to each of the multiple layers included in the deep neural network. For example, the memory 110 may store a first edge sequence output in a first operation cycle that is temporally before a second operation cycle. The first edge sequence may be a binary sequence indicating the connection or disconnection of each of the multiple edges included in a certain layer (e.g., the fourth layer 184) in the deep neural network 150, and the memory 110 may store the first edge sequence as a value related to a certain layer (e.g., the fourth layer 184). In addition, the memory 110 may store each of at least one edge sequence that has been output from operation cycles before the current operation cycle, respectively.

[0077] According to another example, the first edge sequence stored in the memory 110 may be stored as one sequence indicating the connection states of all the edges included in the multiple layers of the deep neural network. Operations executable by the encoding device 100 will be described in detail later with reference to various embodiments of the present disclosure.

[0078] Figure 2 is a flowchart of an encoding method executed by using the encoding device 100 according to an embodiment of the present disclosure.

[0079] In operation S200, the encoding device 100 may store the first data received from the random number generation circuit or the random number generator. The first data may be data including a random number sequence output by the clock signal of the random number generation circuit. Random number sequences will be described in detail with reference to Figure 4 and Figure 5 in detail.

[0080] Meanwhile, the encoding device 100 may not only store the first data received from the random number generation circuit, but also store the first edge sequence output at the first operation cycle of the deep neural network. According to an embodiment of the present disclosure, the first edge sequence may have been used to disconnect at least one of the multiple edges formed between the layers of the deep neural network during the learning process of the learning circuit 120 performed in the first operation cycle.

[0081] In operation S202, the encoding device 100 may receive dropout information of the deep neural network from the learning circuit 120. The dropout information may refer to the dropout ratio of the edge sequence.

[0082] In operation S204, the encoding device 100 may generate second data by using the dropout information and the first data. The second data may indicate a second edge sequence regarding a second operation period. The second edge sequence may have values equal to the random number sequence, but may also be a new type of edge sequence different from other edge sequences. Specifically, the second edge sequence may also be generated by correcting at least one random number value included in the random number sequence based on the first edge sequence. The random number sequence may be sent only from an external device (e.g., a random number generation circuit). However, the random number generation circuit is not limited to the above example and may also be integrated within the encoding device 100. Similarly, the learning circuit 120 may also be integrated within the encoding device 100.

[0083] For example, the encoding circuit 130 according to an embodiment of the present disclosure may perform an operation of comparing the first edge sequence with the random number sequence so that the learning process of the second operation period is not performed by using the same edge sequence as the first edge sequence used in the learning process of the first operation period of the learning circuit 120.

[0084] Alternatively, the encoding circuit 130 according to an embodiment of the present disclosure may generate a second edge sequence based on the random number sequence and the first edge sequence such that the dropout rate of each of the first edge sequence and the second edge sequence is maintained at a constant value. Here, the dropout rate may refer to the ratio between the connected edges and the disconnected edges of the multiple edges included in a certain layer. For example, when the value 0 included in the second edge sequence indicates the disconnection of an edge and the value 1 indicates the connection of an edge, the dropout rate may represent the ratio between the number of bits having the value 0 and the total number of bits included in the second edge sequence.

[0085] Generating the second edge sequence based on the first edge sequence and the random number sequence will be described with reference to an embodiment of the present disclosure.

[0086] In operation S206, the encoding device 100 may output the second data to the learning circuit 120 so that the second data is assigned to the neurons of the deep neural network.

[0087] Figure 3 is a block diagram showing an encoding device 300, a learning circuit 320, and a random number generation circuit 340 according to an embodiment of the present disclosure. Figure 3 The encoding device 300, the memory 310, and the encoding circuit 330 of Figure 1AEncoding device 100, memory 110, and encoding circuit 130. In addition, Figure 3 The learning circuit 320 may correspond to Figure 1A The learning circuit 120. That is, compared with Figure 1A The encoding device 100 shown, Figure 3 The encoding device 300 may further include a random number generation circuit 340.

[0088] According to an embodiment of the present disclosure, the random number generation circuit 340 continuously generates a plurality of random numbers. The plurality of random numbers output from the random number generation circuit 340 are in a sequence form and are generated successively in time. Therefore, the plurality of random numbers generated in the random number generation circuit 340 can be referred to as a random number sequence.

[0089] Specifically, the random number generation circuit 340 according to an embodiment of the present disclosure may generate a random number based on a clock signal as a register including a plurality of register units. Each of the plurality of register units may be formed as a flip-flop. In addition, the random number generation circuit 340 may store the generated random number. The random number generation circuit 340 may store a random number sequence including at least one random number generated in each clock cycle, and send the stored random number sequence to the encoding circuit 330 according to the clock signal. According to an embodiment of the present disclosure, in order to prevent a random number sequence with equal values or the same pattern from being generated in the random number generation circuit 340, the random number generation circuit 340 may include at least one logic circuit gate connected to the flip-flop.

[0090] The encoding circuit 330 may receive the random number sequence generated in the random number generation circuit 340, and may determine whether to output the received random number sequence as a second edge sequence identical to the received random number sequence, or process the received random number sequence and output the processed random number sequence as the second edge sequence.

[0091] Figure 4 Is a view for describing a method of outputting an edge sequence to be used in the learning process of a deep neural network in an encoding device according to an embodiment of the present disclosure.

[0092] The random number generation circuit 410 according to an embodiment of the present disclosure may include a linear feedback shift register (LFSR) that generates a random number sequence.

[0093] For example, the random number generation circuit 410 may include a shift register 411. The shift register 411 includes a plurality of register units and at least one exclusive OR (XOR) gate 412, 413, and 414 connected to the shift register 411. The at least one XOR gate may perform an exclusive OR operation on the input value and output at least the operated output value to the input terminal of the shift register 411. Therefore, the random number generation circuit 410 may generate a random number sequence having values that continuously change over time. In addition, the random number generation circuit 410 may include various types of hardware components for generating random numbers.

[0094] The random number generation circuit 410 may shift the bit values stored in the shift register 411 in response to an input clock signal and generate a random number sequence. Specifically, the random number generation circuit 410 may shift the bit values stored in the shift register 411 at the rising edge or the falling edge of the clock signal and generate a random number sequence.

[0095] In addition, the random number sequence may include a plurality of random numbers obtained in a first clock period. The first clock period may refer to the period of the clock signal required to generate a random number sequence of a specific size. For example, when the random number generation circuit 410 shifts the bit values stored in the shift register 411 at each of the rising edge and the falling edge of the clock signal, 2-bit random numbers may be generated from one clock period. Therefore, in order to generate a random number sequence of 100-bit size, 50 clock periods are required, and in this case, the first clock period may include a time section including 50 clock periods.

[0096] Here, the size of the random number sequence may be determined based on the number of edges included in a certain layer of the deep neural network 150. For example, when the number of edges formed in a certain layer (e.g., the first layer 181) of the deep neural network 150 is 100, a random number sequence of 100-bit size may be formed, and the bit values included in the 100 bits may each have information corresponding to the connection or disconnection of the 100 edges. In addition, the size of the edge sequence (including the first edge sequence) generated by the encoding circuit 426 may be determined based on the number of edges included in a certain layer of the deep neural network 150 and may be the same as the size of the random number sequence.

[0097] However, the random number generation circuit 410 is merely an example of an element that generates random numbers by using hardware components that operate according to a clock signal, and thus may include various hardware components for generating random numbers.

[0098] Figure 5 It is a flowchart of an encoding method performed by using an encoding device according to another embodiment of the present disclosure.

[0099] In operation S500, the memory 422 of the encoding device 420 may store first data received from the random number generation circuit 410. The first data may be data including a random number sequence output by the clock signal of the random number generation circuit 410. In addition, the encoding device 420 may store the first edge sequence that has been output in the first operation cycle.

[0100] According to an embodiment of the present disclosure, the size of the random number sequence generated in the random number generation circuit 410 may be the same as the size of the first edge sequence. According to an embodiment of the present disclosure, the first edge sequence may be information indicating the connection state achieved during the learning process of a plurality of edges included in a certain layer in the first operation cycle. That is, the first edge sequence may be information indicating whether the edges included in a certain layer are connected or disconnected during the previous learning process, and has a bit width equal to the number of edges included in a certain layer.

[0101] The number of bits that can be generated in the random number generation circuit 410 according to an embodiment of the present disclosure may have a bit width greater than the maximum number of edges in each layer of the deep neural network used in the learning circuit 424. According to an embodiment of the present disclosure, the random number generation circuit 410 may generate a binary sequence having a size corresponding to the number of edges in a certain layer during the process of generating the random number sequence to be sent to the encoding circuit 426. According to another embodiment of the present disclosure, based on the number of edges included in a certain layer, the encoding circuit 426 may use a part of the binary sequence generated in the random number generation circuit 410 as the random number sequence to be compared with the first edge sequence.

[0102] In operation S502, the encoding device 420 may receive dropout information of the deep neural network from the learning circuit 424.

[0103] In operation S504, the encoding device 420 may generate second data by using the dropout information and the first data. In other words, the encoding device 420 may generate a second edge sequence based on the random number sequence and the first edge sequence, and the second edge sequence indicates the connection or disconnection of a plurality of edges included in a certain layer in the deep neural network.

[0104] In operation S506, the encoding device 420 may output the second data to the learning circuit 424 for assignment to the neurons of the deep neural network. That is, the encoding device 420 may output the second edge sequence to the learning circuit 424, and thus, the learning circuit 424 may connect or disconnect each of the plurality of edges in a certain layer based on the second edge sequence and perform learning through the layer.

[0105] Specifically, the encoding circuit 426 may generate a second edge sequence such that the ratio between the number of connected edges and the number of disconnected edges in each layer of all the layers included in the deep neural network 150 is constant.

[0106] Reference Figure 1B Referring to the deep neural network 150 shown, the following example is provided, where the first edge sequence indicates the connection or disconnection of each of the plurality of edges included in the third layer 183, and the second edge sequence includes a segment indicating the connection or disconnection of each of the plurality of edges included in the second layer 182. In addition, the first ratio value is the ratio of the number of bits having a value of 0 to the total number of bits in the first edge sequence, which may indicate the ratio between the number of edges included in the third layer 183 and the number of disconnected edges. In addition, the second ratio value is the ratio of the number of bits having a value of 0 to the total number of bits in the second edge sequence, which may indicate the ratio between the number of edges included in the second layer 182 and the number of disconnected edges. When the first ratio value is 40% and the target value of the ratio of disconnected edges among all the edges included in the deep neural network 150 is 50%, the encoding circuit 426 may process the random number sequence such that the second ratio value of the second edge sequence is 60%, so that the ratio of disconnected edges among all the edges is 50%. The target ratio value may be set by the user or by the learning circuit 424 itself.

[0107] Alternatively, the encoding circuit 426 may also generate a second edge sequence such that the output edge sequence has a target value of a constant equal ratio value.

[0108] For example, when the target value is set to 50%, the encoding circuit 426 may generate a second edge sequence such that the ratio value of the number of bits having a value of 0 in the output second edge sequence is 50%. In this case, the first edge sequence may be compared with the random number sequence to correct the random number sequence to have a pattern different from that of the first edge sequence, and the corrected random number sequence may be generated as the second edge sequence. For example, when the random number sequence is 1000100111 and the previously output first edge sequence is 1000100111, the ratio value of each of the random number sequence and the first edge sequence is 50%, that is, equal values. Therefore, the encoding circuit 426 may generate 0111011000 obtained by performing an inverse operation (NOT) on the random number sequence as the second edge sequence, so that the first edge sequence and the second edge sequence have different patterns.

[0109] As described above, when the number of disconnected edges relative to the total number of edges in each layer forming the deep neural network 150 is defined as the "ratio value" or "dropout ratio", the operation of uniformly adjusting the ratio values in the multiple layers included in the deep neural network 150 can be referred to as a balancing operation. As in the above example, the encoding circuit 426 can perform a balancing operation to generate a second edge sequence.

[0110] In operation S508, the encoding device 420 can connect or disconnect each of the multiple edges in a certain layer based on the second data and perform learning through the layer.

[0111] Figure 6 It is a view for describing a method of determining a second edge sequence by comparing a first edge sequence with a random number sequence using an encoding device according to an embodiment of the present disclosure.

[0112] The memory 602, learning circuit 604, and encoding circuit 606 according to an embodiment of the present disclosure can respectively correspond to Figure 1A the memory 110, learning circuit 120, and encoding circuit 130.

[0113] Figure 7 It is a flowchart of a method of determining a second edge sequence by comparing a first edge sequence with a random number sequence according to an embodiment of the present disclosure.

[0114] In operation S700, according to an embodiment of the present disclosure, the memory 602 can store the first edge sequence that has been output in the first operation cycle.

[0115] In operation S702, according to an embodiment of the present disclosure, the encoding circuit 606 can determine whether the random number sequence and the first edge sequence are within the same range. The random number sequence can include at least one random number sequence obtained at the first clock cycle.

[0116] According to an embodiment of the present disclosure, the encoding circuit 606 may determine whether the binary number and the random number sequence constituting the first edge sequence are the same (S702). In operation S702, the term "same" may mean that the two sequences have substantially the same pattern to have equal binary values. In addition, the term "same" may mean that the values 0 or 1 included in the random number sequence or the first edge sequence have an "equal ratio". In addition, the term "same" may be determined based on whether the number of bits having different values between the first edge sequence and the second edge sequence is equal to or less than a preset threshold. For example, when the number of bits having different values between the first edge sequence and the second edge sequence is 10% or less of the total number of bits, it may be determined that the random number sequence and the first edge sequence have values within the same range. That is, in order to determine whether a second edge sequence identical to the random number sequence is generated, the encoding circuit 606 may determine whether the ratio of the values 0 or 1 included in the sequence is within the same range, even if the random number sequence and the first edge sequence are exactly the same or not exactly the same. According to an embodiment of the present disclosure, the encoding circuit 606 may include a comparator that compares the random number sequence with the first edge sequence.

[0117] The encoding circuit 606 may generate a second random number sequence based on the determination result of operation S702 (S703).

[0118] Specifically, according to an embodiment of the present disclosure, when the first edge sequence and the random number sequence are within the same range (S702 - Yes), in operation S704, the encoding circuit 606 may process the random number sequence based on the first edge sequence and generate a second edge sequence that is not within the same range as at least one of the first edge sequence and the random number sequence.

[0119] Regarding the embodiments of the present disclosure to be described below, including Figure 6 the embodiment, an example will be described in which the number of edges in a certain layer included in the deep neural network 150 is 10 and the edge sequence has a binary 10-bit size.

[0120] Referring to Figure 6 , according to an embodiment of the present disclosure, when the edge sequence for connecting or disconnecting an edge in a certain layer in the first operation cycle is 1010101010, the memory 602 may store the binary number sequence 1010101010 as the first edge sequence.

[0121] According to an embodiment of the present disclosure, the random number generation circuit 608 may generate a random number sequence including a plurality of random numbers based on the number of edges in a certain layer. The encoding circuit 606 may compare the random number sequence with a first edge sequence stored in the memory 602. According to an embodiment of the present disclosure, after comparing the random number sequence with the first edge sequence, when it is determined that the two sequences are not within the same range, the encoding circuit 606 may generate a binary number sequence identical to the random number sequence as the second edge sequence.

[0122] According to an embodiment of the present disclosure, in order to determine whether the first edge sequence and the random number sequence are included within the same range, the encoding circuit 606 may compare each bit of the first edge sequence with each bit of the random number sequence to determine the number of different bits. When the number of bits having different values between the first edge sequence and the random number sequence is equal to or greater than a preset threshold, the encoding circuit 606 may determine that the first edge sequence and the random number sequence are not within the same range.

[0123] Reference Figure 6 , for example, when the threshold is 3, the encoding circuit 606 may compare the first edge sequence with the random number sequence based on the threshold 3, and since four bits among each of the corresponding bits in the first edge sequence (1010101010) and the random number sequence (0010100001) have different values, and the number of different bit values of the corresponding bits in the first edge sequence and the random number sequence is greater than 3, the first edge sequence and the random number sequence can be determined not to be within the same range. In this case, the encoding circuit 606 may determine the random number sequence (0010100001) as the second edge sequence.

[0124] According to another embodiment of the present disclosure, the encoding circuit 606 may compare the first edge sequence with the random number sequence based on the threshold 3. Referring to Figure 6 , only one value of each corresponding bit of each of the first edge sequence (1010101010) and the random number sequence (1010101110) is different. Therefore, the number of different bit values of the corresponding bits in the first edge sequence and the random number sequence is equal to or less than the threshold 3, and thus the encoding circuit 606 may determine that the first edge sequence and the random number sequence are within the same range. In this case, the encoding circuit 606 may generate a new edge sequence that is not within the same range as the first edge sequence by processing the random number sequence (1010101110), and may generate the processed edge sequence as the second edge sequence and output the second edge sequence.

[0125] For example, the encoding circuit 606 may perform a bit inversion operation (NOT operation) on a random number sequence to generate a new edge sequence by processing a random number sequence included within the same range as the first edge sequence. Refer to Figure 6 , the encoding circuit 606 may change the binary values of bits having a value of 0 in the random number sequence (1010101110) to 1 and change 1 to 0 to generate a second edge sequence (0101010001). However, the encoding circuit 606 that performs a bit inversion operation on the random number sequence is merely an embodiment of the present disclosure for describing a method of processing a random number sequence, and various methods may be executable such that the method of processing the random number sequence performed by the encoding circuit 606 includes a random number sequence that is not within the same range as the first edge sequence.

[0126] According to an embodiment of the present disclosure, the encoding circuit 606 may generate a second edge sequence different from the first edge sequence in a second operation period based on a first ratio value and a second ratio value, where the first ratio value is the ratio of bit values 0 and 1 included in the random number sequence, and the second ratio value is the ratio of bit values 0 and 1 included in the first edge sequence. The "ratio value" may be defined as the ratio of bit values in a sequence of a certain size. According to an embodiment of the present disclosure, the encoding circuit 606 may determine whether the first ratio value and the second ratio value are within the same range. According to an embodiment of the present disclosure, when the first ratio value and the second ratio value are not within the same range, the encoding circuit 606 may generate a second edge sequence identical to the random number sequence; when the first ratio value and the second ratio value are included within the same range, the encoding circuit 606 may process the random number sequence to generate a second edge sequence that is not within the same range as the first edge sequence.

[0127] According to an embodiment of the present disclosure, when the difference between the first ratio value and the second ratio value is equal to or less than 20%, the first ratio value and the second ratio value may be determined to be within the same range. For example, when the first ratio value is 30% and the second ratio value is less than 10% and greater than 50%, the second ratio value is not included within the same range as the first ratio value, and thus, the encoding circuit 606 may generate a second edge sequence identical to the random number sequence. On the other hand, when the first ratio value is 30% and the second ratio value is equal to or greater than 10% and equal to or less than 50%, the second ratio value is included within the same range as the first ratio value, and thus the encoding circuit 606 may process the random number sequence to generate a second edge sequence that is not within the same range as the first edge sequence. The encoding circuit 606 may perform an operation on the first ratio value (e.g., addition, subtraction, or multiplication of a certain value to the first ratio value) to process the first edge sequence and may generate a second edge sequence based on the operation result.

[0128] According to an embodiment of the present disclosure, the encoding circuit 606 may generate a second edge sequence different from the first edge sequence in a second operation period based on the patterns of the bit values 0 and 1 included in the random number sequence and the patterns of the bit values 0 and 1 included in the first edge sequence. According to an embodiment of the present disclosure, the encoding circuit 606 may determine whether the bit values 0 and 1 included in the random number sequence and the first edge sequence are configured in a specific pattern (e.g., repeating at least one binary number at a specific interval). According to an embodiment of the present disclosure, the encoding circuit 606 may determine whether the random number sequence and the first edge sequence have a specific pattern. When the random number sequence and the first edge sequence are determined to be within the same range, the encoding circuit 606 may process the random number sequence to determine a second edge sequence that is not within the same range as the first edge sequence. The method of processing the random number sequence performed by the encoding circuit 606 may be the method described by referring to various methods in the present disclosure.

[0129] According to an embodiment of the present disclosure, the encoding circuit 606 may process the random number sequence by using one of various processing methods that can make the random number sequence not within the same range. According to an embodiment of the present disclosure, the execution results of some of the various processing methods that can be performed by the encoding circuit 606 may be within the same range as the first edge sequence. The encoding circuit 606 may process the first edge sequence by selecting some other processing methods other than those processing methods having results included within the same range as the first edge sequence to determine the second edge sequence.

[0130] According to an embodiment of the present disclosure, when it is determined that the first edge sequence and the random number sequence are not within the same range (S702 - No), in operation S706, the encoding circuit 606 may determine the random number sequence as the second edge sequence.

[0131] According to an embodiment of the present disclosure, in operation S708, the encoding circuit 606 may output the second edge sequence to the learning circuit 604. Therefore, the learning circuit 604 may connect or disconnect each of the multiple edges in a certain layer based on the second edge sequence and perform learning through the certain layer.

[0132] Figure 8 is a view of the deep neural network 800 connected to the encoding device according to an embodiment of the present disclosure. In Figure 8 the learning circuit 120 may operate through the deep neural network 800. The deep neural network 800 corresponds to Figure 1B the deep neural network 150 shown, so the repeated description of the embodiments regarding Figure 1B will be omitted.

[0133] According to an embodiment of the present disclosure, the deep neural network 800 may be implemented by the learning circuit 120, and the learning circuit 120 may include various processors, including a central processing unit (CPU), a graphics processing unit (GPU), a neural processing unit (NPU), a neural network processor (NNP), etc. That is, the learning circuit 120 may be an element corresponding to a deep neural network implemented by including hardware components such as semiconductors.

[0134] According to an embodiment of the present disclosure, the learning circuit 120 may be manufactured in the form of dedicated hardware for implementing a deep neural network for AI. In addition, a part of the data processing process of the learning circuit 120 disclosed in various embodiments of the present disclosure may be implemented by another dedicated hardware component for a deep neural network. In addition, a part of the data processing process of the learning circuit 120 may be processed by a general-purpose processor (e.g., a CPU or an application processor) or a part of a processor that can only process graphics (e.g., a GPU). When data is processed by a general-purpose processor or a processor that can only process graphics, the data may be processed by at least one software module, and the at least one software module may be provided by an operating system (OS) or an application.

[0135] According to an embodiment of the present disclosure, the deep neural network 800 may include deep neural networks with various configurations, including a convolutional neural network (CNN), a recurrent neural network (RNN), etc. That is, according to an embodiment of the present disclosure, the learning circuit 120 may be a processor that implements various forms of deep neural networks including multiple hidden layers. In particular, CNNs are widely used for image recognition, inference, and classification, and RNNs are widely used for learning sequential data such as speech, music, strings, and moving images.

[0136] Referring to Figure 8 , the deep neural network 800 may include a convolutional layer 805 and a fully connected layer 810. The convolutional layer 805 generates a feature map by performing a convolution operation based on a filter kernel for an input signal, and the fully connected layer 810 performs an inference or classification process on multiple feature maps generated as a result of the convolution operation. The convolution operation performed in the convolutional layer 805 requires a large amount of computation because the convolution operation is performed by moving multiple filter kernels a certain distance over the input of each layer, and a large storage capacity is required in the process of inferring or classifying the feature map generated by the convolutional layer 805. Therefore, when learning is performed on the convolutional layer 805 and the fully connected layer 810, the amount of operation and storage capacity can be effectively reduced by removing duplicate weights or edges or by performing scaling without significantly reducing the accuracy of inference. Therefore, not only in Figure 8In the CNN shown, and also in the RNN including only fully connected layers, the weights and edges used in the networks of various embodiments of the present disclosure can be effectively controlled. Examples of methods for controlling the weights and edges used in a deep neural network according to an embodiment of the present disclosure may include dropout, pruning, dynamic fixed-point methods, but are not limited thereto.

[0137] According to an embodiment of the present disclosure, the inference direction 820 and the learning direction 822 of the deep neural network 800 may be opposite to each other. That is, the learning circuit 120 may perform a forward inference process of performing inference in the direction from the input layer 830 to the output layer 850 to perform an inference process on an input through the deep neural network 800. According to an embodiment of the present disclosure, for a more accurate result of the inference performed based on the inference direction 820, a backward learning process may be performed, in which learning may be performed in the direction from the output layer 850 to the input layer 830. According to an embodiment of the present disclosure, through the backward learning process performed, the learning circuit 120 may sequentially perform an operation of adjusting weights or edges from a layer close to the output end to a layer close to the input end.

[0138] Figure 9 is a view showing the connection or disconnection state of the edges included in each layer of the deep neural network 900 according to an embodiment of the present disclosure.

[0139] As Figure 9 shown, in order to perform the dropout operation, some of the connection edges in each layer among all the connection edges are disconnected.

[0140] According to an embodiment of the present disclosure, the learning circuit 120 may connect or disconnect a plurality of edges included in the layer 910 in the deep neural network 900 based on a second edge sequence. The edge sequence used by the learning circuit 120 according to an embodiment of the present disclosure indicates whether the nodes of a certain layer are connected, and may be a binary number arranged in a certain order. For example, the edge sequence may determine whether the edges of the layer 910 are connected based on the learning direction 922 of the deep neural network 900, and the edge sequence of the layer 910 may be aligned in the following order. Each column in the following table may represent each bit of the edge sequence. The bit having a value of 0 in the following table may indicate that there is no edge or the edge is disconnected between the nodes indicated in the column. The bit having a value of 1 in the following table may indicate that there is an edge or the edge is connected between the nodes indicated in the column.

[0141]

[0142] In the above table, "c_1→b_1" means an edge connecting the node c_1 to the node b_1. In the above edge sequence, when the value of the space "c_1→b_1" is 0, the edge connecting the node c_1 to the node b_1 may be disconnected, asFigure 9 as shown. In addition, Figure 9 "c_1→b_4" shown in the figure represents the edge connecting node c_1 to node b_4. In the above edge sequence, when the value of the space "c_1→b_4" is 1, the edge connecting node c_1 to node b_4 is connectable, as Figure 9 shown.

[0143] However, the above method of aligning the edge sequence is merely an example of determining the order in which the learning circuit 120 is to connect or disconnect the edges, and thus may include various methods of aligning a binary number sequence, where the binary number sequence can be generated to easily determine the edge connection states of multiple nodes.

[0144] According to an embodiment of the present disclosure, the learning circuit 120 may determine that some of the edges included in layer 910 are in a disconnected state based on the obtained second edge sequence. As in the above example, since the edge sequence includes information related to the connection or disconnection of the edges included in a certain layer, the learning circuit 120 may disconnect at least some of the edges included in at least one layer or all layers of the deep neural network based on the edge sequence output from the encoding circuit 130. In addition, the above dropout operation may be performed by a deep neural network corrected based on the edge sequence.

[0145] In addition, referring to Figure 9 , the learning circuit 120 may disconnect all the edges connected to node c_3 during the learning process of layer 910 based on the second edge sequence. Therefore, the learning of layer 910 can be performed when both the information input to node c_3 and the information output from node c_3 are blocked. Referring to Figure 9 , the learning circuit 120 may perform learning when all the edges of nodes b_2, b_5, c_3, and c_5 in layer 910 are disconnected, and may perform dropout by disconnecting all the edges of nodes b_2, b_5, c_3, and c_5. In addition, the learning circuit 120 may perform learning by not only disconnecting the edges of nodes b_2, b_5, c_3, and c_5, but also disconnecting some of the edges connected to other nodes in layer 910.

[0146] Figure 10A shows the process of adjusting the weights of each layer during the learning process performed by the learning circuit 120 according to an embodiment of the present disclosure. Figure 10B shows the process of adjusting the weights of each layer during the learning process performed by the learning circuit 120 according to an embodiment of the present disclosure.

[0147] Referring to Figure 10A, the learning circuit 120 according to an embodiment of the present disclosure may perform an inference process by using multiple layers including a first layer 1010 and a second layer 1012, and may perform a learning process in a direction in which the accuracy of an output value as an inference result increases. According to an embodiment of the present disclosure, the learning process performed by the learning circuit 120 may be in a direction opposite to the method performed during the inference process. Therefore, the learning on the second layer 1012 starts after the learning on the first layer 1010 ends.

[0148] According to an embodiment of the present disclosure, the learning circuit 120 may connect or disconnect multiple edges of the second layer 1012 included in the deep neural network 1000 based on a second edge sequence. The edge sequence used by the learning circuit 120 according to an embodiment of the present disclosure indicates whether each node is connected, and may be a binary number arranged in a specific order. Refer to Figure 10A , the learning circuit 120 may omit some of the edges used in connecting the nodes (nodes b_1, b_2, …, b_n, c_1, c_2, …, c_n) of the second layer 1012 based on the second edge sequence generated by the encoding circuit 130. The learning circuit 120 may perform a learning process in a direction in which the inference accuracy of the deep neural network increases by using the edges that are not omitted during the learning process, and may modify the weights assigned to the edges that are not omitted during the learning process. That is, the learning circuit 120 may perform a learning process of modifying the weights of the second layer 1012 based on the edge connection state determined according to the second edge sequence.

[0149] According to an embodiment of the present disclosure, when the learning circuit 120 finishes the learning process in the second layer 1012, the learning process of the first layer 1010 may start. Refer to Figure 10A , the learning circuit 120 may perform a learning process of modifying the weights of the second layer 1012 based on the second edge sequence generated by using the encoding circuit 130 during a second operation period, and thereby may modify the second layer 1012 into Figure 10B the trained second layer 1022 in . The trained second layer 1022 may affect the learning process of the first layer 1020. Therefore, the first layer 1020 may perform a learning process of modifying the weights included in the first layer 1020 while considering the weights of the second layer 1022.

[0150] According to an embodiment of the present disclosure, the learning circuit 120 may perform an inference process based on weights of multiple layers of a deep neural network determined through learning performed in a second operation cycle. To improve the accuracy of the result value output as an inference result after the second operation cycle, the learning circuit 120 may iteratively repeat the learning process according to the above-described embodiment of the present disclosure. Accordingly, the encoding device 100 may store a second edge sequence related to a layer at the second operation cycle in the memory 110, perform an inference process based on the trained weights according to the second edge sequence, and then obtain a third edge sequence from the memory 110 at a third operation cycle different from the second operation cycle. The third edge sequence may correspond to the edge sequence stored in the memory 110 at the third operation cycle. Accordingly, the encoding circuit 130 may generate, at a fourth operation cycle occurring after the third operation cycle, a connection or disconnection of multiple edges included in a certain layer included in the deep neural network based on a random number sequence newly generated in response to the third edge sequence and a clock signal.

[0151] The first edge sequence, the second edge sequence, the third edge sequence, and the fourth edge sequence used by the encoding device 100 according to an embodiment of the present disclosure correspond to the edge sequences generated and output to the learning circuit 120 in each operation cycle of the encoding circuit 130, respectively, and may be generated and processed according to the above various embodiments of the present disclosure.

[0152] Figure 11 is a block diagram of an encoding device 1100 for performing encoding by using weights of edges determined in a previous operation cycle according to an embodiment of the present disclosure.

[0153] According to an embodiment of the present disclosure, the encoding device 1100 includes a memory 1110 and an encoding circuit 1120. The memory 1110 stores a first edge sequence indicating connection states of multiple edges constituting a certain layer determined in a first operation cycle. The encoding circuit 1120 generates a second edge sequence at a second operation cycle different from the first operation cycle. The second edge sequence is a set of binary numbers of connection states of multiple edges. The learning circuit 1130 may perform an inference and learning process based on the edge sequence determined by the encoding circuit 1120. In addition, the register 1140 may store weights of multiple edges constituting a certain layer.

[0154] According to an embodiment of the present disclosure, Figure 11 The memory 1110, the encoding circuit 1120, and the learning circuit 1130 of Figure 1A may be respectively similar to the memory 110, the encoding circuit 130, and the learning circuit 120 of

[0155] According to an embodiment of the present disclosure, the memory 1110 and the register 1140 perform similar functions of storing the edge sequence and edge weights indicating the connection state of the edges. Therefore, the features of the memory 1110 and the register 1140 can be implemented by a single component performing the storage function or multiple separate memories. However, for ease of explanation, the memory 1110 and the register 1140 will be described separately.

[0156] Figure 12 is a flowchart of a process in which the encoding device 1100 generates a second edge sequence based on a plurality of edge weights stored in the register 1140 according to an embodiment of the present disclosure.

[0157] In operation S1200, according to an embodiment of the present disclosure, the register 1140 of the encoding device 1100 may store, in a first operation cycle, a first edge weight including the weights of a plurality of edges constituting a certain layer. According to an embodiment of the present disclosure, the first edge weight stored in the register 1140 may be information including the edge weights of a certain layer.

[0158] In operation S1202, the encoding circuit 1120 of the encoding device 1100 may compare the magnitude of the first edge weight stored in operation S1200 with the magnitude of a preset threshold weight. According to an embodiment of the present disclosure, the encoding circuit 1120 may include a comparator for comparing the first edge weight with the preset threshold weight.

[0159] In operation S1204, based on the comparison result of operation S1202, the encoding circuit 1120 of the encoding device 1100 may generate a second edge sequence, in which an edge having a weight included in the first edge weight and greater than the threshold weight indicates a connection, and an edge having a weight equal to or less than the threshold weight indicates a disconnection. According to an embodiment of the present disclosure, the second edge sequence may be a binary number sequence, and the value 1 may be information indicating the connection of an edge, and the value 0 may be information indicating the disconnection of an edge.

[0160] In operation S1206, the learning circuit 1130 may connect or disconnect each of a plurality of edges in a certain layer based on the second edge sequence received from the encoding circuit 1120, and perform learning through that layer. The weights of the plurality of edges determined as the learning result are stored again in the register 1140 and may be used in the next operation cycle to be compared with the preset threshold weight. According to an embodiment of the present disclosure, the features of the operation performed by the learning circuit 1130 in operation S1206 may be features similar to those of the learning circuit described with reference to various embodiments of the present disclosure. Therefore, a detailed description thereof will be omitted.

[0161] According to an embodiment of the present disclosure, the encoding device 1100 may perform a learning process of a deep neural network by using an edge sequence indicating a first edge weight stored in the register 1140 and an edge connection state of a certain layer. According to an embodiment of the present disclosure, the learning circuit 1130 may perform a learning process by using a second edge sequence generated based on a result of comparing a first edge sequence stored in the memory 1110 with a random number sequence obtained from a random number generation circuit. The first edge weight determined according to the learning result may be stored in the register 1140, and in the next operation cycle, the encoding circuit 1120 may generate a new edge weight by comparing the first edge weight with a preset threshold weight. The learning circuit 1130 may perform a learning process by using the edge sequence generated in the above process.

[0162] Figure 13 FIG. is a block diagram showing an encoding device 1300, a learning circuit 1330, and a counter 1340 according to an embodiment of the present disclosure.

[0163] According to an embodiment of the present disclosure, Figure 13 The features of the memory 1310, the encoding circuit 1320, and the learning circuit 1330 of may be similar to those of the memory 110, the encoding circuit 130, and the learning circuit 120 described above, and thus detailed descriptions thereof will be omitted. Hereinafter, features of using weights stored in registers will be described. Figure 1A

[0164] According to an embodiment of the present disclosure, the counter 1340 may count the number of times a certain condition is satisfied during the process in which the encoding circuit 1320 generates a second edge sequence. According to an embodiment of the present disclosure, the counter 1340 may include a counter that counts the number of occurrences and restarts counting when the maximum count of the number of occurrences is reached, where the counter may be an element designed to change the state of a flip - flop in a predetermined order and includes a register whose state changes in a predetermined order according to an input pulse.

[0165] For example, the counter 1340 may count the number of times the bits of the first edge sequence and the bits of the random number sequence are the same in the process where the encoding circuit 1320 compares the first edge sequence with the random number sequence. Based on the calculation result of the counter 1340, the encoding circuit 1320 may use the random number sequence without any change to determine whether to generate a second edge sequence, or generate a second edge sequence by processing the random number sequence. In addition, the counter 1340 may generate a second edge sequence by calculating the ratio (discard rate) of the bit value 0 included in the first edge sequence and the ratio of the bit value 0 included in the random number sequence. The process of generating a second edge sequence by using the discard rate has been described above with reference to various embodiments of the present disclosure, and thus the detailed description thereof will be omitted.

[0166] According to another example, the encoding circuit 1320 may compare the first edge weight of a certain layer stored in Figure 11 the register 1140 or the memory 1110 with a preset threshold weight, and the counter 1340 may calculate the number of edges among the multiple edges constituting a certain layer that have a weight greater than the preset threshold weight. The encoding circuit 1320 may determine the number of edges in a connected state (or the number of edges in a disconnected state) among the multiple edges of a certain layer based on the calculation result of the counter 1340, and may generate a second edge sequence based on the determined number of edges. The method of generating a second edge sequence by the encoding circuit 1320 based on the discard rate determined according to the calculation result of the counter 1340 has been described above with reference to various embodiments of the present disclosure, and thus the detailed description thereof will be omitted.

[0167] That is, the counter 1340 that can be used in various embodiments of the present disclosure can be widely used to count the number of times that specific conditions are met in the process of comparing the first edge sequence and the random number sequence, the first edge weight, etc. Examples of the counter 1340 may include counters implemented in various flip-flop structures, such as asynchronous counters, synchronous counters, UP counters, etc.

[0168] Figure 14 is a block diagram showing an encoding device 1400, a learning circuit 1430, and a selector 1440 according to an embodiment of the present disclosure.

[0169] According to an embodiment of the present disclosure, the encoding device 1400 may include a memory 1410 and an encoding circuit 1420, and may be connected to a learning circuit 1430. In addition, the encoding device 1400 may be connected to a selector 1440, a register 1442, and a random number generation circuit 1444. According to an embodiment of the present disclosure, the features of the memory 1410, the encoding circuit 1420, and the learning circuit 1430 may be similar to the features of the memory 110, the encoding circuit 130, and the learning circuit 120 described above, and thus their detailed descriptions will be omitted. In addition, Figure 14 the features of the register 1442 and the random number generation circuit 1444 may also be similar to Figure 11 the features of the register 1140 and Figure 3 the random number generation circuit 340, and thus their detailed descriptions will be omitted.

[0170] Referring to Figure 14 , the selector 1440 may obtain specific information from each of the register 1442 and the random number generation circuit 1444, and selectively output the two pieces of information. According to an embodiment of the present disclosure, the selector 1440 may include a multiplexer that obtains an n-bit (n>0) specific selection signal and selectively outputs the specific information obtained from the register 1442 and the random number generation circuit 1444. The output information of the register 1442 or the random number generation circuit 1444 may be used by the memory 1410, the encoding circuit 1420, and the learning circuit 1430, as in the various embodiments described above of the present disclosure. That is, the selector 1440 may selectively output 2^n types of input information that can be determined based on the n-bit selection signal.

[0171] That is, the encoding device 1400 connected to the selector 1440 may selectively use the process of determining a second edge sequence based on the result of comparing the first edge weight stored in the register 1442 with a preset threshold weight, and the process of determining a second edge sequence based on the result of comparing the first edge sequence stored in the memory 1410 with a random number sequence obtained from the random number generation circuit 1444.

[0172] Specifically, in order for the learning circuit 1430 to perform the above-described discard operation, the selector 1440 may be operated such that the signal generated in the random number generation circuit 1444 is output. Therefore, the random number sequence output from the random number generation circuit 1444 may be sent to the encoding circuit 1420, and the encoding circuit 1420 may generate an edge sequence by using the sent random number sequence and send the edge sequence to the learning circuit 1430.

[0173] In addition, in order for the learning circuit 1430 to perform the above-described pruning operation, the selector 1440 is operable such that the weight value stored in the register 1442 is output. Accordingly, the weight value output from the register 1442 can be sent to the learning circuit 1430. Thus, the learning circuit 1430 can perform learning based on the weight value sent from the register 1442, or can perform a correction operation on the weight value in the direction of increased accuracy.

[0174] Figure 15 is a block diagram showing an encoding device 1500, a learning circuit 1530, a register 1542, a counter 1544, a random number generation circuit 1546, and a selector 1540 according to an embodiment of the present disclosure.

[0175] According to an embodiment of the present disclosure, Figure 15 the features of the memory 1510, the encoding circuit 1520, and the learning circuit 1530 may be similar to Figure 1A the features of the memory 110, the encoding circuit 130, and the learning circuit 120, and thus a detailed description thereof will be omitted. In addition, Figure 15 the features of the register 1542, the counter 1544, and the random number generation circuit 1546 may be respectively similar to Figure 11 the register 1140, Figure 13 the counter 1340, and Figure 3 the features of the random number generation circuit 340, and thus a detailed description thereof will be omitted.

[0176] According to an embodiment of the present disclosure, the selector 1540 may include a multiplexer that can obtain specific information from the register 1542, the counter 1544, and the random number generation circuit 1546 and selectively output the information; and a demultiplexer that obtains an intermediate operation result of the learning circuit 1530 and outputs the intermediate operation result so that the result is stored in the register 1542.

[0177] According to an embodiment of the present disclosure, the learning circuit 1530 may perform a learning process based on a first edge sequence or a second edge sequence, and perform an inference process based on the learning result. The learning circuit 1530 may store an intermediate operation result of the learning process at a second operation cycle in the register 1542 through the selector 1540, and according to an embodiment of the present disclosure, the counter 1544 may determine whether an overflow has occurred based on the value of the intermediate operation result, and may calculate the number of times an overflow has occurred based on the determined result. The encoding device 1500 may control a deep neural network to perform inference and learning by using a dynamic fixed-point method, in which the fixed point may be dynamically changed based on the number of overflows of the second operation cycle calculated by the counter 1544. That is, when performing an inference and learning process by using the deep neural network of the learning circuit 1530, in order to effectively control input and output information while preventing an overflow in the register 1542, the intermediate operation result may be stored in the register 1542, and the counter 1544 may count the number of overflows.

[0178] According to an embodiment of the present disclosure, the learning circuit 1530 may compare the number of overflows calculated by the counter 1544 with a threshold number of overflows to determine whether to perform a learning and inference process by using a dynamic fixed-point method. That is, the learning circuit 1530 may determine whether to perform a learning and inference process by using a comparator based on a dynamic fixed-point method of the result of comparing the number of overflows calculated by using the counter 1544 with the threshold number of overflows. For example, when the number of overflows calculated by using the counter 1544 is equal to or higher than the threshold number of overflows, the learning circuit 1530 may perform a learning or inference process in a certain layer by changing the fixed point by using a dynamic fixed-point method.

[0179] According to an embodiment of the present disclosure, the encoding device 1500 connected to the selector 1540 may selectively use at least one of the following operations: an operation of determining a second edge sequence based on a result of comparing a first edge weight to be stored in the register 1542 with a preset threshold weight; an operation of determining a second edge sequence based on a result of comparing a first edge sequence stored in the memory 1510 with a random number sequence obtained by using a random number generation circuit 1546; or a learning process of calculating the number of overflows of an intermediate calculation result stored in the register 1542 performed by using the counter 1544.

[0180] As described above, a method for performing learning of a deep neural network according to an embodiment of the present disclosure and an apparatus for performing the method can quickly and easily obtain an edge sequence to be used in a subsequent operation cycle based on the edge sequence information that has been used and stored in the hardware device itself in a previous operation cycle. In addition, by generating an edge sequence to be used in a subsequent operation cycle based on the edge sequence information used in a previous operation cycle and a random number sequence obtained in the current operation cycle, the operation execution rate of the deep neural network can be improved, and a hardware device that performs a dropout operation through the deep neural network can be easily implemented.

[0181] In addition, an edge sequence can be obtained without an additional software module for obtaining the edge sequence. Therefore, the storage size of a hardware device that performs operations through a deep neural network can be reduced, and thus the size of the hardware device itself can be reduced.

[0182] Although the present disclosure has been specifically shown and described with reference to embodiments of the present disclosure, those of ordinary skill in the art will understand that various changes in form and detail can be made in the present disclosure without departing from the spirit and scope of the present disclosure defined by the appended claims. Therefore, the embodiments of the present disclosure should be considered merely as descriptive and not for the purpose of limitation. The scope of the present disclosure is defined not by the detailed description of the present disclosure but by the appended claims, and all differences within the scope will be construed as being included in the present disclosure.

[0183] Embodiments of the present disclosure can be written as a computer program and can be implemented using a general-purpose digital computer that executes the program. Examples of computer-readable recording media include storage media such as magnetic storage media (e.g., ROM, floppy disks, hard disks, etc.) and optical recording media (e.g., CD-ROM or DVD).

[0184] According to the method for performing deep neural network learning and the apparatus for performing the method of the present disclosure, the amount of operations can be reduced and the standby time can be minimized, thereby quickly performing learning through the deep neural network.

[0185] Specifically, according to the method for performing deep neural network learning and the apparatus for performing the method of the present disclosure, the connection or disconnection of edges or nodes constituting a layer forming a deep neural network can be quickly controlled.

[0186] Specifically, according to the method for performing deep neural network learning and the method of the present disclosure, instead of software implementation, hardware components are used to perform a dropout operation to solve the problem of overfitting that occurs in learning through a deep neural network, thereby minimizing the amount of operations and increasing the operation rate.

[0187] It should be understood that the embodiments of the present disclosure described herein should be considered as merely descriptive and not for the purpose of limitation. The description of features or aspects in each embodiment of the present disclosure should generally be considered as applicable to other similar features or aspects in other embodiments of the present disclosure.

[0188] Although one or more embodiments of the present disclosure have been illustrated with reference to the accompanying drawings, those of ordinary skill in the art will understand that various changes in form and detail may be made in the present disclosure without departing from the spirit and scope defined by the appended claims.

Claims

1. An encoding device, comprising: A memory that stores a random number sequence generated by a random number generator; And An encoder configured to: Transmit image data to a deep neural network; Receive a first edge sequence of the deep neural network, where the first edge sequence indicates the connection or disconnection of a plurality of first edges included in the first layer of the deep neural network; Generate a second edge sequence indicating the connection or disconnection of a plurality of second edges included in the second layer of the deep neural network by processing the random number sequence based on a first ratio value between bits with a bit value of 0 and bits with a bit value of 1 in the random number sequence and a second ratio value between bits with a bit value indicating a disconnected edge and bits with a bit value indicating a connected edge in the first edge sequence; Output the second edge sequence for configuring the connection or disconnection of the plurality of second edges; And Receive output data output from the deep neural network, where the output data is generated by the second layer in which the plurality of second edges have been configured based on the second edge sequence performing learning on the image data; wherein, the encoder is further configured to: Determine whether the first ratio value and the second ratio value are within the same range; and Based on determining that the first ratio value and the second ratio value are within the same range, generate a second edge sequence that is at least one of the first edge sequence and the random number sequence not within the same range by processing the random number sequence based on the first edge sequence, or Based on determining that the first ratio value and the second ratio value are not within the same range, provide the random number sequence as the second edge sequence.

2. The encoding device according to claim 1, wherein, The random number sequence is based on the clock signal of the random number generator.

3. The encoding device according to claim 2, wherein, The size of the random number sequence is determined based on the number of the plurality of first edges in the first layer.

4. The encoding device according to claim 2, wherein, The first edge sequence, the random number sequence, and the second edge sequence have a bit width formed by binary numbers.

5. The encoding device according to claim 3, wherein, The size of the random number sequence is equal to the number of the plurality of first edges in the first layer.

6. The encoding device according to claim 1, wherein The plurality of second edges included in the second layer of the deep neural network are connected or disconnected based on the second edge sequence.

7. The encoding device according to claim 2, wherein, The encoder is further configured to: obtain the weights of the plurality of first edges, perform a pruning operation on the random number sequence based on the result of comparing the weights with a preset threshold weight, and generate the second edge sequence based on the pruning operation.

8. The encoding device according to claim 1, Among them, The encoder is further configured to: Receive, via a selector, a determination result of whether an overflow has occurred in an intermediate operation result, where the selector is configured to select one of multiple types of input signals and output the selected signal; And Perform a dynamic fixed-point operation to modify the representable range of the information used in the deep neural network based on whether an overflow has occurred.

9. The encoding device according to claim 1, wherein The encoder is further configured to determine whether the first ratio value and the second ratio value are within the same range based on whether the difference between the first ratio value and the second ratio value is equal to or less than a preset value.

10. A coding method for a coding device, the coding method comprising: Transmitting image data to a deep neural network; Storing a random number sequence generated by a random number generator; Receiving a first edge sequence from the deep neural network, the first edge sequence indicating connection or disconnection of a plurality of first edges included in a first layer of the deep neural network; Generating a second edge sequence indicating connection or disconnection of a plurality of second edges included in a second layer of the deep neural network by processing the random number sequence based on a first ratio value between bits having a bit value of 0 in the random number sequence and bits having a bit value of 1 in the random number sequence, and a second ratio value between bits having a bit value indicating a disconnected edge in the first edge sequence and bits having a bit value indicating a connected edge in the first edge sequence; Outputting the second edge sequence for configuring connection or disconnection of the plurality of second edges; And Receiving output data output from the deep neural network, where the output data is generated by the second layer in which the plurality of second edges have been configured based on the second edge sequence performing learning on the image data, wherein generating the second edge sequence includes: Determining whether the first ratio value and the second ratio value are within the same range; and Based on determining that the first ratio value and the second ratio value are within the same range, generating the second edge sequence that is not within the same range as at least one of the first edge sequence and the random number sequence by processing the random number sequence based on the first edge sequence, or Based on determining that the first ratio value and the second ratio value are not within the same range, providing the random number sequence as the second edge sequence.

11. The encoding method according to claim 10, wherein The random number sequence is based on a clock signal of the random number generator.

12. The encoding method according to claim 11, wherein, The size of the random number sequence is determined based on the number of the plurality of first edges in the first layer.

13. The encoding method according to claim 11, wherein, The first edge sequence, the random number sequence, and the second edge sequence have a bit width formed by binary numbers.

14. The encoding method according to claim 12, wherein, The size of the random number sequence is equal to the number of the plurality of first edges in the first layer.

15. The encoding method according to claim 11, wherein, The plurality of second edges included in the second layer of the deep neural network are connected or disconnected based on the second edge sequence.

16. The coding method according to claim 11, further comprising: Obtaining weights of the plurality of first edges; Performing a pruning operation on the random number sequence based on a result of comparing the weights of the plurality of first edges with a preset threshold weight; And Generating the second edge sequence based on the pruning operation.

17. The encoding method according to claim 11, wherein, Determining whether the first ratio value and the second ratio value are within the same range includes: Determine whether the first ratio value and the second ratio value are within the same range based on whether the difference between the first ratio value and the second ratio value is equal to or less than a preset value.

Images