Neuromorphic device for implementing neural network and method of operating same

Abstract

A neuromorphic device realizing a neural network includes: a memory storing at least one program; an on-chip memory including a crossbar array circuit; and at least one processor that drives the neural network by executing the at least one program, wherein the at least one processor receives an audio signal, inputs the audio signal to the neural network trained based on predetermined training data, and outputs a speech recognition result, and the neural network may include a Fourier transform layer that converts the audio signal from a time band to a frequency band to generate a frequency signal, a mel generation layer that generates a mel spectrogram from the frequency signal, and an output layer that outputs the speech recognition result based on speech features of the mel spectrogram.

Description

[Technical field]

[0001] The present invention relates to a neuromorphic device that realizes a neural network and an operating method thereof, and more particularly to a system that performs voice recognition using an edge AI chip without using a cloud server or a physical server and an operating method thereof. [Background technology]

[0002] With the development of Internet technology, human interaction with computer devices becomes more frequent. In the process of human interaction with computer devices, it is very important to accurately recognize natural language and determine user intent.

[0003] However, current voice recognition is performed in a cloud environment, and in the case of a cloud-based voice recognition system, the input voice must be sent to the cloud, which limits real-time voice recognition. In particular, if the communication connection is unstable or a problem occurs at the time of voice recognition, there is a problem that voice recognition itself becomes impossible.

[0004] The above-mentioned background art is technical information possessed by the inventor for the purpose of deriving the present invention or acquired in the process of deriving the present invention, and is not necessarily publicly known art that was disclosed to the general public prior to the filing of the application of the present invention. Summary of the Invention [Problem to be solved by the invention]

[0005] The objective of the present disclosure is to provide a neuromorphic device that realizes a neural network and an operating method thereof. The problems to be solved by the present disclosure are not limited to the technical problems described above, and other technical problems not described will be clearly understood by a person skilled in the art from the description of the present invention, and will be further clearly understood by the embodiments of the present disclosure. It will be understood that the problems and advantages to be solved by the present disclosure can be realized by the means and combinations thereof described in the claims. [Means for solving the problem]

[0006] As a means for solving the above-mentioned technical problem, a first aspect of the present disclosure can provide a neuromorphic device including: a memory storing at least one program; an on-chip memory including a crossbar array circuit; and at least one processor for driving the neural network by executing the at least one program, wherein the at least one processor receives an audio signal, inputs the audio signal to a neural network trained based on predetermined training data, and outputs a speech recognition result, and the neural network includes a Fourier transform layer that converts the audio signal from a time band to a frequency band to generate a frequency signal, a mel generation layer that generates a mel spectrogram from the frequency signal, and an output layer that outputs the speech recognition result based on speech features of the mel spectrogram.

[0007] A second aspect of the present disclosure can provide a method for operating a neuromorphic device that realizes a neural network, the method including steps of receiving an audio signal, inputting the audio signal to a neural network trained based on predetermined training data, and outputting a speech recognition result, the neural network including a Fourier transform layer that converts the audio signal from a time band to a frequency band to generate a frequency signal, a mel generation layer that generates a mel spectrogram from the frequency signal, and an output layer that outputs the speech recognition result based on speech features of the mel spectrogram.

[0008] A third aspect of the present disclosure can provide a computer-readable recording medium having recorded thereon a program for executing the method of the second aspect on a computer.

[0009] In addition, other methods and devices for implementing the present invention, and a computer-readable recording medium having a program for executing the method recorded thereon can also be provided.

[0010] Other aspects, features, and advantages beyond those discussed above will become apparent from the accompanying drawings, the claims, and the following detailed description of the invention. Effect of the Invention

[0011] According to the above-mentioned means for solving the problems disclosed herein, it is possible to provide a neuromorphic device that has low dependency on a server, reduces communication costs, and improves security for personal information.

[0012] In addition, according to the problem-solving means of the present disclosure, it is possible to provide a neuromorphic device that prevents bottlenecks caused by data transmission and reception and thereby improves speed.

[0013] The effects of the embodiments are not limited to those described above, and other effects not described will be clearly understood by a person having ordinary skill in the art from the description of the present invention. [Brief description of the drawings]

[0014] [Figure 1] 1A and 1B are diagrams illustrating a neuromorphic chip structure according to an embodiment. [Diagram 2] FIG. 1 is a diagram for explaining the architecture of a fully-connected neural network (FCNN) according to one embodiment. [Diagram 3] FIG. 2 is a block diagram showing the hardware configuration of a neuromorphic device according to one embodiment. [Figure 4A] FIG. 1 is an illustrative diagram for comparing a Von Neumann structure and a PIM (Processing-In Memory) structure. [Figure 4B] FIG. 1 is an illustrative diagram for comparing a Von Neumann structure and a PIM (Processing-In Memory) structure. [Figure 5A] FIG. 2 is a diagram illustrating a method of operating a neural network according to an embodiment. [Figure 5B] FIG. 2 is a diagram illustrating a method of operating a neural network according to an embodiment. [Figure 6A] FIG. 13 is a diagram for comparing vector-matrix multiplication according to one embodiment with operations performed in a neural network. [Figure 6B] FIG. 13 is a diagram for comparing vector-matrix multiplication according to one embodiment with operations performed in a neural network. [Figure 7] FIG. 10 is a diagram for explaining an example in which a convolution operation is performed in a neural network according to an embodiment. [Figure 8] FIG. 2 is an exemplary diagram of an implementation of a neural network according to one embodiment. [Figure 9]1 is an exemplary implementation diagram of a crossbar array circuit according to one embodiment. [Figure 10] 1 is a flowchart of a method of operating a neuromorphic device according to one embodiment. [Figure 11] FIG. 1 is a block diagram of a neuromorphic device according to one embodiment of the present invention. BEST MODE FOR CARRYING OUT THEINVENTION

[0015] A neuromorphic device realizing a neural network includes: a memory storing at least one program; an on-chip memory including a crossbar array circuit; and at least one processor that drives the neural network by executing the at least one program, wherein the at least one processor receives an audio signal, inputs the audio signal to the neural network trained based on predetermined training data, and outputs a speech recognition result, and the neural network may include a Fourier transform layer that converts the audio signal from a time band to a frequency band to generate a frequency signal, a mel generation layer that generates a mel spectrogram from the frequency signal, and an output layer that outputs the speech recognition result based on speech features of the mel spectrogram. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0016] In describing the present invention, if a specific description of related publicly known technology is deemed to obscure the gist of the present invention, the detailed description may be omitted, and unless otherwise defined, all terms used in this specification have the same meaning as commonly understood by a person having ordinary skill in the art to which the present invention belongs.

[0017] The appearances of the phrases "in one embodiment," "in accordance with one embodiment," "by an implementation of one embodiment," and the like in this specification do not necessarily all refer to the same embodiment.

[0018] Since various modifications can be made to the embodiments and various forms can be taken, some embodiments are illustrated in the drawings and described in detail. However, it should be understood that the embodiments are not limited to the specific disclosed forms, but include all modifications, equivalents, or alternatives included in the spirit and technical scope of the embodiments. The terms used in the specification are used merely for the purpose of describing the embodiments, and are not intended to limit the embodiments.

[0019] The terms used in the embodiments are selected as general terms that are currently widely used as much as possible, taking into consideration the functions in the embodiments, but they may vary depending on the intentions of engineers involved in the technical field to which the embodiments belong, precedents, the emergence of new technologies, etc. In addition, in certain cases, the applicant may arbitrarily select terms, and in such cases, the meanings of the terms are described in detail in the relevant section. Therefore, the terms used in the embodiments should be defined based on the meanings of the terms and the overall content of the embodiments, rather than simply the names of the terms.

[0020] Some embodiments of the present disclosure may be illustrated in terms of functional block configurations and various processing steps. Some or all of such functional blocks may be implemented in any number of hardware and / or software configurations that perform specific functions. For example, the functional blocks of the present disclosure may be implemented by one or more microprocessors or by circuit configurations for a given function.

[0021] Also, for example, the functional blocks of the present disclosure may be implemented in various programming or scripting languages. The functional blocks may be implemented in algorithms executed on one or more processors. Further, the present disclosure may employ conventional techniques for electronic configuration, signal processing, and / or data processing, etc.

[0022] Terms such as "database," "element," "means," and "configuration" can be used broadly and are not limited to mechanical and physical configurations. Furthermore, terms such as "part" and "module" described in the specification refer to a unit that processes at least one function or operation, and can be realized by hardware or software, or a combination of hardware and software.

[0023] In addition, the connecting lines or connecting members between components shown in the drawings are merely exemplary of functional and / or physical connections or circuit connections, and in an actual device, connections between components may be represented by various alternative or additional functional connections, physical connections, or circuit connections.

[0024] In addition, terms including ordinal numbers such as "first" and "second" used in this specification may be used to describe various components, but the components should not be limited by the terms. The terms are used only for the purpose of distinguishing one component from another component.

[0025] In addition, some components in the drawings may be shown somewhat exaggerated in size, ratio, etc. Furthermore, components shown in one drawing may not be shown in the other drawings.

[0026] Throughout the specification, the term "embodiment" is an arbitrary division for easily describing the invention in the present disclosure, and each embodiment does not necessarily have to be mutually exclusive. For example, a configuration disclosed in one embodiment can be applied and / or realized in other embodiments, and can be applied and / or realized with modifications without departing from the scope of the present disclosure.

[0027] Furthermore, the terms used in the present disclosure are intended to describe the embodiments, and are not intended to limit the embodiments. In the present disclosure, the singular form includes the plural form unless otherwise specified.

[0028] Hereinafter, the embodiments of the present disclosure will be described in detail with reference to the accompanying drawings so that those skilled in the art can easily carry out the embodiments. However, the embodiments of the present disclosure can be realized in various different forms and are not limited to the embodiments described in the present disclosure.

[0029] Based on this, the present invention will be described in detail below with reference to the drawings.

[0030] FIG. 1 is a diagram for explaining a neuromorphic chip structure according to an embodiment.

[0031] Referring to Fig. 1, a neuromorphic chip 1 is hardware that generates a circuit that mimics the morphology of neurons to mimic the functions of the human brain. In other words, the neuromorphic chip 1 refers to a computer chip that mimics the structure of the nervous system.

[0032] Neuromorphic Chip 1 is composed only of the circuits necessary for neural network calculations, so it can achieve gains of hundreds of times in terms of power, area, and speed. Unlike conventional computers, the human brain does not consume much power even when processing a large amount of data, so Neuromorphic Chip 1 mimics the way the brain works by configuring a structure connecting neurons and synapses in parallel and disconnecting the connections when data is not being processed, thereby saving energy.

[0033] For example, conventional computers with the von Neumann architecture process input data sequentially, which is excellent for executing precisely written programs. However, it has problems such as limited power consumption and low efficiency in pattern recognition and real-time recognition.

[0034] In contrast, the neuromorphic chip 1 uses analog operations in which data changes gradually between various states, rather than digital data such as 0 and 1. In other words, the parallel-configured artificial neurons operate in an event-driven manner without a clock. This makes it possible to efficiently process irregular characters, sounds, and images that were difficult for conventional computers to intuitively recognize. Specifically, data can be processed in parallel by distributing neurons, which are nerve cells, and synapses, which are connecting lines, using silicon transistor circuits and memory elements.

[0035] In one embodiment, when input data such as an image, video, or voice is input to the neuromorphic chip 1 of Fig. 1, an internal operation is performed on the input data in the neuromorphic chip 1, and predetermined output data is output. The predetermined output data may include a voice / image / video recognition result based on feature classification of the input data. For example, when a voice signal is input as input data, the predetermined output data may include a voice recognition result based on binary classification of whether or not the input data includes a preset keyword.

[0036] On the other hand, the data input to the neuromorphic chip 1 is not limited to the above-mentioned images, videos, or audio, but may include data in various forms such as text.

[0037] FIG. 2 is a diagram for explaining the architecture of a fully-connected neural network (FCNN) according to one embodiment.

[0038] Referring to Figure 2, FCNN, which is a fully connected layer, means a convolutional neural network in which all neurons in one layer are connected to all neurons in the next layer. It is a layer used to classify data by a matrix flattened in the form of a one-dimensional array.

[0039] In FCNN, there is a node value for each node, and a weight and bias for each neuron. When moving from one layer to another, the value obtained by multiplying each node by its weight and adding the bias becomes the node value of the next layer. Here, if the calculated value satisfies certain conditions, the output value that has passed through the activation function is input as the node value of the next layer and activated, and if the calculated value does not satisfy certain conditions, an activation function intervenes to inactivate the node via the activation function.

[0040] Therefore, since the output value differs depending on the type of activation function, it is important to use an appropriate activation function as needed. Typical examples are the ReLU function and the Softmax function.

[0041] Since only one-dimensional array data can be input to FCNN, when the input data is an image of a three-dimensional array consisting of vertical, horizontal, and channel (color), it has the disadvantage that it must be flattened to one-dimensional data and input to FCNN. In other words, there is a disadvantage that the spatial information of the image is ignored, and therefore features contained in the shape cannot be extracted. Therefore, in the case of FCNN, it is highly useful when data that can be realized as one-dimensional array data, such as audio data, is input. According to one embodiment, the neural network described later may be FCNN.

[0042] FIG. 3 is a block diagram showing a hardware configuration of a neuromorphic device according to an embodiment.

[0043] The neuromorphic device 300 can be realized by various devices such as a PC (personal computer), a server device, a mobile device, and an embedded device, and specific examples thereof include, but are not limited to, smartphones, tablet devices, AR (Augmented Reality) devices, IoT (Internet of Things) devices, autonomous automobiles, robotics, medical devices, etc., which perform voice recognition, image recognition, image classification, etc. using a neural network. Furthermore, the neuromorphic device 300 may be a dedicated hardware accelerator (HW accelerator) mounted on such devices, and the neuromorphic device 300 may be, but is not limited to, a hardware accelerator such as an NPU (neural processing unit), a TPU (Tensor Processing Unit), or a Neural Engine, which is a dedicated module for driving a neural network.

[0044] The neuromorphic device 300 may include a processor and a memory. Only components according to this embodiment are shown in the neuromorphic device 300 shown in Fig. 3, and it is obvious to a person skilled in the art that the neuromorphic device 300 may further include other general-purpose components in addition to the components shown in Fig. 3.

[0045] According to one embodiment, the neuromorphic device 300 may include an input / output interface 310 .

[0046] In one embodiment, the I / O interface 310 can transmit data from within the neuromorphic device 300 to an external source and receive data from an external source into the neuromorphic device 300. The signals of the I / O interface 310 can be unidirectional or bidirectional signals, single-ended or differential-mode, and can conform to one of several other I / O interface standards.

[0047] According to an embodiment, the input / output interface 310 may further include an audio receiver, such as an audio ADC or DAC, which may digitize audio input from a microphone, may include an amplifier, and may support multiple sampling rates.

[0048] In one embodiment, the input / output interface 310 can receive an audio signal from outside the neuromorphic device 300. The input / output interface 310 can also receive an input of a voice recognition result, which is an output value of the neural network 330, and transmit it to the arithmetic circuit 350 or another external unit (not shown).

[0049] In one embodiment, the input / output interface 310 may further include a general-purpose input / output (GPIO), an integrated interchip sound (I2S), an inter-integrated circuit (I2C), a serial peripheral interface (SPI), a universal asynchronous receiver / transmitter (UART), a pulse width modulation (PWM), and the like.

[0050] According to one embodiment, a neuromorphic device 300 may include analog devices 320, 340.

[0051] The analog devices 320, 340 can provide functions for stable power supply, system monitoring and supervision in the edge AI chip.

[0052] Specifically, the analog devices 320, 340 are devices that operate on parameters that respond to continuous physical quantities such as voltage, resistance, rotation, pressure, etc. in supervisory control, data collection, and automated control. The analog devices 320, 340 may include analog display devices and analog input / output devices, and may further include analog-to-digital converters and digital-to-analog converters.

[0053] In addition, the analog devices 320 and 340 can realize power management and supervision functions of the neuromorphic device 300. The analog devices 320 and 340 further include a Low Voltage Detector (LVD) that can operate a Power-On Reset (POR) when the digital power supply (VDD) falls below a safe operating level to prevent unstable operation of the neuromorphic device 300.

[0054] According to one embodiment, the neuromorphic device 300 may include a computational circuit 350 .

[0055] The arithmetic circuit 350 may include an MCU (Micro Controller Unit), a DMA (Direct Memory Access), and the like.

[0056] The MCU can play a role of control and management so that efficient algorithms can be executed in the PIM structure. The PIM structure may be suitable for the calculation of the neural network according to an embodiment of the present invention because of its high efficiency of MAC (Multiply And Accumulate) operations. The PIM structure will be described later with reference to FIG. 4b.

[0057] DMA can transfer data quickly between peripherals and memory, or between memories. Data can be moved quickly with DMA without the CPU, which can be used for other tasks. DMA can be located inside SRAM and can help accelerate data movement in neural networks.

[0058] The neuromorphic device 300 according to an embodiment can be realized by an edge AI chip. Edge AI refers to a technology of executing AI algorithms in a hardware device using edge computing based on data generated in a system. AI processing is mainly performed in cloud-based data centers that require huge computing capacity, so there is a high dependency on servers. In contrast, when edge AI is used, the execution of AI algorithm operations is performed locally, so there is less dependency on the cloud (server), which reduces communication costs, and privacy is protected because sensitive personal information is not sent to the cloud. Therefore, by configuring the neuromorphic device 300 with an edge AI chip, not only can costs be reduced and security be improved, but also a highly responsive system can be realized because the execution of operations is immediately processed within the same hardware.

[0059] 4a and 4b are illustrative diagrams for comparing the Von Neumann structure and the PIM (Processing-In Memory) structure.

[0060] Referring to FIG. 4a, the von Neumann architecture is a computer architecture proposed by John von Neumann, and is a stored-program computer architecture consisting of a typical three-tiered structure of a main memory device, a central processing unit, and an input / output device.

[0061] The von Neumann architecture has the advantage that when changing to a different task in a computing device, there is no need to rearrange the hardware (wires, etc.), and only the software (program) needs to be changed, which greatly improves versatility. However, since it executes a series of instructions, each of which consists of changing the value of a specific memory location, it causes a serious problem in the design of high-speed computers, known as the von Neumann bottleneck phenomenon.

[0062] To solve the von Neumann bottleneck phenomenon, alternative solutions that have been proposed include the Harvard architecture, which divides memory into a section that stores instructions and a section that stores data; the PIM architecture, which not only stores data in memory but also performs calculations on it; and neuromorphic computing, which is an integrated circuit in the form of an artificial neural network that mimics the brain structure of higher animals, in which numerous units that combine calculation and memory functions are connected in parallel in a mesh shape, and then each unit operates in an event-driven manner.

[0063] With reference to FIG. 4b, it can be seen that the PIM structure consists of a processor and a memory with computing capabilities.

[0064] In the conventional Von Neumann architecture, all data in the memory is moved to the processor for calculation, but in the PIM architecture, when an instruction word is delivered to the processor, the calculation is performed in the memory and only the result data is sent to the processor, so there is no need to move a large amount of data, which effectively solves the Von Neumann bottleneck phenomenon mentioned above. It also has the advantage of significantly reducing power consumption.

[0065] Returning again to FIG. 3, a neuromorphic device 300 according to one embodiment may include a neural network 330 .

[0066] In one embodiment, the neural network 330 may output output data by extracting features related to input data using multiple layers. For example, the neural network 330 may receive an audio signal as input and output a speech recognition result. As described above, the neural network 330 may filter or classify the features of the input audio signal to determine the type of the audio signal, such as whether the audio signal includes a specific keyword, and output the determination result as a speech recognition result.

[0067] Here, the neural network 330 may be configured as an orthogonal matrix, with inputs being row-wise and outputs being generated column-wise, although the configuration of the neural network 330 is not limited thereto.

[0068] In one embodiment, the neural network 330 may be a model trained based on predetermined training data. As described above, each layer constituting the neural network 330 may have weights and / or bias values. Since the number of nodes in one layer is the size of the input tensor, there may be hundreds or thousands of nodes in each layer. When multiple such layers are stacked, the number of weights connected to the nodes of the next layer becomes countless, so it is difficult to set all the weights and / or bias values. Therefore, by learning, it is possible to detect the layer (i.e., weights and / or biases) that is most optimized for the desired classification for a huge amount of data.

[0069] For example, as a learning method for the neural network 330, deep learning can detect and set the most suitable weights in the neural network by combining three methods: loss function or cost function, optimization, and back propagation.

[0070] The neural network 330 can perform calculations using only on-chip memory without using an external memory. For example, the neural network 330 can perform calculations for each layer on a PIM basis using only on-chip memory without using an external memory (e.g., off-chip memory, etc.), thereby performing calculations without memory updates while processing an audio signal. Specifically, the neural network 330 can perform PIM-based calculations in which each memory cell is directly connected to a processor.

[0071] However, since PIM requires high memory bandwidth, the power consumption of the computing device may be limited because the memory is sensitive to high temperature. Therefore, a new hardware structure may be required to produce a chip with a high performance PIM structure, which may increase manufacturing costs. Therefore, the PIM structure may be advantageous for performing a relatively small amount of calculations or simple calculations. In contrast, the neural network 330 may be configured with a memory capable of realizing multi-bit in order to overcome the disadvantages of such a PIM structure. For example, the neural network 330 may be configured with a memory capable of realizing 7 bits 128 analog memory states. By configuring the neural network 330 with a large capacity, it is possible to process a huge amount of data with low power and high performance even when used for a long period of time, unlike general PIM chips that show problems such as heat generation and performance degradation.

[0072] 5a and 5b are diagrams illustrating a method of operating a neural network according to an embodiment.

[0073] 5a, the neural network may include multiple cores, each of which may be implemented with Resistive Crossbar Memory Arrays (RCA). Specifically, each of the cores may include multiple presynaptic neurons 510, multiple postsynaptic neurons 520, and synapses 530 providing respective connections between the multiple presynaptic neurons 510 and the multiple postsynaptic neurons 520.

[0074] In one embodiment, the core of the neural network includes four presynaptic neurons 510, four post-synaptic neurons 520, and 16 synapses 530, but the number of these can be changed in various ways. If the number of presynaptic neurons 510 is N (where N is a natural number equal to or greater than 2) and the number of post-synaptic neurons 520 is M (where M is a natural number equal to or greater than 2 and may be the same as or different from N), N*M synapses 530 may be arranged in a matrix.

[0075] Specifically, a wiring 512 connected to each of the plurality of presynaptic neurons 510 and extending in a first direction (e.g., horizontal direction), and a wiring 522 connected to each of the plurality of postsynaptic neurons 520 and extending in a second direction (e.g., vertical direction) intersecting the first direction can be provided. Hereinafter, for convenience of explanation, the wiring 512 extending in the first direction is referred to as a row line, and the wiring 522 extending in the second direction is referred to as a column line. A plurality of synapses 530 are arranged at each intersection of the row wiring 512 and the column wiring 522, and the corresponding row wiring 512 and the corresponding column wiring 522 can be connected to each other.

[0076] The presynaptic neuron 510 plays a role of generating a signal, for example, a signal corresponding to specific data, and transmitting it to the row wiring 512, and the post-synaptic neuron 520 plays a role of receiving and processing a synaptic signal through the synapse element 530 via the column wiring 522. The presynaptic neuron 510 may correspond to an axon, and the post-synaptic neuron 520 may correspond to a neuron. However, whether a neuron is a presynaptic neuron or a post-synaptic neuron may be determined according to a relative relationship with other neurons. For example, when the presynaptic neuron 510 receives a synaptic signal in relation to other neurons, it may function as a post-synaptic neuron. Similarly, when the post-synaptic neuron 520 transmits a signal in relation to other neurons, it may function as a pre-synaptic neuron. The presynaptic neuron 510 and the post-synaptic neuron 520 may be realized by various circuits such as CMOS.

[0077] The connection between the presynaptic neuron 510 and the postsynaptic neuron 520 can be made via a synapse 530. Here, the synapse 530 is an element whose electrical conductance or weight changes in response to an electrical pulse, such as a voltage or current, applied across both ends.

[0078] The core of the neural network may be constructed using a multi-level memory such as a ReRAM (Resistive RAM) in which the synapses 530 are constructed of variable resistors, an FeRAM (Ferroelectric RAM) in which the synapses 530 are constructed of ferroelectrics, a PRAM (Phase-change RAM) in which the synapses 530 are constructed of chalcogenide glass that changes from an amorphous state to a crystalline state when heat is applied, an MRAM (Magnetic RAM) in which the synapses 530 are constructed of magnetic elements, or a NAND / NOR flash memory.

[0079] The synapse 530 may include, for example, a variable resistance element. The variable resistance element is an element that can be switched between different resistance states depending on a voltage or current applied across both ends, and may have a single-layer structure or a multi-layer structure including various materials that can have multiple resistance states, such as transition metal oxides, metal oxides such as perovskite-based materials, phase-change materials such as chalcogenide-based materials, ferroelectric materials, and ferromagnetic materials.

[0080] The core synapse 530 can be realized to have various characteristics that are distinct from the variable resistance element in the memory, such as showing analog behavior in which the conductivity changes gradually according to the number of input electrical pulses without an abrupt change in resistance between the set operation and the reset operation, because the characteristics required for the variable resistance element in the memory are different from the characteristics required for the synapse 530 in the core of the neural network.

[0081] Specifically, the neuromorphic chip can apply varying voltages to the core synapses 530 through analog devices, thereby gradually changing the resistance or weight of the synapses 530.

[0082] The operation of the above-mentioned neural network will be described below with reference to Fig. 5b. For convenience of explanation, the row wirings 512 can be referred to as the first row wiring 512A, the second row wiring 512B, the third row wiring 512C, and the fourth row wiring 512D from the top, and the column wirings 522 can be referred to as the first column wiring 522A, the second column wiring 522B, the third column wiring 522C, and the fourth column wiring 522D from the left.

[0083] 5b, in an initial state, all the synapses 530 may be in a state in which the conductance is relatively low, i.e., in a high resistance state. If at least some of the synapses 530 are in a low resistance state, an initialization operation may be further required to bring them to a high resistance state. Each of the synapses 530 may have a predetermined threshold required for a change in resistance and / or conductance. More specifically, when a voltage or current smaller than a predetermined threshold is applied across each synapse 530, the conductance of the synapse 530 does not change, and when a voltage or current larger than the predetermined threshold is applied to the synapse 530, the conductance of the synapse 530 may change.

[0084] In that state, an input signal corresponding to the particular data can enter the row wires 512 in response to the output of the presynaptic circuit 510 to operate to output the particular data as a result of the particular column wires 522. For example, the input signal may be manifested by the application of an electrical pulse to each of the row wires 512, and the column wires 522 may be driven with an appropriate voltage or current for output.

[0085] As another example, the column wire 522 that outputs the specific data may not be determined. In that case, the current flowing through each column wire 522 is measured while an electrical pulse corresponding to the specific data is applied to the row wires 512, and the column wire 522 that first reaches a predetermined threshold current, for example, the third column wire 522C, may be the column wire 522 that outputs the specific data.

[0086] By using the method described above, different data can be output to different column wirings 522, respectively.

[0087] 6a and 6b are diagrams for comparing vector-matrix multiplication according to one embodiment with the operations performed in a neural network.

[0088] 6a and 6b are diagrams for comparing vector-matrix multiplication according to one embodiment with the operations performed in a neural network.

[0089] First, referring to FIG. 6a, the convolution operation between the input data and the kernel may be performed using vector-matrix multiplication. For example, the pixel data of the input data may be represented by a matrix X 610, and the kernel value may be represented by a matrix W 611. The pixel data of the output data may be represented by a matrix Y 612, which is a result of the multiplication operation between the matrix X 610 and the matrix W 611.

[0090] Referring to Fig. 6b, a vector multiplication operation may be performed using a core of a neural network. In comparison with Fig. 6a, pixel data of input data is received as an input value of the core, and the input value may be a voltage 620. In addition, a kernel value is stored in a synapse, i.e., a memory cell, of the core, and the kernel value stored in the memory cell may be a conductance 621. Thus, an output value of the core may be represented by a current 622, which is a result of a multiplication operation between the voltage 620 and the conductance 621.

[0091] FIG. 7 is a diagram for explaining an example in which a convolution operation is performed in a neural network according to an embodiment.

[0092] The neural network may receive an audio signal 710, and the neural network core 700 may be implemented with RCA (Resistive Crossbar Memory Arrays). Here, the audio signal 710 may be converted from a time band to a frequency band by a Fourier transform layer implemented in the neural network core 700, converted into a mel-spectrogram by a mel generation layer implemented in the neural network core 700, and a convolution operation may be performed on pixel data corresponding to the mel-spectrogram. Therefore, in the following description of FIG. 7, pixel data of the audio signal 710 may refer to pixel data of the converted frequency band or pixel data of the generated mel-spectrogram.

[0093] In one embodiment, when the core 700 is a matrix of size N×M, where N and M are natural numbers greater than or equal to 2, the number of pixel data of the audio signal 710 may be less than or equal to the number of columns M of the core 700. The pixel data of the audio signal 710 may be parameters in floating point or fixed point format.

[0094] The neural network may receive pixel data in the form of a digital signal and convert the received pixel data into a voltage in the form of an analog signal using a Digital-Analog Converter (DAC) 720. The pixel data of the audio signal 710 may have various bit resolution values, such as 1-bit, 4-bit, 8-bit resolution, etc. In one embodiment, the neural network may convert the pixel data into a voltage using the DAC 720 and then receive the voltage as an input value 701 of the core 700.

[0095] Furthermore, a learned kernel value may be stored in the core 700 of the neural network. The kernel value may be stored in a memory cell of the core, and the kernel value stored in the memory cell may be a conductance 702. Here, the neural network can calculate an output value by performing a vector multiplication operation between an input value 701, for example, an input voltage and a conductance 702, and the output value can be expressed as a current 703. That is, the neural network can output a result value that is the same as the result of a convolution operation between an audio signal and a kernel using the core 700.

[0096] Since the output value 703 output from the core 700, e.g., the output current, is an analog signal, the neural network can use an ADC (Analog Digital Converter) 730 to use the output current 703 as input data for another core. The neural network can use the ADC 730 to convert the output current 703, which is an analog signal, into a digital signal. In one embodiment, the neural network can use the ADC 730 to convert the output current 703 into a digital signal having the same bit resolution as the pixel data of the audio signal 710. For example, if the pixel data of the audio signal 710 has a 1-bit resolution, the neural network can use the ADC 730 to convert the output current 703 into a digital signal with a 1-bit resolution.

[0097] The neural network can use an activation unit 740 to apply an activation function to the digital signal converted by the ADC 730. The activation function can be a Sigmoid function, a Tanh function, or a ReLU (Rectified Linear Unit) function, but is not limited to these.

[0098] The digital signal to which the activation function has been applied can be used as an input value of another core 750. When the digital signal to which the activation function has been applied is used as an input value of another core 750, the above-mentioned process can be similarly applied to the other core 750.

[0099] On the other hand, the core 700 and the other core 750 are not physically separated, but may refer to each of the cores 700, 750 in which the variable resistance element values ​​of the synapses are changed according to the weight and / or bias values ​​possessed by each of the cores 700, 750.

[0100] FIG. 8 is an example diagram of a neural network implementation according to one embodiment.

[0101] 8, the neural network may include a Fourier Transform layer 810 that converts a received audio signal from a time band to a frequency band to generate a frequency signal, a Mel generation layer 820 that generates a Mel spectrogram from the frequency signal, one or more hidden layers 830, 840 that classify features of the audio signal based on the Mel spectrogram, and an output layer 850 that outputs a speech recognition result. The order of each layer may be the same as that shown in FIG. 8, but is not limited thereto.

[0102] In one embodiment, the Fourier transform layer 810 can Fourier transform a time-band audio signal to generate a frequency signal in a frequency band. As described above, the audio signal can have a time band because the audio signal can be data that an input / output interface (or an audio receiving unit included in the input / output interface) digitizes a voice input from a microphone. Therefore, the Fourier transform layer 810 can convert the data into frequency band data by Fourier calculation so as to be suitable for input to other layers 820 to 850 for classification of the audio signal. For example, the Fourier transform layer 810 can generate a frequency signal by DFT (Discrete Fourier Transform), which is a Fourier transform for a discontinuous discrete function.

[0103] In one embodiment, the neural network further includes a digital-to-analog converter (DAC) for converting a digital audio signal into an analog signal, and the received audio signal may be converted into an analog signal by the digital-to-analog converter and then input into the Fourier transform layer 810.

[0104] In one embodiment, the Mel generation layer 820 can receive a frequency signal input from the Fourier transform layer 810 and generate a mel spectrogram corresponding to the audio signal.

[0105] A spectrogram is a graph that visualizes the spectrum of an audio signal. The x-axis of a spectrogram represents time, and the y-axis represents frequency, and the value of each frequency per time can be displayed in color according to the magnitude of the value. Here, since the result of performing a Fourier transform on an audio signal is a complex value, it is possible to take the absolute value of the complex value to eliminate phase information and generate a spectrogram that contains only magnitude information.

[0106] On the other hand, the Mel spectrogram is a spectrogram whose frequency interval is re-adjusted to the Mel scale. The human hearing organ is more sensitive in the low frequency band than in the high frequency band, and the Mel scale reflects this characteristic and shows the relationship between physical frequency and the frequency that humans actually perceive. The Mel spectrogram can be generated by applying a filter bank based on the Mel scale to the spectrogram.

[0107] In one embodiment, one or more hidden layers 830, 840 may receive an input of a mel spectrogram corresponding to an audio signal from the mel generation layer 820 and classify phonetic features of the audio signal or the mel spectrogram.

[0108] One or more hidden layers 830, 840 can classify the mel spectrogram according to a predetermined criterion by using a tensor corresponding to the input mel spectrogram as a presynaptic neuron and reflecting weights to the presynaptic neuron. Therefore, the more hidden layers there are, the more sophisticated the processing of input data can be, improving the function of the algorithm.

[0109] In one embodiment, one or more hidden layers 830, 840 may be located between the mel generation layer 820 and the output layer 850. Thus, the one or more hidden layers 830, 840 may receive an input of a mel spectrogram from the mel generation layer 820, classify features, and output the classification result for transmission to the output layer 850.

[0110] In one embodiment, the output layer 850 may receive voice features or the results of classifying voice features from the mel generation layer 820 or one or more hidden layers 830, 840 and output a voice recognition result. For example, the output layer 850 may output a voice recognition result by determining whether the result of classifying voice features by one or more hidden layers 830, 840 corresponds to any one of a plurality of preset keywords. Specifically, the neuromorphic device may store a plurality of preset keywords. Here, the hidden layers 830, 840 classify the features of the audio signal, and the output layer 850 may receive the classification result and determine whether the result corresponds to any one of a plurality of preset keywords.

[0111] For example, the keywords may be commands consisting of a predetermined number of syllables, such as "turn on the lights," "turn off the lights," "brighten the lights," "dim the lights," "block blue light," and "warm lights." Thus, the input / output interface receives an audio signal including life noise and natural language, and classifies the features of the audio signal using the Fourier transform layer 810, the Mel generation layer 820, and one or more hidden layers 830 and 840, so that the output layer 850 can determine whether the classification result indicates that the received audio signal corresponds to any one of the preset keywords. That is, if the received audio signal corresponds to any one of the preset keywords, the neuromorphic device outputs the keyword as a speech recognition result via the output layer 850, and if the received audio signal does not correspond to any one of the preset keywords, the neuromorphic device outputs a null value or does not output a null value, and receives a next audio signal to repeat the calculations by the above-mentioned layers. In one embodiment, the keywords may be set to about 20 or less.

[0112] In one embodiment, when the audio signal corresponds to a negative signal similar to at least one of a plurality of preset keywords, the output layer 850 outputs a null value as a speech recognition result, or receives the next audio signal without outputting it, and can repeat the calculation by each layer described above. The negative signal may mean a keyword that is similar to any one of a plurality of preset keywords and is likely to be recognized as a similar keyword by the neural network. For example, if the preset keyword is "auto care", "watch" may be a negative signal. Thus, the negative signal may be preset to prevent the neural network from erroneously recognizing a negative signal similar to a keyword as a keyword.

[0113] In one embodiment, the neural network further includes an analog-to-digital converter (ADC) that converts the speech recognition result corresponding to an analog signal into a digital signal, and the feature classification result or the speech recognition result output from the output layer may be converted into a digital signal by the analog-to-digital converter and then output via the input / output interface.

[0114] In one embodiment, when the neural network determines that the audio signal does not correspond to any of a plurality of preset keywords based on the mel spectrogram output from the mel generation layer 820, the neural network does not input the mel spectrogram to the hidden layers 830 and 840 and outputs a null value as a speech recognition result, or does not output the mel spectrogram and receives the next audio signal to repeat the calculations by each layer. That is, when the neural network receives an audio signal, it performs calculations on the data using a default value (Default) up to the Fourier transform layer 810 and the mel generation layer 820, and when it determines that the audio signal is not a human voice or does not correspond to any one of a plurality of preset keywords based on the mel spectrogram, it may not transmit data to the hidden layers 830 and 840 and / or the output layer 850. Since it is not necessary to perform calculations in the hidden layers 830 and 840 and / or the output layer 850 for all audio signals, the amount of calculations may be reduced to increase the calculation efficiency of the neuromorphic device and prevent excessive heat generation in the case of a PIM structure.

[0115] Specifically, the neural network receives an audio signal, and if the received input signal is not included in the trained model data by the inference model trained with predetermined training data, the neural network can suspend the calculation operation and switch to a low power mode (sleep mode). That is, if the input signal is merely noise including everyday noise in the surrounding environment, unnecessary power consumption can be minimized by not performing calculations by the hidden layers 830 and 840. On the other hand, if the received input signal is included in the trained model data, the neural network can perform classification and output of the voice recognition result by the next hidden layers 830 and 840 and the output layer 850, and then switch to a low power mode (sleep mode).

[0116] Meanwhile, each layer 810-850 of the neural network is not physically separated, but may represent a core in which the variable resistance element value of the synapse is changed according to the weight of each layer. Referring again to FIG. 3 for explanation, each layer 810-850 of the neural network may be realized by the neural network 330.

[0117] For example, when an audio signal is received via the input / output interface 310, the audio signal converted into an analog signal by the digital-to-analog converter 320 may be input to the neural network 330 in which the variable resistor element values ​​of the synapses are set corresponding to the weights of the Fourier transform layer 810, and a frequency signal in which the time band of the audio signal is changed to a frequency band may be generated. The frequency signal may then be converted into a digital signal by the analog-to-digital converter 340 and input to the arithmetic circuit 350. In addition, the frequency signal in which the output of the arithmetic circuit 350 is converted into an analog signal by the digital-to-analog converter 320 may be input to the neural network 330 in which the variable resistor element values ​​of the synapses are set corresponding to the weights of the mel generation layer 820, and a mel spectrogram of the frequency signal may be generated. The mel spectrogram may then be converted into a digital signal by the analog-to-digital converter 340 and input to the arithmetic circuit 350. Similarly, the process of classifying the features of an audio signal based on a neural network 330 in which variable resistance element values ​​of synapses are set corresponding to the weights of one or more hidden layers 830, 840 and an output layer 850, and outputting a voice recognition result overlaps with the above-mentioned process, and therefore will not be described here.

[0118] In one embodiment, the neural network may be trained with predetermined training data. For example, the predetermined training data may be audio data generated by combining a plurality of preset keywords and a noise set. In the field of voice recognition, the training data utilized for training the neural network is mainly data containing a mixture of a plurality of preset keywords and noise such as life noise, and a neural network with good recognition performance can be constructed by repeatedly training the neural network. However, in order to construct a neural network with good keyword recognition performance by using keyword voices in a severe noise environment as training data, a large amount of training data and a training process are required. In contrast, when a neural network is trained using a plurality of preset keyword voices as training data and then a data set generated by combining the plurality of preset keyword voices and noise data is used as training data, the amount of training data is smaller than that of the above-mentioned training method, which is efficient and can reduce the cost for generating the training data.

[0119] In one embodiment, one or more hidden layers 830, 840 may be configured with weights adjusted based on predetermined learning data. The calculation of one or more hidden layers 830, 840 is performed using the resistance values ​​of the variable resistance elements of the synapses of the neural network as weights. Here, each weight may be adjusted based on the above-mentioned predetermined learning data, that is, audio data generated by combining a plurality of preset keyword voices and noise sets.

[0120] FIG. 9 is an exemplary diagram illustrating an implementation of a crossbar array circuit according to an embodiment.

[0121] 9, a crossbar array circuit 900 that realizes a neural network may include multiple subcircuits 910, 920, and 930. The subcircuits 910, 920, and 930 are circuits formed by a combination of multiple cores that constitute the crossbar array circuit 900, and synaptic weights may be set so that each of the subcircuits 910, 920, and 930 corresponds to each layer that constitutes the neural network.

[0122] In one embodiment, the crossbar array circuit 900 may include a first sub-circuit 910, a second sub-circuit 920, and a third sub-circuit 930, where the first sub-circuit 910 may implement a Fourier transform layer, the second sub-circuit 920 may implement a Mel generation layer, and the third sub-circuit 930 may implement an output layer.

[0123] In one embodiment, weights corresponding to the Fourier transform layer may be stored in the synapses of the first sub-circuit 910. For example, the first sub-circuit 910 can generate a frequency signal by receiving a time-band audio signal as input data via a presynaptic neuron, and transmitting a synaptic signal through a synapse in which weights corresponding to the Fourier transform layer are stored as output data via a post-synaptic neuron.

[0124] In one embodiment, weights corresponding to the mel generation layer may be stored in synapses of the second subcircuit 920. For example, the second subcircuit 920 may generate a mel spectrogram by receiving a frequency signal as input data via a presynaptic neuron, and transmitting a synaptic signal through a synapse in which weights corresponding to the mel generation layer are stored as output data via a post-synaptic neuron.

[0125] In one embodiment, weights corresponding to the output layer may be stored in synapses of the third subcircuit 930. For example, the third subcircuit 930 may receive a mel spectrogram or a feature of the mel spectrogram as input data via a presynaptic neuron, and transmit a synaptic signal through a synapse in which weights corresponding to the output layer are stored as output data via a post-synaptic neuron, thereby outputting a speech recognition result.

[0126] Similarly, the crossbar array circuit 900 may further include another sub-circuit (not shown) for realizing a hidden layer in addition to the first to third sub-circuits 910, 920, and 930. The other sub-circuit (not shown) for realizing the hidden layer may be a circuit in which weights corresponding to the hidden layer are stored in synapses. For example, the sub-circuit (not shown) may receive a mel spectrogram as input data via a presynaptic neuron, and transmit a synaptic signal that has passed through a synapse in which weights corresponding to the hidden layer are stored as output data via a post-synaptic neuron, thereby classifying the voice feature of the mel spectrogram.

[0127] Meanwhile, the input data of each of the subcircuits 910, 920, and 930 can be received not through presynaptic neurons but through row wiring or column wiring connected to the subcircuit realizing the previous layer. Similarly, the output data of each of the subcircuits 910, 920, and 930 can be transmitted not through post-synaptic neurons but through row wiring or column wiring connected to the subcircuit realizing the next layer. The manner in which each of the subcircuits 910, 920, and 930 transmits and receives input data and output data is not limited thereto. Also, although FIG. 9 shows that the crossbar array circuit 900 includes three subcircuits 910, 920, and 930, the number of subcircuits is not limited thereto, and the regions of the subcircuits 910, 920, and 930 in the crossbar array circuit 900, and the configuration and position of the synapses are not limited thereto.

[0128] The crossbar array circuit 900 according to the embodiment described above is included in an on-chip memory and can realize a neural network by PIM-based calculation. Such a neuromorphic device can not only reduce the chip volume but also achieve high performance relative to power consumption, so that it can output speech recognition results for a relatively small number of keywords very efficiently.

[0129] FIG. 10 is a flowchart of a method of operating a neuromorphic device according to one embodiment.

[0130] In step 1010, a neuromorphic device (hereafter referred to as "device") may receive an audio signal.

[0131] In step 1020, the device may input the audio signal to a neural network trained based on predetermined training data and output a speech recognition result.

[0132] In one embodiment, the neural network may include a Fourier transform layer that converts the audio signal from a time domain to a frequency domain to generate a frequency signal, a mel generation layer that generates a mel spectrogram from the frequency signal, and an output layer that outputs a speech recognition result based on speech features of the mel spectrogram.

[0133] In one embodiment, the neural network may further include one or more hidden layers that classify audio features of the mel spectrogram, where the one or more hidden layers may be located between the mel generation layer and the output layer.

[0134] In one embodiment, the output layer may output a speech recognition result by determining whether the audio signal corresponds to any one of a plurality of predefined keywords as a result of classification of speech features by the hidden layer.

[0135] In one embodiment, the predetermined training data is audio data generated from a combination of a plurality of preset keywords and noise sets, and the one or more hidden layers may be configured with weights adjusted based on the predetermined training data.

[0136] In one embodiment, the neural network may output a null value as a speech recognition result if the audio signal corresponds to a negative signal similar to at least one of a plurality of predefined keywords.

[0137] In one embodiment, the device may have a memory cell (or on-chip memory) directly connected to a processor, and may perform processing-in-memory (PIM)-based calculations.

[0138] In one embodiment, the device may convert an audio signal corresponding to a digital signal into an analog signal and input it to a Fourier transform layer, and convert a voice recognition result corresponding to an analog signal output from an output layer into a digital signal.

[0139] In one embodiment, in response to determining based on the mel spectrogram that the audio signal does not correspond to any of a plurality of predefined keywords, the device may not input the mel spectrogram to the hidden layer.

[0140] 11 is a block diagram of a neuromorphic device according to one embodiment of the present invention. In the following description, descriptions that overlap with the descriptions regarding the above-mentioned FIGS.

[0141] The processor serves to control the overall functions for executing the neuromorphic device 1100. For example, the processor 1110 generally controls the neuromorphic device 1100 by executing a program stored in the memory 1120 in the neuromorphic device 1100. The processor 1110 can be realized by a central processing unit (CPU), a graphics processing unit (GPU), an application processor (AP), etc. provided in the neuromorphic device 1100, but is not limited thereto.

[0142] The memory 1120 is hardware that stores various data processed within the neuromorphic device 1100. For example, the memory can store data processed by the neuromorphic device 1100 and data to be processed. The memory 1120 can also store applications, drivers, etc. that are run by the neuromorphic device 1100. The memory includes random access memory (RAM) such as dynamic random access memory (DRAM) and static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), CD-ROM, Blu-ray or other optical disk storage, hard disk drive (HDD), solid state drive (SSD), or flash memory.

[0143] Unlike the neural network 1 shown in Fig. 1, an actual neural network driven by the neuromorphic device 1100 may be realized with a more complex architecture, so that the processor 1110 performs operations corresponding to a very large operation count, ranging from hundreds of millions to tens of billions, and the frequency with which the processor 1110 accesses the memory 1130 for operations may also increase dramatically.

[0144] Thus, the memory 1120 may be an on-chip memory. The neuromorphic device 1100 according to an embodiment includes the memory 1120 only in the form of an on-chip memory, and can perform operations without accessing the external memory 1130. For example, the memory 1120 may be an SRAM realized in the form of an on-chip memory. In this case, unlike the above, the type of memory that is mainly used for the external memory 1130, such as a DRAM, a ROM, a HDD, or an SSD, may not be used for the memory 1120.

[0145] The processor 1110 can read / write neural network data, such as voice data (audio signal), from the memory 1120 and execute the neural network using the read / written data. When the neural network is executed, the processor 1110 can repeatedly perform operations on the audio signal to generate data related to the output. That is, the processor 1110 can receive the audio signal at a predetermined period and repeatedly perform the above-mentioned layer-related operations at each predetermined period.

[0146] In one embodiment, the processor 1110 is directly connected to the memory 1120 and can perform PIM-based operations to drive the neural network, where the memory 1120 may include a crossbar array circuit that receives instructions from the processor 1110 and performs in-memory operations.

[0147] In one embodiment, the neuromorphic device 1100 may be a server. The server may be realized by a computer device or multiple computer devices that communicate over a network to provide instructions, code, files, content, services, etc. The server may receive data required for speech recognition and perform speech recognition based on the received data.

[0148] Meanwhile, embodiments of the present invention may be realized in the form of a computer program executable by various components on a computer, and such a computer program may be recorded on a computer-readable medium, including magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and hardware devices specially configured to store and execute program instructions, such as ROMs, RAMs, and flash memories.

[0149] On the other hand, the computer program may be one specially designed and constructed for the present invention, or it may be one that is well known and available to those skilled in the art of computer software. Examples of computer programs include not only machine code, such as that produced by a compiler, but also high-level language code that is executed by a computer using an interpreter or the like.

[0150] According to one embodiment, the methods according to various embodiments of the present disclosure can be provided in a computer program product. The computer program product can be traded between sellers and buyers as a commodity. The computer program product can be distributed in the form of a machine-readable storage medium (e.g., a compact disc read only memory (CD-ROM)) or can be distributed in the form of an application store (e.g., the Play Store). TM) or directly between two user devices. In the case of online distribution, at least a part of the computer program product may be at least temporarily stored or temporarily generated on a machine-readable storage medium, such as a memory of a manufacturer's server, an application store's server, or an intermediary server.

[0151] Unless there is an explicit order description or a contrary description of the steps constituting the method according to the present invention, the steps can be performed in any suitable order. The present invention is not necessarily limited to the order of the steps described above. The use of all examples or exemplary terms in the present invention is merely for the purpose of explaining the present invention in detail, and the scope of the present invention is not limited by the examples or exemplary terms unless limited by the claims. In addition, those skilled in the art will understand that various modifications, combinations and changes can be made within the scope of the claims or their equivalents according to design conditions and factors.

[0152] Therefore, the concept of the present invention should not be limited to the above-described embodiments, and all scopes equivalent to or modified equivalently to the scope of the attached claims, as well as the scope of the present invention, fall within the scope of the concept of the present invention.

Claims

1. In neuromorphic devices that realize neural networks, a memory having at least one program stored therein; an on-chip memory including a crossbar array circuit; at least one processor for executing said at least one program to drive said neural network; The at least one processor Receives audio signals, inputting the audio signal into the neural network trained based on predetermined training data, and outputting a speech recognition result; The neural network comprises: A neuromorphic device comprising: a Fourier transform layer that converts the audio signal from a time band to a frequency band to generate a frequency signal; a mel generation layer that generates a mel spectrogram from the frequency signal; and an output layer that outputs the speech recognition result based on speech features of the mel spectrogram.

2. The neural network comprises: The neuromorphic device of claim 1 , further comprising one or more hidden layers that classify the speech features of the mel spectrogram, the one or more hidden layers being located between the mel generation layer and the output layer.

3. The output layer is The neuromorphic device of claim 2, wherein the hidden layer outputs the speech recognition result by determining whether the audio signal corresponds to any one of a plurality of preset keywords as a result of classifying the speech features.

4. The predetermined learning data is audio data generated by combining the plurality of preset keywords and noise sets; The one or more hidden layers include: The neuromorphic device according to claim 3 , wherein the weights are adjusted based on the predetermined learning data.

5. The neural network comprises: The neuromorphic device of claim 3 , further comprising: a null value outputted as the speech recognition result when the audio signal corresponds to a negative signal similar to at least one of the plurality of preset keywords.

6. The processor, The neuromorphic device of claim 1 , directly connected to the on-chip memory and performing PIM (Processing-in-Memory) based computation.

7. The processor, converting the audio signal corresponding to a digital signal into an analog signal and inputting the analog signal into the Fourier transform layer; The neuromorphic device according to claim 1 , wherein the speech recognition result corresponding to an analog signal output from the output layer is converted into a digital signal.

8. The processor, The neuromorphic device of claim 2 , further comprising: in response to determining, based on the mel spectrogram, that the audio signal does not correspond to any of a plurality of preset keywords, the mel spectrogram is not input to the hidden layer.

9. 1. A method of operating a neuromorphic device implementing a neural network, comprising: receiving an audio signal; and inputting the audio signal into a neural network trained based on predetermined training data, and outputting a speech recognition result. The neural network comprises: a Fourier transform layer that converts the audio signal from a time band to a frequency band to generate a frequency signal, a mel generation layer that generates a mel spectrogram from the frequency signal, and an output layer that outputs the speech recognition result based on speech features of the mel spectrogram.

10. A computer-readable recording medium having recorded thereon a program for executing the method according to claim 9 on a computer.

Images